atoti_parquet.ParquetLoad

atoti_parquet.ParquetLoad(
    path: Path | str,
    *,
    client_side_encryption: ClientSideEncryptionConfig | None = None,
    columns: Mapping[str, str] = frozendict({}),
)

The definition of an Apache Parquet file load. The load() outcome depends on the error_handling argument. For instance, the following Parquet file has a Count column stored as float64, but table expects int.

Parquet is strongly typed, so the type mismatch is caught up front and no rows load regardless of error_handling. This argument only controls whether an error or warning is raised and what is logged. See CsvLoad for an example where invalid rows are skipped individually.

>>> import pyarrow as pa
>>> import pyarrow.parquet as pq
>>> file_path = tmp_path / "data.parquet"
>>> pq.write_table(
...     pa.table(
...         {
...             "City": ["Paris", "London"],
...             "Count": pa.array([3.14, 2.72]),
...         }
...     ),
...     file_path,
... )
>>> table = session.create_table(
...     "Error handling",
...     data_types={"City": "String", "Count": "int"},
...     keys={"City"},
... )

"fail" raises an error and interrupts the load:

>>> table += ("Rome", 42)
>>> table.head()
      Count
City
Rome     42
>>> from atoti_parquet import ParquetLoad
>>> parquet_load = ParquetLoad(file_path)
>>> with session.tables.data_transaction():  
...     table.load(parquet_load, error_handling="fail")
Traceback (most recent call last):
    ...
py4j.protocol.Py4JError: ...Caused by: com.activeviam.tech.core.api.exceptions.ConfigurationException: Incompatible Type: datastore type 'int' doesn't match parquet type 'double' for the store field 'Count' and parquet field 'Count'...
>>> table.head()
      Count
City
Rome     42
>>> exception_message = "Caused by: com.activeviam.tech.core.api.exceptions.ConfigurationException: Incompatible Type: datastore type 'int' doesn't match parquet type 'double' for the store field 'Count' and parquet field 'Count'"
>>> _count_transaction_exceptions(exception_message) > 0
True

"warn" skips the incompatible data and raises a Python warning:

>>> from warnings import catch_warnings, simplefilter
>>> with (
...     session.tables.data_transaction(),
...     catch_warnings(record=True) as warnings,
... ):
...     simplefilter("always")
...     table.load(parquet_load, error_handling="warn")
>>> [warning.message for warning in warnings]
[UserWarning("Some errors occurred while loading data into t['Error handling'].")]
>>> table.head()
      Count
City
Rome     42
>>> _count_transaction_exceptions(exception_message) > 0
True

"log" skips the incompatible data and logs the errors:

>>> with session.tables.data_transaction():
...     table.load(parquet_load, error_handling="log")
>>> table.head()
      Count
City
Rome     42
>>> _count_transaction_exceptions(exception_message) > 0
True

"skip" skips the incompatible data without logging:

>>> with session.tables.data_transaction():
...     table.load(parquet_load, error_handling="skip")
>>> table.head()
      Count
City
Rome     42
>>> _count_transaction_exceptions(exception_message)
0

See also: The other DataLoad implementations.

Attributes

path

The path to the Parquet file. If a path pointing to a directory is provided, all of the files with the .parquet extension in the directory will be loaded into the same table and, as such, they are all expected to share the same schema. The path can also be a glob pattern (e.g. "path/to/directory/*.parquet").

client_side_encryption

ClientSideEncryptionConfig | None

columns

Mapping from file column names to table column names. When the mapping is not empty, columns of the file absent from the mapping keys will not be loaded. Other parameters accepting column names expect to be passed table column names (i.e. values of this mapping) and not file column names.

Releases and upgrades

Getting started

Guides

API reference

ParquetLoad

Attributes

path

client_side_encryption

columns

​Attributes

​path

​client_side_encryption

​columns

Attributes

path

client_side_encryption

columns