atoti.Table.load()

Table.load(
    data: Table | DataFrame | DataLoad,
    /,
    *,
    error_handling: Literal[‘fail’, ‘warn’, ‘log’, ‘skip’] = 'fail',
) → None

Load data into the table.

Parameters

data

The data to load. Besides the various DataLoad implementations, there is built-in support for:

pyarrow.Table:

>>> from datetime import date
>>> import pyarrow as pa
>>> table = session.create_table(
...     "Sales",
...     data_types={
...         "ID": "String",
...         "Product": "String",
...         "Price": "int",
...         "Quantity": "int",
...         "Date": "LocalDate",
...     },
...     keys={"ID"},
... )
>>> arrow_table = pa.Table.from_pydict(
...     {
...         "ID": pa.array(["ab", "cd"]),
...         "Product": pa.array(["phone", "watch"]),
...         "Price": pa.array([699, 349]),
...         "Quantity": pa.array([1, 2]),
...         "Date": pa.array([date(2024, 3, 5), date(2024, 12, 12)]),
...     }
... )
>>> table.load(arrow_table)
>>> table.head().sort_index()
   Product  Price  Quantity       Date
ID
ab   phone    699         1 2024-03-05
cd   watch    349         2 2024-12-12

pandas.DataFrame:

>>> import pandas as pd
>>> pandas_dataframe = pd.DataFrame(
...     {
...         "ID": ["ef", "gh"],
...         "Product": ["laptop", "book"],
...         "Price": [2599, 19],
...         "Quantity": [3, 5],
...         "Date": [date(2023, 8, 10), date(2024, 1, 13)],
...     }
... )
>>> table.load(pandas_dataframe)
>>> table.head().sort_index()
   Product  Price  Quantity       Date
ID
ab   phone    699         1 2024-03-05
cd   watch    349         2 2024-12-12
ef  laptop   2599         3 2023-08-10
gh    book     19         5 2024-01-13

Anything that can be converted to a pyarrow.Table or a pandas.DataFrame can be loaded too:

NumPy array:

>>> import numpy as np
>>> numpy_array = np.asarray(
...     [
...         ["ij", "watch", 299, 1, date(2022, 7, 20)],
...         ["kl", "keyboard", 69, 1, date(2023, 5, 8)],
...     ],
...     dtype=object,
... )
>>> pandas_dataframe_from_numpy = pd.DataFrame(
...     numpy_array, columns=list(table)
... )
>>> table.load(pandas_dataframe_from_numpy)
>>> table.head(10).sort_index()
     Product  Price  Quantity       Date
ID
ab     phone    699         1 2024-03-05
cd     watch    349         2 2024-12-12
ef    laptop   2599         3 2023-08-10
gh      book     19         5 2024-01-13
ij     watch    299         1 2022-07-20
kl  keyboard     69         1 2023-05-08

Spark DataFrame:

This requires:

A Spark version supporting Java 25 (pyspark >= 4.2).
A JDK bundling the jdk.incubator.vector module, used by Spark’s launcher (jdk4py does not bundle this module).

>>> from pyspark.sql import Row, SparkSession
>>> spark = SparkSession.builder.getOrCreate()
>>> spark_dataframe = spark.createDataFrame(
...     [
...         Row(
...             ID="mn",
...             Product="glasses",
...             Price=129,
...             Quantity=2,
...             Date=date(2021, 3, 3),
...         ),
...         Row(
...             ID="op",
...             Product="battery",
...             Price=49,
...             Quantity=2,
...             Date=date(2024, 11, 7),
...         ),
...     ]
... )
>>> arrow_table_from_spark = spark_dataframe.toArrow()
>>> spark.stop()
>>> table.load(arrow_table_from_spark)
>>> table.head(10).sort_index()
     Product  Price  Quantity       Date
ID
ab     phone    699         1 2024-03-05
cd     watch    349         2 2024-12-12
ef    laptop   2599         3 2023-08-10
gh      book     19         5 2024-01-13
ij     watch    299         1 2022-07-20
kl  keyboard     69         1 2023-05-08
mn   glasses    129         2 2021-03-03
op   battery     49         2 2024-11-07

The += operator provides syntax sugar to load a single row, given either as a:

tuple:

>>> table += ("qr", "mouse", 29, 3, date(2024, 11, 7))
>>> table.head(10).sort_index()
     Product  Price  Quantity       Date
ID
ab     phone    699         1 2024-03-05
cd     watch    349         2 2024-12-12
ef    laptop   2599         3 2023-08-10
gh      book     19         5 2024-01-13
ij     watch    299         1 2022-07-20
kl  keyboard     69         1 2023-05-08
mn   glasses    129         2 2021-03-03
op   battery     49         2 2024-11-07
qr     mouse     29         3 2024-11-07

Mapping:

>>> table += {  # The order of the keys does not matter.
...     "Product": "screen",
...     "Quantity": 1,
...     "Price": 599,
...     "Date": date(2023, 5, 8),
...     "ID": "st",
... }
>>> table.head(10).sort_index()
     Product  Price  Quantity       Date
ID
ab     phone    699         1 2024-03-05
cd     watch    349         2 2024-12-12
ef    laptop   2599         3 2023-08-10
gh      book     19         5 2024-01-13
ij     watch    299         1 2022-07-20
kl  keyboard     69         1 2023-05-08
mn   glasses    129         2 2021-03-03
op   battery     49         2 2024-11-07
qr     mouse     29         3 2024-11-07
st    screen    599         1 2023-05-08

error_handling

How to handle rows that cannot be loaded. Atoti tables are strongly typed and reject rows whose values do not match the table’s atoti.Column.data_types. For instance, the following Arrow table has a Count column stored as string and thus cannot be loaded into table’s int Count column.

>>> arrow_table = pa.table(
...     {
...         "City": ["Paris"],
...         "Count": ["not a number"],
...     }
... )
>>> table = session.create_table(
...     "Error handling",
...     data_types={"City": "String", "Count": "int"},
...     keys={"City"},
... )

"fail" raises an error and interrupts the load:

>>> table += ("Rome", 42)
>>> table.head()
      Count
City
Rome     42

>>> with session.tables.data_transaction():  
...     table.load(arrow_table, error_handling="fail")
Traceback (most recent call last):
    ...
py4j.protocol.Py4JError: ...
Caused by: java.lang.IllegalArgumentException: Field 'Count': Arrow type VarCharVector (Utf8) has no dedicated reader for store type INTEGER...

The table is left unchanged:

>>> table.head()
      Count
City
Rome     42

The error can be found in logs_path too:

>>> exception_message = "java.lang.IllegalArgumentException: Field 'Count': Arrow type VarCharVector (Utf8) has no dedicated reader for store type INTEGER"
>>> _count_transaction_exceptions(exception_message) > 0
True

"warn" skips the incompatible data and raises a Python warning:

>>> from warnings import catch_warnings, simplefilter
>>> with (
...     session.tables.data_transaction(),
...     catch_warnings(record=True) as warnings,
... ):
...     simplefilter("always")
...     table.load(arrow_table, error_handling="warn")

The table is still unchanged:

>>> table.head()
      Count
City
Rome     42

A Python warning is raised:

>>> [warning.message for warning in warnings]
[UserWarning("Some errors occurred while loading data into t['Error handling'].")]

The error is logged:

>>> _count_transaction_exceptions(exception_message) > 0
True

"log" skips the incompatible data and logs the errors:

>>> with session.tables.data_transaction():
...     table.load(arrow_table, error_handling="log")
>>> table.head()
      Count
City
Rome     42
>>> _count_transaction_exceptions(exception_message) > 0
True

"skip" skips the incompatible data without logging:

>>> with session.tables.data_transaction():
...     table.load(arrow_table, error_handling="skip")
>>> table.head()
      Count
City
Rome     42
>>> _count_transaction_exceptions(exception_message)
0

Error handling is most useful with loosely typed sources such as CsvLoad and JdbcLoad, where a value incompatible with its column’s type is only discovered as each row is parsed, partway through the load.

See also: DataLoad, load_async(), data_transaction(), and infer_data_types().

Releases and upgrades

Getting started

Guides

API reference

Table.load()

Parameters

data

error_handling

​Parameters

​data

​error_handling

Parameters

data

error_handling