> ## Documentation Index
> Fetch the complete documentation index at: https://docs.activeviam.com/llms.txt
> Use this file to discover all available pages before exploring further.

# CsvLoad

<span id="atoti.CsvLoad" />

> atoti.CsvLoad(<br />
>     *path*: [Path](https://docs.python.org/3/library/pathlib.html#pathlib.Path) | [str](https://docs.python.org/3/library/stdtypes.html#str),<br />
>     \*,<br />
>     *array\_separator*: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = `None`,<br />
>     *client\_side\_encryption*: [ClientSideEncryptionConfig](./atoti.ClientSideEncryptionConfig#atoti.ClientSideEncryptionConfig) | [None](https://docs.python.org/3/library/constants.html#None) = `None`,<br />
>     *columns*: [Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)\[[str](https://docs.python.org/3/library/stdtypes.html#str), [str](https://docs.python.org/3/library/stdtypes.html#str)] | [Sequence](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence)\[[str](https://docs.python.org/3/library/stdtypes.html#str)] = `frozendict({})`,<br />
>     *date\_patterns*: [Mapping](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping)\[[str](https://docs.python.org/3/library/stdtypes.html#str), [str](https://docs.python.org/3/library/stdtypes.html#str)] = `frozendict({})`,<br />
>     *encoding*: [str](https://docs.python.org/3/library/stdtypes.html#str) = `'utf-8'`,<br />
>     *separator*: [str](https://docs.python.org/3/library/stdtypes.html#str) | [None](https://docs.python.org/3/library/constants.html#None) = `','`,<br />
>     *true\_values*: [Set](https://docs.python.org/3/library/collections.abc.html#collections.abc.Set)\[[Any](https://docs.python.org/3/library/typing.html#typing.Any)] = `frozenset({})`,<br />
>     *false\_values*: [Set](https://docs.python.org/3/library/collections.abc.html#collections.abc.Set)\[[Any](https://docs.python.org/3/library/typing.html#typing.Any)] = `frozenset({})`,<br />
>     *process\_quotes*: [bool](https://docs.python.org/3/library/functions.html#bool) | [None](https://docs.python.org/3/library/constants.html#None) = `True`,<br />
> )

The definition of a CSV file load.

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> import csv
>>> with open(file_path, "w") as csv_file:
...     writer = csv.writer(csv_file)
...     writer.writerows(
...         [
...             ("city", "area", "country", "population"),
...             ("Tokyo", "Kantō", "Japan", 14_094_034),
...             ("Johannesburg", "Gauteng", "South Africa", 4_803_262),
...             (
...                 "Barcelona",
...                 "Community of Madrid",
...                 "Madrid",
...                 3_223_334,
...             ),
...         ]
...     )
```

Using [`columns`](#atoti.CsvLoad.columns) to drop the **population** column and rename and reorder the remaining ones:

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> csv_load = tt.CsvLoad(
...     file_path,
...     columns={"city": "City", "area": "Region", "country": "Country"},
... )
>>> session.tables.infer_data_types(csv_load)
{'City': 'String', 'Region': 'String', 'Country': 'String'}
```

Creating a table and loading data into it from a headerless CSV file:

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> with open(file_path, "w") as csv_file:
...     writer = csv.writer(csv_file)
...     writer.writerows(
...         [
...             ("Tokyo", "Kantō", "Japan", 14_094_034),
...             ("Johannesburg", "Gauteng", "South Africa", 4_803_262),
...             (
...                 "Madrid",
...                 "Community of Madrid",
...                 "Spain",
...                 3_223_334,
...             ),
...         ]
...     )
>>> csv_load = tt.CsvLoad(
...     file_path,
...     columns=["City", "Area", "Country", "Population"],
... )
>>> data_types = session.tables.infer_data_types(csv_load)
>>> data_types
{'City': 'String', 'Area': 'String', 'Country': 'String', 'Population': 'int'}
>>> table = session.create_table(
...     "Columns example",
...     data_types=data_types,
...     keys={"Country"},
... )
>>> table.load(csv_load)
>>> table.head().sort_index()
                      City                 Area  Population
Country
Japan                Tokyo                Kantō    14094034
South Africa  Johannesburg              Gauteng     4803262
Spain               Madrid  Community of Madrid     3223334
```

[`true_values`](#atoti.CsvLoad.true_values) and [`false_values`](#atoti.CsvLoad.false_values) default behavior is to only parse `"True"` and `"true"` has `True` and `"False"` and `"false"` as `False`:

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> with open(file_path, "w") as csv_file:
...     writer = csv.writer(csv_file)
...     writer.writerows(
...         [
...             (
...                 "ID",
...                 "No & Yes",
...                 "no & yes (lower case)",
...                 "False & True",
...                 "false & true (lower case)",
...                 "0 & 1",
...             ),
...             ("abc", "No", "no", "False", "false", 0),
...             ("def", "Yes", "yes", "True", "true", 1),
...             ("ghi", "", "", "", "", ""),
...         ]
...     )
>>> csv_load = tt.CsvLoad(file_path)
>>> data_types = session.tables.infer_data_types(csv_load)
>>> data_types
{'ID': 'String', 'No & Yes': 'String', 'no & yes (lower case)': 'String', 'False & True': 'boolean', 'false & true (lower case)': 'boolean', '0 & 1': 'int'}
>>> table = session.create_table(
...     "Default true_values and false_values example",
...     data_types=data_types,
...     keys={"ID"},
... )
>>> table.load(csv_load)
```

Missing values in `"boolean"` columns become `False` as shown in [`atoti.Column.default_value`](./atoti.Column.default_value#atoti.Column.default_value):

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> table.head().sort_index()
    No & Yes no & yes (lower case)  False & True  false & true (lower case)  0 & 1
ID
abc       No                    no         False                      False      0
def      Yes                   yes          True                       True      1
ghi      N/A                   N/A         False                      False   <NA>
```

Extra values can be specified in addition to the case insensitive `"true"` or `"false"`:

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> csv_load = tt.CsvLoad(
...     file_path,
...     false_values={"No", "0"},
...     true_values={"Yes", "1"},
... )
>>> data_types = session.tables.infer_data_types(csv_load)
>>> data_types
{'ID': 'String', 'No & Yes': 'boolean', 'no & yes (lower case)': 'String', 'False & True': 'boolean', 'false & true (lower case)': 'boolean', '0 & 1': 'boolean'}
>>> table = session.create_table(
...     "Custom true_values and false_values example",
...     data_types=data_types,
...     keys={"ID"},
... )
>>> table.load(csv_load)
>>> result = table.head().sort_index()
>>> result
     No & Yes no & yes (lower case)  False & True  false & true (lower case)  0 & 1
ID
abc     False                    no         False                      False  False
def      True                   yes          True                       True   True
ghi     False                   N/A         False                      False  False
```

The [`load()`](./atoti.Table.load#atoti.Table.load) outcome depends on the *error\_handling* argument.
For instance, the following CSV file has a badly formed line #4 where the **City** name contains an unquoted comma in **New,York**, producing an extra field on that row:

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> from os import linesep
>>> _ = file_path.write_text(
...     linesep.join(
...         (
...             "City,Count",
...             "Paris,100",
...             "London,80",
...             "New,York,80",
...             "Berlin,70",
...             "Jakarta,75",
...         )
...     )
... )
>>> table = session.create_table(
...     "Error handling",
...     data_types={"City": "String", "Count": "int"},
...     keys={"City"},
... )
```

* `"fail"` raises an error on the first invalid row and interrupts the load:
  ```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
  >>> table += ("Rome", 42)
  >>> table.head()
        Count
  City
  Rome     42
  >>> csv_load = tt.CsvLoad(file_path)
  >>> with session.tables.data_transaction():  
  ...     table.load(csv_load, error_handling="fail")
  Traceback (most recent call last):
      ...
  py4j.protocol.Py4JError: ...Caused by: java.lang.NumberFormatException: Error at index 0 in: "York"...
  >>> table.head()
        Count
  City
  Rome     42
  >>> exception_message = (
  ...     "Exception while loading source: CsvSource. At line 3, line=New,York,80"
  ... )
  >>> _count_transaction_exceptions(exception_message) > 0
  True
  ```
* `"warn"` skips the invalid rows and raises a Python warning:
  ```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
  >>> from warnings import catch_warnings, simplefilter
  >>> table.drop()
  >>> with (
  ...     session.tables.data_transaction(),
  ...     catch_warnings(record=True) as warnings,
  ... ):
  ...     simplefilter("always")
  ...     table.load(csv_load, error_handling="warn")
  >>> [warning.message for warning in warnings]
  [UserWarning("Some errors occurred while loading data into t['Error handling'].")]
  >>> table.head().sort_index()
           Count
  City
  Berlin      70
  Jakarta     75
  London      80
  Paris      100
  >>> _count_transaction_exceptions(exception_message) > 0
  True
  ```
* `"log"` skips the invalid rows and logs the errors:
  ```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
  >>> table.drop()
  >>> with session.tables.data_transaction():
  ...     table.load(csv_load, error_handling="log")
  >>> table.head().sort_index()
           Count
  City
  Berlin      70
  Jakarta     75
  London      80
  Paris      100
  >>> _count_transaction_exceptions(exception_message) > 0
  True
  ```
* `"skip"` skips the invalid rows without logging:
  ```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
  >>> table.drop()
  >>> with session.tables.data_transaction():
  ...     table.load(csv_load, error_handling="skip")
  >>> table.head().sort_index()
           Count
  City
  Berlin      70
  Jakarta     75
  London      80
  Paris      100
  >>> _count_transaction_exceptions(exception_message)
  0
  ```

<Callout icon="link">
  **See also**:
  The other [`DataLoad`](./atoti.data_load.DataLoad#atoti.data_load.DataLoad) implementations.
</Callout>

### Attributes

<h4 id="atoti.CsvLoad.path">
  *path*
</h4>

The path to the CSV file to load.

`.gz`, `.tar.gz` and `.zip` files containing compressed CSV(s) are also supported.

The path can also be a glob pattern (e.g. `"path/to/directory/*.csv"`).

<h4 id="atoti.CsvLoad.array_separator">
  *array\_separator*
</h4>

The character separating array elements.

If not `None`, any field containing this separator will be parsed as an [`array`](./atoti.array#module-atoti.array).

<h4 id="atoti.CsvLoad.client_side_encryption">
  *client\_side\_encryption*
</h4>

> [ClientSideEncryptionConfig](./atoti.ClientSideEncryptionConfig#atoti.ClientSideEncryptionConfig) | [None](https://docs.python.org/3/library/constants.html#None)

<h4 id="atoti.CsvLoad.columns">
  *columns*
</h4>

The collection used to name, rename, or filter the CSV file columns.

* If an empty collection is passed, the CSV file must have a header.
  : The CSV column names must follow the [`Table`](./atoti.Table#atoti.Table) column names.
* If a non empty [`Mapping`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Mapping) is passed, the CSV file must have a header and the mapping keys must be column names of the CSV file.
  : Columns of the CSV file absent from the mapping keys will not be loaded.
  The mapping values correspond to the [`Table`](./atoti.Table#atoti.Table) column names.
  The other attributes of this class accepting column names expect to be passed values of this mapping, not keys.
* If a non empty [`Sequence`](https://docs.python.org/3/library/collections.abc.html#collections.abc.Sequence) is passed, the CSV file must not have a header and the sequence must have as many elements as there are columns in the CSV file.
  : The sequence elements correspond to the [`Table`](./atoti.Table#atoti.Table) column names.

<h4 id="atoti.CsvLoad.date_patterns">
  *date\_patterns*
</h4>

A column name to [date pattern](https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/time/format/DateTimeFormatter.html) mapping that can be used when the built-in date parsers fail to recognize the formatted dates in the CSV file.

<h4 id="atoti.CsvLoad.encoding">
  *encoding*
</h4>

The encoding to use to read the CSV file.

<h4 id="atoti.CsvLoad.separator">
  *separator*
</h4>

The character separating the values of each line.

If `None`, it will be inferred in a preliminary partial load.

<h4 id="atoti.CsvLoad.true_values">
  *true\_values*
</h4>

The strings that will be parsed as `True`, in addition to case insensitive `"True"`.

<h4 id="atoti.CsvLoad.false_values">
  *false\_values*
</h4>

The strings that will be parsed as `False`, in addition to case insensitive `"False"`.

<h4 id="atoti.CsvLoad.process_quotes">
  *process\_quotes*
</h4>

Whether double quotes should be processed to follow the official CSV specification:

* `True`:
  > Each field may or may not be enclosed in double quotes (however some programs, such as Microsoft Excel, do not use double quotes at all).
  > If fields are not enclosed with double quotes, then double quotes may not appear inside the fields.
  >
  > * A double quote appearing inside a field must be escaped by preceding it with another double quote.
  > * Fields containing line breaks, double quotes, and commas should be enclosed in double-quotes.
* `False`: all double-quotes within a field will be treated as any regular character, following Excel’s behavior.
  : In this mode, it is expected that fields are not enclosed in double quotes.
  It is also not possible to have a line break inside a field.
* `None`: the behavior will be inferred in a preliminary partial load.