> ## Documentation Index
> Fetch the complete documentation index at: https://docs.activeviam.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Data loading

> How an Atoti Enterprise Risk Suite solution loads CSV data: default channels and topics ship automatically, a single `IDataLoadConfigurer` bean extends or overrides them, and loading runs on startup or via a REST API, sourced from a local filesystem or a cloud bucket (AWS S3, Azure, GCP).

This page describes how CSV data is loaded into an Atoti Enterprise Risk Suite solution: how a
solution ships its default loading, how a customer application overrides or extends it, and how a
load is triggered and sourced from a local filesystem or a cloud bucket (AWS S3, Azure, GCP).

## How does data loading work?

Each solution ships its default channels and CSV topics. These are present as soon as the
solution's starter is on the classpath. However, no data loads until the application requests a load
of the topics to load. See [How does an application request a load?](#how-does-an-application-request-a-load).

A customer application customizes or extends loading with its own bean of type
`IDataLoadConfigurer`, from `com.activeviam.source.common.api.foundry.config`. This customer bean
runs after the solution's default loading, against the same configurator instance. It can fetch a
channel or topic the solution already declared and change it. It can also declare new channels and
topics.

A multi-solution application runs more than one solution in the same process. Each co-deployed
solution's default channels and topics are jointly shipped and available for loading.

Two independent aspects of loading are configured separately:

* **What is loaded**: the channels (target stores) and topics (CSV sources) declared by the
  solution's default loading and any customer override. See [What declares the channels and topics?](#what-declares-the-channels-and-topics).
* **When it runs**: usually an initial load on startup, plus a runtime REST API for individual
  as-of dates. See [How is data loaded on startup?](#how-is-data-loaded-on-startup) and
  [How is data loaded at runtime?](#how-is-data-loaded-at-runtime).

## What declares the channels and topics?

A **channel** is source-agnostic and model-bound. It targets a datastore table and selects which
fields it fills. It can also attach a calculator or a custom tuple publisher. A **topic** is the source:
for CSV, it can be a file path or cloud location, plus a file pattern. It also carries the parsers that
map columns to fields. A topic is attached to one or more channels by name.

Channels and topics are independent. A topic can feed several channels, and a channel can be fed by
several topics. Their names are fixed strings.

A cube publishes these names through two dedicated beans, each with one named accessor per channel
or topic, the way the cube's database model carries one per table. Both include the channels and
topics the cube shares with other cubes, named as those cubes name them.

* `I[Cube name]CubeChannels` (e.g. `IVaRCubeChannels`) returns the channel names, plus the names of
  the post-load triggers the cube declares.
* `I[Cube name]CubeTopics` (e.g. `IVaRCubeTopics`) returns the topic names, plus the aliases grouping
  them (see [How does an application request a load?](#how-does-an-application-request-a-load)).

So `topics.markToFuture()` is the name of the topic reading the mark-to-future files, and
`channels.markToFuture()` the name of the channel filling the mark-to-future table. A customer
overrides or extends loading by referring to these names rather than writing them out, the same way
application code refers to levels, measures, and tables by name.

The following example shows what a customer's `IDataLoadConfigurer` looks like when it declares a
new channel and one CSV topic for a `Positions` table:

```java theme={"languages":{"custom":["/engine/python-sdk/6.2/languages/pycon.tmLanguage.json"]}}
@Bean
IDataLoadConfigurer dataLoadConfigurer(final DataLoadProperties props) {
  return configurator -> {
    final var table = "Positions";
    configurator.channels().addChannel(table, table);
    configurator
        .topics()
        .addTopic(table, ILocalCsvTopicConfigurator.class)
        // base-path carries the #{AS_OF_DATE} placeholder, so the pattern stays date-agnostic
        .path(props.getBasePath())
        .filePattern("glob:positions/*.csv")
        .parsersFromTable(table) // drive parsing from the target table's field types
        .attachChannel(table);
  };
}
```

* `addChannel` creates a channel targeting the given table. `attachChannel` attaches a topic to a
  channel by name. `parsersFromTable` registers parsers driven by the target table's field
  types, so plain, header-based files need no manual column mapping.
* Each load request targets one as-of date. When `base-path` contains the `#{AS_OF_DATE}`
  placeholder, as it does in a generated application, the date is substituted into it, and the
  topic's date-agnostic file pattern (`positions/*.csv`) resolves under that date's folder. Without
  the placeholder, every request reads the files directly under `base-path`, whatever the date.
* A cloud topic uses the same code, but with `IAwsCsvTopicConfigurator` (or the Azure/GCP
  equivalent) and a `.bucket(...)` call. A solution declares one mutually exclusive bean per
  `topic-type` value it ships topics for, plus one declaring its channels alone for the values it
  does not, so your own topics for those still have channels to attach to.

See [How is the data source configured?](#how-is-the-data-source-configured) for the full
`base-path`, `topic-type`, and cloud property reference.

See [What happens when solutions
share a table?](#what-happens-when-solutions-share-a-table) for how additional channels and topics interact with the tables a
solution already loads.

A customer application overrides loading with its own `IDataLoadConfigurer` bean. The following
example fetches a topic the solution's default loading already declared and repoints it at a
different file pattern. It names the topic through the cube's topics bean (`IPfeCubeTopics`
here) rather than as a literal, so an override cannot drift from what the solution declared:

```java theme={"languages":{"custom":["/engine/python-sdk/6.2/languages/pycon.tmLanguage.json"]}}
@Bean
IDataLoadConfigurer markToFutureOverride(final IPfeCubeTopics topics) {
  // Fetches the topic the solution's default loading already declared and repoints it; this runs
  // after those defaults, against the same IDataLoadConfigurator. The name comes from the cube's
  // topics API, so an override cannot drift from what the solution declared.
  return configurator ->
      configurator
          .topics()
          .topic(topics.markToFuture(), ICsvTopicConfigurator.class)
          .filePattern("glob:markToFuture/corrected/*.csv");
}
```

The same `configurator.topics().topic(name, ICsvTopicConfigurator.class)` call also works to change the
path, add parsers, or attach the topic to additional channels. Use `ICsvTopicConfigurator`
rather than a source-specific type such as `ILocalCsvTopicConfigurator`: the lookup would fail when the
topic is declared against a different one. Naming the shared interface keeps the override
working regardless of the source `topic-type` selects.

> **Note:** an application declares at most one `IDataLoadConfigurer` bean. It is injected as a
> single bean by type, so a second bean of that type fails startup with
> `NoUniqueBeanDefinitionException`. The two examples above show two things that one bean can do:
> declare a new channel and topic, and fetch and repoint an existing one. They are not two separate
> beans.

> **Tip:** name each topic after the table it loads. The data-loading API builds an
> implicit topic for every requested name. A name that does not match a datastore table still
> works. It produces a harmless, noisy `Implicit topic for table '…' cannot be built` warning at
> load time.

## What happens when solutions share a table?

Several solutions can run in one application, and some of them share data model components: the
counterparty and trade attributes tables are common to the risk cubes, and the netting-set tables are
specific to Counterparty Credit Risk (Enterprise Risk). Each solution's shipped loading declares what it needs
against those tables, without knowing which other solutions are deployed.

Identical declarations are reused rather than duplicated. Whichever solution is configured first
declares the channel and the topic, and the ones after it reuse them. A file feeding a table that
several solutions use is therefore read by one topic, into that one table, through one channel each
of their cubes reads from. Nothing has to be configured for this: a solution deployed on its own is
always the one declaring, so its loading is unchanged either way.

A customer application declares its own channel names, so it declares them directly. Which topic it
needs depends on where its rows go. A channel filling a table of its own can attach to the topic the
solution already declared, leaving the file read once and its rows delivered to both tables. A
channel filling a table the solution already loads needs a topic of its own over the same files,
because a topic only fans out to channels on different tables.

Both cases refer to something the solution named: a topic to attach to, or the files it reads. Take
those from the solution's topics and file-patterns beans rather than writing them out, so they cannot drift from what
the solution actually declares. The example below is Counterparty Credit Risk (Enterprise Risk); every solution publishes
the equivalent bean:

```java theme={"languages":{"custom":["/engine/python-sdk/6.2/languages/pycon.tmLanguage.json"]}}
@Bean
IDataLoadConfigurer sharedTableChannels(
    final IPfeCube cube,
    final IPfeCubeTopics topics,
    final DataLoadProperties props,
    final ITuplePublisher<Object> myPublisher) {
  return configurator -> {
    // Fills a table of its own, so it can share the topic the solution already declared: the file
    // stays read once and its rows reach both tables.
    configurator.channels().addChannel("MY_TRADE_ATTRIBUTES", "MyTradeAttributes");
    configurator
        .topics()
        // ITopicConfigurator, not the local type: the solution declares this topic as the AWS one
        // when topic-type is AWS, and topic() rejects a type the configurator is not.
        .topic(topics.tradeAttributes(), ITopicConfigurator.class)
        .attachChannel("MY_TRADE_ATTRIBUTES");
    // Fills the same table as the solution's channel, through a publisher of its own, so it needs
    // a topic of its own over the same files.
    final var simulationDates = cube.databaseModel().simulationDates().name();
    configurator
        .channels()
        .addChannel("MY_SIMULATION_DATES", simulationDates)
        .tuplePublisher(myPublisher);
    configurator
        .topics()
        .addTopic("MY_SIMULATION_DATES", ILocalCsvTopicConfigurator.class)
        .path(props.getBasePath())
        .filePattern("glob:{,**/}SimulationDates{,_*}.csv")
        .parsersFromTable(simulationDates)
        .attachChannel("MY_SIMULATION_DATES");
  };
}
```

Declaring channels and topics is not enough on its own. A load must be requested. The suite provides
the machinery for both ways below.

> **Note:** requesting a load belongs to the application rather than to the starter. A starter
> carries the cube, its channels, and its topics; the application decides when they are read. An
> application that opts out starts normally and serves its cube, but loads nothing, because no load
> is ever requested.

Each solution also declares two **aliases** over the topics it ships, and publishes their names as
`loadAlias()` and `unloadAlias()` on its topics bean (`I[Cube name]CubeTopics`). An application names
aliases rather than individual topics, so adding a topic to a solution does not change any
application's configuration. The unload alias covers the topics scoped on the as-of date. It holds
the same topics as the load alias unless some are reloaded wholesale, such as market-data curves,
which stay loaded when an as-of date is unloaded.

## How does an application request a load?

Import the as-of-date data-load configuration:

```java theme={"languages":{"custom":["/engine/python-sdk/6.2/languages/pycon.tmLanguage.json"]}}
@SpringBootApplication
@Import(AsOfDateDataLoadConfiguration.class)
public class MyApplication { ... }
```

That single import gives the application three things: the startup load of the configured dates, the
REST endpoints described below, and the load and unload events any post-load action listens to. One
import covers every cube the application deploys, however many starters it depends on.

## What does a load cover?

`atoti.solution.data-load.as-of-date.topics` names what each request reads. An entry is a topic name
or an alias over several:

| Property | Type | Default | Description |
| - | - | - | - |
| `atoti.solution.data-load.as-of-date.topics` | `List<String>` | `[]` | Topics or aliases loaded for each as-of date. |
| `atoti.solution.data-load.as-of-date.unload-topics` | `List<String>` | unset | What to unload, when it differs from `topics`. Unset unloads the same names; an empty list unloads nothing. A name nothing declares is skipped with a warning. |

```yaml theme={"languages":{"custom":["/engine/python-sdk/6.2/languages/pycon.tmLanguage.json"]}}
atoti:
  solution:
    data-load:
      as-of-date:
        topics:
          - PFE_LOAD
          - VAR_LOAD
        unload-topics:
          - PFE_UNLOAD
          - VAR_UNLOAD
```

Every name resolves into one request per date, so a table two cubes share is read once rather than
once per cube. A name nothing declares is skipped with a warning, in `topics` and in `unload-topics`
alike, rather than failing the request.

Add each solution's unload alias to `unload-topics`. Leaving it unset unloads the names in `topics`,
which includes data that is reloaded wholesale and declares no unload condition, such as a
market-data curve, so the unload fails with `Creating implicit unload topics is not allowed`. An
empty list unloads nothing at all.

## How is data loaded on startup?

Once the database is available, the configured as-of dates are loaded, one scoped request per date,
in chronological order:

| Property | Type | Default | Description |
| - | - | - | - |
| `atoti.solution.data-load.as-of-date.initial-dates.enabled` | `boolean` | `true` | Whether the initial load runs on startup. |
| `atoti.solution.data-load.as-of-date.initial-dates.dates` | `Set<LocalDate>` | `[]` | As-of dates to load on startup (ISO format, e.g. `2024-01-15`). |

For example:

```yaml theme={"languages":{"custom":["/engine/python-sdk/6.2/languages/pycon.tmLanguage.json"]}}
atoti:
  solution:
    data-load:
      as-of-date:
        topics:
          - PFE_LOAD
        unload-topics:
          - PFE_UNLOAD
        initial-dates:
          enabled: true
          dates:
            - 2024-01-15
            - 2024-01-16
```

The dates are application-wide: one request per date covers every cube deployed, rather than one set
of dates per cube.

See [Configuration properties](../reference/configuration-properties#as-of-date-data-load) for the
full property reference.

## How is data loaded at runtime?

The same import exposes a REST controller mapped at `/activeviam/as-of-date/rest/v1`, which loads or unloads a single
as-of date at runtime:

* `POST /activeviam/as-of-date/rest/v1/{asOfDate}` loads data for the given as-of date. The `asOfDate` path variable
  must be in `yyyy-MM-dd` format.
* `DELETE /activeviam/as-of-date/rest/v1/{asOfDate}` unloads the data previously loaded for that as-of date. It
  removes that date's rows from every table scoped on the as-of date. A table not scoped on the
  as-of date, such as a market-data curve table, keeps its rows. Each solution's data-loading page
  lists which tables lose rows and which keep them.

Both operations cover every configured topic in a single scoped request, across every cube the
application deploys. A date that resolves to no configured topic answers normally and loads nothing.

Data a solution derives from what it loads, such as flattened hierarchies and vectorized metrics,
commits in the same transaction as the rows it comes from. A runtime load rebuilds it for that
as-of date and an unload removes it, so it stays consistent with its sources, not only after the
startup load. The in-memory caches a solution builds from a table are refreshed the same way, with
no separate step to run.

Set `atoti.solution.data-load.as-of-date.controller.enabled` to `false` to leave the endpoints out,
for a deployment that must not accept load or unload requests over HTTP. The startup load is
unaffected.

## How is the data source configured?

Where CSV files are read from is controlled by the properties under
`atoti.solution.data-load.csv`. This is a single, shared contract every solution consumes instead
of declaring its own copy:

| Property | Description |
| - | - |
| `topic-type` | `LOCAL` (default), `AWS`, `AZURE` or `GCP`. Selects where the topics read from. |
| `base-path` | Path under which the data folders live, including the `#{AS_OF_DATE}` placeholder substituted with each loaded date (e.g. `/path/to/data/#{AS_OF_DATE}`). A local directory, or the in-bucket prefix for cloud sources. |
| `container` | Cloud sources only: the storage name, either the S3/GCP bucket or the Azure blob container. Ignored for local loading. |

### Local filesystem (default)

The default needs no extra setup beyond pointing at the data:

```yaml theme={"languages":{"custom":["/engine/python-sdk/6.2/languages/pycon.tmLanguage.json"]}}
atoti:
  solution:
    data-load:
      csv:
        topic-type: LOCAL
        base-path: /path/to/data/#{AS_OF_DATE}   # #{AS_OF_DATE} is substituted with each loaded date
```

So a load scoped to `2026-01-09` reads from, for example, `/path/to/data/2026-01-09/positions/*.csv`.

### AWS S3

Loading from S3 requires three additional pieces of setup.

1. **Add the AWS cloud-source dependency** to the application module:

   ```xml theme={"languages":{"custom":["/engine/python-sdk/6.2/languages/pycon.tmLanguage.json"]}}
   <dependency>
     <groupId>com.activeviam.source</groupId>
     <artifactId>cloud-source-aws</artifactId>
     <version>${atoti-server.version}</version>
   </dependency>
   ```

2. **Point the configuration at the bucket and create the S3 client:**

   ```yaml theme={"languages":{"custom":["/engine/python-sdk/6.2/languages/pycon.tmLanguage.json"]}}
   # Creates the default S3Client bean (standard AWS credential/region chain).
   dlc:
     create-default-cloud-client:
       aws: true
   atoti:
     solution:
       data-load:
         csv:
           topic-type: AWS
           base-path: my/in-bucket/prefix/#{AS_OF_DATE}
           container: my-bucket-name
   ```

3. **Provide AWS credentials and a region.** Both are resolved through the standard AWS SDK chain
   (environment variables, `~/.aws/credentials` and `~/.aws/config`, instance profiles, …).

> **Note:** `dlc.create-default-cloud-client.aws: true` is mandatory. The AWS source connector only
> registers when an `S3Client` bean is present. Without it, the application starts but **silently
> loads nothing**: the topics are declared but never fetched.

> **Note:** the default `S3Client` is built **without an explicit region**, so a region must be
> resolvable from the SDK chain (for example the `AWS_REGION` environment variable). If none is
> found, the application fails to start with `Unable to load region from any of the providers`.

Azure and GCP follow the same pattern: set `topic-type` to `AZURE`/`GCP`, add the matching
`cloud-source-azure`/`cloud-source-gcp` dependency, and enable
`dlc.create-default-cloud-client.azure`/`.gcp`.

## What file formats are expected?

Each solution loads data from CSV files, organized into dated folders selected by the `#{AS_OF_DATE}`
placeholder in `base-path`. Each file pattern maps to a target datastore table via a glob pattern
supported by the data-loading API. For common file format definitions, see the
[input file formats reference](../reference/input-file-formats).

## How must CSV files be named?

A CSV file loads into a topic when its name matches `<Name>.csv` or `<Name>_<suffix>.csv`.
`<Name>` is the topic's file name, for example `TradeAttributes` or `MTF`. The [input file formats](../reference/input-file-formats) reference pages list the `<Name>` of
every topic.

The underscore separates the required name from an optional, free-form suffix. A suffix lets one
table's data be split across several files. For example, `TradeAttributes.csv` and
`TradeAttributes_PreTrade.csv` can coexist, or `MTF_1.csv` and `MTF_2.csv`. Each currency
pair can get its own `FXScenarios_<pair>.csv` file.

A name that continues past `<Name>` without an underscore does not load. Neither does a name that
merely ends with `<Name>`. This keeps one topic from picking up another topic's files. The following
examples use the `TradeAttributes` topic:

| File name | Loads into `TradeAttributes`? |
| - | - |
| `TradeAttributes.csv` | Yes |
| `TradeAttributes_PreTrade.csv` | Yes |
| `TradeAttributesPreTrade.csv` | No: not separated from `TradeAttributes` by an underscore |
| `TradeAttributes2.csv` | No: not separated from `TradeAttributes` by an underscore |
| `MyTradeAttributes.csv` | No: the name only ends with `TradeAttributes` |
| `tradeattributes.csv` | No: the match is case-sensitive |

The match applies at any depth below the as-of-date folder resolved from `base-path` (see
[How is the data source configured?](#how-is-the-data-source-configured)). A file can sit directly
in that folder or in any folder below it. For example, `Trades/TradeAttributes.csv` and
`Trades/Current/MTF_1.csv` both match. Folder names play no part in the match. A file whose
name matches no topic is ignored rather than rejected.

Every topic a solution ships uses the glob `glob:{,**/}<Name>{,_*}.csv`. A topic overridden through a
custom `IDataLoadConfigurer` keeps that pattern unless the override sets another one, and a topic the
application declares itself uses the pattern it sets. See
[What declares the channels and topics?](#what-declares-the-channels-and-topics) for how a customer
application changes a topic's file pattern.

## Related pages

* [Input file formats](../reference/input-file-formats)
* [Configuration properties](../reference/configuration-properties)
* [Database model](./database/database-model)
* [Atoti Configuration API: Data loading reference](../../foundry/data-loading)
* [Atoti Configuration API: CSV topic reference](../../foundry/topics/csv)