Data loading

​

How does data loading work?

• What is loaded: the channels (target stores) and topics (CSV sources) declared by the solution’s default loading and any customer override. See What declares the channels and topics?.
• When it runs: usually an initial load on startup, plus a runtime REST API for individual as-of dates. See How is data loaded on startup? and How is data loaded at runtime?.

​

What declares the channels and topics?

• I[Cube name]CubeChannels (e.g. IVaRCubeChannels) returns the channel names, plus the names of the post-load triggers the cube declares.
• I[Cube name]CubeTopics (e.g. IVaRCubeTopics) returns the topic names, plus the aliases grouping them (see How does an application request a load?).

@Bean
IDataLoadConfigurer dataLoadConfigurer(final DataLoadProperties props) {
  return configurator -> {
    final var table = "Positions";
    configurator.channels().addChannel(table, table);
    configurator
        .topics()
        .addTopic(table, ILocalCsvTopicConfigurator.class)
        // base-path carries the #{AS_OF_DATE} placeholder, so the pattern stays date-agnostic
        .path(props.getBasePath())
        .filePattern("glob:positions/*.csv")
        .parsersFromTable(table) // drive parsing from the target table's field types
        .attachChannel(table);
  };
}

• addChannel creates a channel targeting the given table. attachChannel attaches a topic to a channel by name. parsersFromTable registers parsers driven by the target table’s field types, so plain, header-based files need no manual column mapping.
• Each load request targets one as-of date. When base-path contains the #{AS_OF_DATE} placeholder, as it does in a generated application, the date is substituted into it, and the topic’s date-agnostic file pattern (positions/*.csv) resolves under that date’s folder. Without the placeholder, every request reads the files directly under base-path, whatever the date.
• A cloud topic uses the same code, but with IAwsCsvTopicConfigurator (or the Azure/GCP equivalent) and a .bucket(...) call. A solution declares one mutually exclusive bean per topic-type value it ships topics for, plus one declaring its channels alone for the values it does not, so your own topics for those still have channels to attach to.

@Bean
IDataLoadConfigurer markToFutureOverride(final IPfeCubeTopics topics) {
  // Fetches the topic the solution's default loading already declared and repoints it; this runs
  // after those defaults, against the same IDataLoadConfigurator. The name comes from the cube's
  // topics API, so an override cannot drift from what the solution declared.
  return configurator ->
      configurator
          .topics()
          .topic(topics.markToFuture(), ICsvTopicConfigurator.class)
          .filePattern("glob:markToFuture/corrected/*.csv");
}

Note: an application declares at most one IDataLoadConfigurer bean. It is injected as a single bean by type, so a second bean of that type fails startup with NoUniqueBeanDefinitionException. The two examples above show two things that one bean can do: declare a new channel and topic, and fetch and repoint an existing one. They are not two separate beans.

Tip: name each topic after the table it loads. The data-loading API builds an implicit topic for every requested name. A name that does not match a datastore table still works. It produces a harmless, noisy Implicit topic for table '…' cannot be built warning at load time.

​

What happens when solutions share a table?

@Bean
IDataLoadConfigurer sharedTableChannels(
    final IPfeCube cube,
    final IPfeCubeTopics topics,
    final DataLoadProperties props,
    final ITuplePublisher<Object> myPublisher) {
  return configurator -> {
    // Fills a table of its own, so it can share the topic the solution already declared: the file
    // stays read once and its rows reach both tables.
    configurator.channels().addChannel("MY_TRADE_ATTRIBUTES", "MyTradeAttributes");
    configurator
        .topics()
        // ITopicConfigurator, not the local type: the solution declares this topic as the AWS one
        // when topic-type is AWS, and topic() rejects a type the configurator is not.
        .topic(topics.tradeAttributes(), ITopicConfigurator.class)
        .attachChannel("MY_TRADE_ATTRIBUTES");
    // Fills the same table as the solution's channel, through a publisher of its own, so it needs
    // a topic of its own over the same files.
    final var simulationDates = cube.databaseModel().simulationDates().name();
    configurator
        .channels()
        .addChannel("MY_SIMULATION_DATES", simulationDates)
        .tuplePublisher(myPublisher);
    configurator
        .topics()
        .addTopic("MY_SIMULATION_DATES", ILocalCsvTopicConfigurator.class)
        .path(props.getBasePath())
        .filePattern("glob:{,**/}SimulationDates{,_*}.csv")
        .parsersFromTable(simulationDates)
        .attachChannel("MY_SIMULATION_DATES");
  };
}

Note: requesting a load belongs to the application rather than to the starter. A starter carries the cube, its channels, and its topics; the application decides when they are read. An application that opts out starts normally and serves its cube, but loads nothing, because no load is ever requested.

​

How does an application request a load?

@SpringBootApplication
@Import(AsOfDateDataLoadConfiguration.class)
public class MyApplication { ... }

​

What does a load cover?

atoti:
  solution:
    data-load:
      as-of-date:
        topics:
          - PFE_LOAD
          - VAR_LOAD
        unload-topics:
          - PFE_UNLOAD
          - VAR_UNLOAD

​

How is data loaded on startup?

atoti:
  solution:
    data-load:
      as-of-date:
        topics:
          - PFE_LOAD
        unload-topics:
          - PFE_UNLOAD
        initial-dates:
          enabled: true
          dates:
            - 2024-01-15
            - 2024-01-16

​

How is data loaded at runtime?

• POST /activeviam/as-of-date/rest/v1/{asOfDate} loads data for the given as-of date. The asOfDate path variable must be in yyyy-MM-dd format.
• DELETE /activeviam/as-of-date/rest/v1/{asOfDate} unloads the data previously loaded for that as-of date. It removes that date’s rows from every table scoped on the as-of date. A table not scoped on the as-of date, such as a market-data curve table, keeps its rows. Each solution’s data-loading page lists which tables lose rows and which keep them.

​

How is the data source configured?

​

Local filesystem (default)

atoti:
  solution:
    data-load:
      csv:
        topic-type: LOCAL
        base-path: /path/to/data/#{AS_OF_DATE}   # #{AS_OF_DATE} is substituted with each loaded date

​

AWS S3

1. Add the AWS cloud-source dependency to the application module:
   <dependency>
     <groupId>com.activeviam.source</groupId>
     <artifactId>cloud-source-aws</artifactId>
     <version>${atoti-server.version}</version>
   </dependency>
   
2. Point the configuration at the bucket and create the S3 client:
   # Creates the default S3Client bean (standard AWS credential/region chain).
   dlc:
     create-default-cloud-client:
       aws: true
   atoti:
     solution:
       data-load:
         csv:
           topic-type: AWS
           base-path: my/in-bucket/prefix/#{AS_OF_DATE}
           container: my-bucket-name
   
3. Provide AWS credentials and a region. Both are resolved through the standard AWS SDK chain (environment variables, ~/.aws/credentials and ~/.aws/config, instance profiles, …).

Note: dlc.create-default-cloud-client.aws: true is mandatory. The AWS source connector only registers when an S3Client bean is present. Without it, the application starts but silently loads nothing: the topics are declared but never fetched.

Note: the default S3Client is built without an explicit region, so a region must be resolvable from the SDK chain (for example the AWS_REGION environment variable). If none is found, the application fails to start with Unable to load region from any of the providers.

​

What file formats are expected?

​

How must CSV files be named?

​

Related pages

• Input file formats
• Configuration properties
• Database model
• Atoti Configuration API: Data loading reference
• Atoti Configuration API: CSV topic reference