Adding a file - Atoti Hub

This page provides a description of how to load a new CSV file into a new datastore in Atoti CVA Risk Capital. It walks through every layer of the data load pipeline introduced after the migration to the DLC framework: a new store, a new publisher target, a new topic, and registration for either initial-load or on-demand loading. The techniques employed are generic examples that can be extended, adapted, and repeated for any use case.

Step 1 - move the data files

Place the new CSV files under an appropriate folder in cvarc-application-tests/src/test/resources/data-samples/data/{AsOfDate}/. This example uses two files placed under the 2018-12-05 folder. 123_Custom_1.csv

AsOfDate	RiskClass	CustomProjected	UselessData
2018-12-05	commodity	340000	x
2018-12-05	interest rate	2000	a

987_Custom_2.csv

AsOfDate	RiskClass	CustomProjected	UselessData
2018-12-05	foreign exchange	12341234	z
2018-12-05	equity	222222222	f

UselessData will be ignored — the topic’s parser column list (Step 3) determines which columns are read.

Step 2 - define the store and a reference

Add the new store and any references to existing stores in the addModifications() method of DatastoreCustomisationsConfig at cvarc-application/src/main/java/com/activeviam/cvarc/application/cfg/DatastoreCustomisationsConfig.java. This class lives in your project’s cvarc-application module and is intended to be edited directly.

@Override
public void addModifications(IDatastoreConfigurator configurator) {
    configurator.addStore(CVARCParameters.CVA_SA_SCHEMA_NAME, StartBuilding.store()
            .withStoreName("Custom")
            .withField("AsOfDate", LOCAL_DATE).asKeyField()
            .withField("RiskClass", STRING).asKeyField()
            .withField("CustomProjected", DOUBLE)
            .withValuePartitioningOn("AsOfDate")
            .build());

    configurator.addReference(CVARCParameters.CVA_SA_SCHEMA_NAME,
            configurator.referenceBuilder(CVARCParameters.CVA_SA_SCHEMA_NAME)
                    .fromStore(SAStoreNames.SA_BASE_STORE_NAME)
                    .toStore("Custom")
                    .withName("BaseStoreToCustom")
                    .withMapping(SAStoreNames.SA_BASE_AS_OF_DATE, "AsOfDate")
                    .withMapping(SAStoreNames.SA_BASE_RISK_CLASS, "RiskClass")
                    .build());
}

Step 3 - define a publisher target, a source, and a topic

Create a new @Configuration class in cvarc-application/src/main/java/com/activeviam/cvarc/application/cfg/extensions/. It declares three DLC beans:

a TargetDescription — the publisher that writes parsed rows into the new store;
a LocalCsvSourceDescription — the source that scans CSV files under a directory;
a CsvTopicDescription — the topic that ties a file pattern to a parser and to the publisher target.

Mirror SASourceDescription (cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/dlc/SASourceDescription.java) and TargetDescriptionsConfig (in the same package) for the canonical patterns.

package com.activeviam.cvarc.application.cfg.extensions;

import com.activeviam.database.datastore.api.IDatastore;
import com.activeviam.io.dlc.impl.description.source.LocalCsvSourceDescription;
import com.activeviam.io.dlc.impl.description.topic.CsvTopicDescription;
import com.activeviam.io.dlc.impl.description.topic.channel.ChannelDescription;
import com.activeviam.io.dlc.impl.description.topic.channel.publish.TargetDescription;
import com.activeviam.io.dlc.impl.description.topic.parser.CsvParserDescription;
import com.activeviam.io.dlc.impl.utils.NamedEntityResolverService;
import com.activeviam.source.common.api.impl.TuplePublisher;
import lombok.RequiredArgsConstructor;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

import java.util.List;
import java.util.Set;

@Configuration
@RequiredArgsConstructor
public class CustomFileSourceDescription {

    public static final String CSV_TOPIC__CUSTOM = "Custom";
    public static final String CUSTOM_PUBLISHER = "customPublisher";
    public static final String CUSTOM_SOURCE_NAME = "customSource";

    private final IDatastore datastore;

    @Bean
    public TargetDescription customPublisher() {
        return TargetDescription.of(CUSTOM_PUBLISHER, "Custom",
                scope -> new TuplePublisher<>(datastore, "Custom"));
    }

    @Bean
    public LocalCsvSourceDescription customSourceDescription() {
        return LocalCsvSourceDescription.builder(CUSTOM_SOURCE_NAME, "./src/test/resources/data-samples/data")
                .topicsToInclude(Set.of(CSV_TOPIC__CUSTOM))
                .build();
    }

    @Bean
    public CsvTopicDescription customTopic(NamedEntityResolverService namedEntityResolver) {
        return CsvTopicDescription.builder(CSV_TOPIC__CUSTOM, "glob:**/*Custom*.csv")
                .parser(CsvParserDescription.builder().columns(
                        List.of("AsOfDate", "RiskClass", "CustomProjected")
                ).build())
                .channel(ChannelDescription.builder(namedEntityResolver.getTarget(CUSTOM_PUBLISHER)).build())
                .build();
    }
}

The parser’s columns(...) list determines which columns are read. Columns present in the file but absent from the list (e.g. UselessData) are skipped.

Step 4 - wire the extension into ApplicationConfig

Add CustomFileSourceDescription to the @Import list of ApplicationConfig in cvarc-application/src/main/java/com/activeviam/cvarc/application/cfg/ApplicationConfig.java.

Step 5 - register the topic for loading

A topic that has been declared but never registered for loading does nothing. There are two common ways to register it:

Path A - initial-load via alias

AliasesDescriptionConfig (cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/dlc/AliasesDescriptionConfig.java) groups topics under named aliases that InitialDataLoadConfig then loads at startup. Subclass it to append your new topic to an existing alias (or define a new one):

package com.activeviam.cvarc.application.cfg.extensions;

import com.activeviam.cvarc.starter.cfg.impl.dlc.AliasesDescriptionConfig;
import com.activeviam.io.dlc.impl.description.AliasesDescription;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.context.annotation.Primary;

import java.util.Set;

import static com.activeviam.cvarc.application.cfg.extensions.CustomFileSourceDescription.CSV_TOPIC__CUSTOM;

@Configuration
@Primary
public class ExtendedAliasesDescriptionConfig extends AliasesDescriptionConfig {

    @Bean
    @Override
    public AliasesDescription aliasesDescription() {
        return super.aliasesDescription().toBuilder()
                .alias(CSV_TOPIC__ALIAS__SA_DATA, Set.of(CSV_TOPIC__CUSTOM))
                .build();
    }
}

super.aliasesDescription().toBuilder() inherits every alias the parent declares; the additional .alias(...) call merges in the new topic. Add ExtendedAliasesDescriptionConfig to the @Import list of ApplicationConfig.

Path B - on-demand via data load controller

For loads triggered by an event or REST call rather than at startup, declare an ApplicationRunner @Bean that calls the IDataLoadControllerService. Mirror InitialDataLoadConfig.initialDataLoad() (cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/dlc/InitialDataLoadConfig.java):

@Bean
public ApplicationRunner loadCustomData(IDataLoadControllerService dlc) {
    return args -> {
        final DlcScope scope = new DlcScope();
        scope.put(AS_OF_DATE_SCOPE_KEY, "2018-12-05");
        dlc.execute(DlcLoadRequest.builder()
                .topics(CSV_TOPIC__CUSTOM)
                .scope(scope)
                .build());
    };
}

The same dlc.execute(...) call can be invoked from any Spring component (e.g. a REST controller) to load data on demand.

Step 6 - expose the new fields

To expose the new store’s fields as hierarchies or measures, see Adding a Hierarchy and Adding New Cube Measures. Each cube’s selection must be extended to include a reference into the new store before the fields become reachable.

​Step 1 - move the data files

​Step 2 - define the store and a reference

​Step 3 - define a publisher target, a source, and a topic

​Step 4 - wire the extension into ApplicationConfig

​Step 5 - register the topic for loading

​Path A - initial-load via alias

​Path B - on-demand via data load controller

​Step 6 - expose the new fields

Step 1 - move the data files

Step 2 - define the store and a reference

Step 3 - define a publisher target, a source, and a topic

Step 4 - wire the extension into ApplicationConfig

Step 5 - register the topic for loading

Path A - initial-load via alias

Path B - on-demand via data load controller

Step 6 - expose the new fields