☰
Navigation :

Adding a File

This page provides a description of how to add new files to Atoti CVA Risk Capital in a new datastore. The techniques employed are generic examples that can be extended, adapted, and repeated for any use case that we need. In all cases, we make minimal changes to the Reference Implementation.

Step 1 - Moving the Data

Move any custom data to an appropriate folder in /cvarc-starter/src/test/resources/data-samples/data.
This example moves the data (shown below) into the /2018-12-05 folder.

123_Custom_1.csv

AsOfDate RiskClass CustomProjected UselessData
2018-12-05 commodity 340000 x
2018-12-05 interest rate 2000 a

987_Custom_2.csv

AsOfDate RiskClass CustomProjected UselessData
2018-12-05 foreign exchange 12341234 z
2018-12-05 equity 222222222 f

note

UselessData should be ignored.

Step 2 - Define Customisations

Append the Datastore Model Config by creating a class that extends the appropriate Datastore Model Config, in this case SADatastoreModelConfig:

  1. Within this class, define the name of the new store and the fields it will contain, by making a method that adds a store builder to the IDatastoreConfigurator.
  2. Add a store reference to the configurator by using the Reference builder.
  3. Call the method building our new store in createStores(IDatastoreConfigurator configurator) method.
@Configuration
public class ExtendedSADatastoreModelConfig extends SADatastoreModelConfig {
    // 1. This will be all the fields you wish to store in your cube for that particular store.
    public static void customStoreDescription(IDatastoreConfigurator configurator){
        configurator.addStore("Schema", StartBuilding.store()
                .withStoreName("Custom")
                .withField("AsOfDate", LOCAL_DATE).asKeyField()
                .withField("RiskClass", STRING).asKeyField()
                //asKeyField() denotes a field(s) that can uniquely identify data, key(s) should not resolve to multiple peices of data
                .withField("CustomProjected", DOUBLE)
                //You can have unlimited numbers of Fields by stringing .withField methods before the .build() method
                // We wont want to add any fields we want to ignore here
                .build());
    }
    // 2. appended reference list
    @Override
    public static void createReferences(IDatastoreConfigurator configurator){
        configurator
                .addReference("Schema", configurator.referenceBuilder(CVARCParameters.CVA_SA_SCHEMA_NAME)
                        .fromStore(SAStoreNames.SA_BASE_STORE_NAME)
                        .toStore(SAStoreNames.SA_CARVED_NETTING_SET_STORE_NAME)
                        .withName(SAStoreNames.SA_BASE_TO_CARVED_NETTING_SET_REF)
                        .withMapping(SAStoreNames.SA_BASE_AS_OF_DATE, SAStoreNames.SA_CARVED_NETTING_SET_AS_OF_DATE).build());

    }

    public static void createStores(IDatastoreConfigurator configurator) {
        customStoreDescription(configurator);
    }
}

Step 3 - Application Config

Replace SADatastoreModelConfig with ExtendedSADatastoreModelConfig so that your changes can be picked up in the ApplicationConfig class located in /cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/ApplicationConfig.java

@Configuration
@Import(value = {

        // data model
        BADatastoreModelConfig.class,
//      SADatastoreModelConfig.class,               // here we comment out our Datastore Model Config
        ExtendedSADatastoreModelConfig.class,       // and in its place write our extended one
        DatastoreConfig.class,
        OlapModelConfig.class,

Replace SADatastoreModelConfig with ExtendedSADatastoreModelConfig so that your changes can be picked up in DatastoreConfig class located in cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/DatastoreConfig.java

    @Configuration
    @Profile("!" + CVARCEnvConstants.SP_PROFILE__QUERY_NODE)
    @Import(value = {
            BADatastoreModelConfig.class,
//          SADatastoreModelConfig.class,           // here we comment out the original Datastore Model Config
            ExtendedSADatastoreModelConfig.class,   // and in its place write our extended one
    })

Step 4 - Source Config / ETL Code

Modify the appropriate csv source config. In this case SaCsvSourceConfig, by creating a new class inside of /cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/, which extends SaCsvSourceConfig.

  1. Define the csv fields of your file.
  2. Modify your bulkload and incremental load.
  3. Lastly, append the topic channels map.
public abstract class ExtendedSaCsvSourceConfig extends SaCsvSourceConfig {

    // 1. Define CSV fields
    protected List<String> generateCustomCsvFields(){
        List<String> customCvaFields = new ArrayList<>();
        customCvaFields.add("AsOfDate");
        customCvaFields.add("RiskClass");
        customCvaFields.add("CustomProjected");
        customCvaFields.add("UselessData");
        return customCvaFields;
    }

    // 2. defining topic for bulk load and incremental load
    @Override
    public CsvScopedFetchSource bulkLoadCsvSource(String sourceName) {
        final CsvScopedFetchSource<Path> source = super.bulkLoadCsvSource(sourceName);
        source.addTopic(
                createDirectoryTopic(
                        source, "Custom", "./src/test/resources/data-samples" , "/data",
                        "glob:**/*Custom*.csv", source.createParserConfiguration(generateCustomCsvFields(), 1)
            // the int 1 is passed into the createParserConfiguration method to denote that our topic should skip the first line of our source
                )
        );
        return source;
    }

    @Override
    public CsvScopedFetchSource incrLoadCsvSource(String sourceName) {
        final CsvScopedFetchSource<Path> source = super.incrLoadCsvSource(sourceName);
        source.addTopic(
                createDirectoryTopic(
                        source, "Incremental-Custom", "./src/test/resources/data-samples", "/data",
                        "glob:**/*Custom*.csv", source.createParserConfiguration(generateCustomCsvFields(), 1)
                )
        );
        return source;
    }

    // 3. Appending Topic Channel Map
    @Override
    public Map<String, Pair<List<String>, ITuplePublisher>> createTopicChannelsMap() {
        Map<String, Pair<List<String>, ITuplePublisher>> extendedTopicChannels = super.createTopicChannelsMap();

        extendedTopicChannels.put("Custom", new Pair(Arrays.asList("Custom"),
                new TuplePublisher<IFileInfo<Path>>(datastoreConfig.datastore(),
                        "Custom")));

        return extendedTopicChannels;
    }

}

Step 5 - Application Config

Replace SaCsvSourceConfig with ExtendedSaCsvSourceConfig, so that your changes can be picked up in ApplicationConfig located in cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/ApplicationConfig.java

        // data sourcing
        SharedCsvSourceConfig.class,
        SharedAzureCsvSourceConfig.class,
        SharedAwsCsvSourceConfig.class,
        SharedGoogleCsvSourceConfig.class,

//      SaCsvSourceConfig.class,                        // here we comment out the original Source Config
        ExtendedSaCsvSourceConfig.class,                // and replace with our Extended Source Config
        SaAzureCsvSourceConfig.class,
        SaAwsCsvSourceConfig.class,
        SaGoogleCsvSourceConfig.class,

Step 6 - Data Load Controller

Append the list of topics that get registered in DataLoadControllerConfig to include the new ‘Custom’ Topic so that your new data can be loaded into the Solution.

    //--SA data
            final List<String> saDataTopics = new ArrayList<>(
                    Arrays.asList(
                            saCsvSource.CSV_TOPIC__CVA_DELTA_SENSITIVITIES,
                            saCsvSource.CSV_TOPIC__HEDGED_DELTA_SENSITIVITIES,
                            saCsvSource.CSV_TOPIC__CVA_VEGA_SENSITIVITIES,
                            saCsvSource.CSV_TOPIC__HEDGED_VEGA_SENSITIVITIES,
                            saCsvSource.CSV_TOPIC__INTEREST_RATE_DELTA_RISK_WEIGHT,
                            saCsvSource.CSV_TOPIC__COUNTERPARTY_CREDIT_SPREAD_DELTA_RISK_WEIGHT,
                            saCsvSource.CSV_TOPIC__REFERENCE_CREDIT_DELTA_RISK_WEIGHT,
                            saCsvSource.CSV_TOPIC__EQUITY_DELTA_RISK_WEIGHT,
                            //jdbcSourceConfig.JDBC_TOPIC__EQUITY_DELTA_RISK_WEIGHT,//Register JDBC Topic
                            saCsvSource.CSV_TOPIC__COMMODITY_DELTA_RISK_WEIGHT,
                            saCsvSource.CSV_TOPIC__INTEREST_RATE_DELTA_CORRELATION,
                            saCsvSource.CSV_TOPIC__COUNTERPARTY_CREDIT_SPREAD_DELTA_CROSS_BUCKET_CORRELATION,
                            saCsvSource.CSV_TOPIC__REFERENCE_CREDIT_CROSS_BUCKET_CORRELATION,
                            saCsvSource.CSV_TOPIC__SA_CVA_VEGA_RISK_WEIGHT,
                            "Custom"                                                    // here we add one more element to the saDataTopics List Custom
                            ));

            controller.registerTopicAlias(TOPIC__SA_DATA, saDataTopics);
}

To expose fields as hierarchies or measures, see the examples in Adding a Hierarchy and Adding New Cube Measures.