☰
Navigation :

Adding a File

This page provides a description of how to add new files to the CVARC Accelerator in a new data store. The techniques employed are generic examples that can be extended, adapted and repeated for any use case that we need. In all cases, we make minimal changes to the Accelerator Reference Implementation.

Step 1 - Moving the Data

We want to move any custom data to an appropriate folder inside of /cvarc-starter/src/test/resources/data-samples/data for our example we will move our data (shown below) into /2018-12-05 folder.

123_Custom_1.csv

AsOfDate RiskClass CustomProjected UselessData
2018-12-05 commodity 340000 x
2018-12-05 interest rate 2000 a

987_Custom_2.csv

AsOfDate RiskClass CustomProjected UselessData
2018-12-05 foreign exchange 12341234 z
2018-12-05 equity 222222222 f

Keep in mind we want to ignore UselessData.

Step 2 - Define Customisations

Append the Datastore Model Config by creating a class which extends the appropriate Datastore Model Config, in this case SADatastoreModelConfig:

  1. Within this class define the name of the new store and the fields it will contain by making a method that returns a StoreDescription
  2. Append the Reference List by overriding the method which defines a Reference List for the appropriate cube (SA) in this case saReferences()
  3. Add the store to our store Description by overriding the appropriate method in this case saStoreDescription()
@Configuration
public class ExtendedSADatastoreModelConfig extends SADatastoreModelConfig {
    // 1. This will be all the fields you wish to store in your cube for that particular store.
    public IStoreDescription customStoreDescription(){
        return new StoreDescriptionBuilder()
                .withStoreName("Custom")
                .withField("AsOfDate", LOCAL_DATE).asKeyField()
                .withField("RiskClass", STRING).asKeyField()
                //asKeyField() denotes a field(s) that can uniquely identify data, key(s) should not resolve to multiple peices of data
                .withField("CustomProjected", DOUBLE)
                //You can have unlimited numbers of Fields by stringing .withField methods before the .build() method
                // We wont want to add any fields we want to ignore here
                .build();
    }
    // 2. appended reference list
    @Override
    public List<IReferenceDescription> saReferences() {
        List<IReferenceDescription> references = super.saReferences();

        references.add(ReferenceDescription.builder()
                .fromStore("Vega")
                .toStore("Custom")
                .withName("VegaToCustom")
                .withMapping("RiskClass", "RiskClass")
                //You can have unlimited numbers of Mappings by stringing .withMapping methods.
                .build());
        //You can add as many references as you like by adding them to the reference List
    return references;
    }
    // 3. appended store description
    @Override
    public List<IStoreDescription> saStoreDescription() {
        List<IStoreDescription> description = super.saStoreDescription();
        description.add(customStoreDescription());
        return description;
    }
}

Step 3 - Application Config

Replace SADatastoreModelConfig with ExtendedSADatastoreModelConfig so that our changes can be picked up in ApplicationConfig class located in /cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/ApplicationConfig.java

@Configuration
@Import(value = {

        // data model
        BADatastoreModelConfig.class,
//      SADatastoreModelConfig.class,               // here we comment out our Datastore Model Config
        ExtendedSADatastoreModelConfig.class,       // and in its place write our extended one
        DatastoreConfig.class,
        OlapModelConfig.class,

Replace SADatastoreModelConfig with ExtendedSADatastoreModelConfig so that our changes can be picked up in DatastoreConfig class located in cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/DatastoreConfig.java

    @Configuration
    @Profile("!" + CVARCEnvConstants.SP_PROFILE__QUERY_NODE)
    @Import(value = {
            BADatastoreModelConfig.class,
//          SADatastoreModelConfig.class,           // here we comment out the original Datastore Model Config
            ExtendedSADatastoreModelConfig.class,   // and in its place write our extended one
    })

Step 4 - Source Config / ETL Code

Modify the appropriate csv source config in this case SaCsvSourceConfig by creating a new class inside of /cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/ which extends SaCsvSourceConfig.

  1. Define the csv fields of our file
  2. Modify our bulkload and incremental load
  3. Lastly we need to append the topic channels map
public abstract class ExtendedSaCsvSourceConfig extends SaCsvSourceConfig {

    // 1. Define CSV fields
    protected List<String> generateCustomCsvFields(){
        List<String> customCvaFields = new ArrayList<>();
        customCvaFields.add("AsOfDate");
        customCvaFields.add("RiskClass");
        customCvaFields.add("CustomProjected");
        customCvaFields.add("UselessData");
        return customCvaFields;
    }

    // 2. defining topic for bulk load and incremental load
    @Override
    public CsvScopedFetchSource bulkLoadCsvSource(String sourceName) {
        final CsvScopedFetchSource<Path> source = super.bulkLoadCsvSource(sourceName);
        source.addTopic(
                createDirectoryTopic(
                        source, "Custom", "./src/test/resources/data-samples" , "/data",
                        "glob:**/*Custom*.csv", source.createParserConfiguration(generateCustomCsvFields(), 1)
            // the int 1 is passed into the createParserConfiguration method to denote that our topic should skip the first line of our source
                )
        );
        return source;
    }

    @Override
    public CsvScopedFetchSource incrLoadCsvSource(String sourceName) {
        final CsvScopedFetchSource<Path> source = super.incrLoadCsvSource(sourceName);
        source.addTopic(
                createDirectoryTopic(
                        source, "Incremental-Custom", "./src/test/resources/data-samples", "/data",
                        "glob:**/*Custom*.csv", source.createParserConfiguration(generateCustomCsvFields(), 1)
                )
        );
        return source;
    }

    // 3. Appending Topic Channel Map
    @Override
    public Map<String, Pair<List<String>, ITuplePublisher>> createTopicChannelsMap() {
        Map<String, Pair<List<String>, ITuplePublisher>> extendedTopicChannels = super.createTopicChannelsMap();

        extendedTopicChannels.put("Custom", new Pair(Arrays.asList("Custom"),
                new TuplePublisher<IFileInfo<Path>>(datastoreConfig.datastore(),
                        "Custom")));

        return extendedTopicChannels;
    }

}

Step 5 - Application Config

Replace SaCsvSourceConfig with ExtendedSaCsvSourceConfig so that our changes can be picked up in ApplicationConfig located in cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/ApplicationConfig.java

        // data sourcing
        SharedCsvSourceConfig.class,
        SharedAzureCsvSourceConfig.class,
        SharedAwsCsvSourceConfig.class,
        SharedGoogleCsvSourceConfig.class,

//      SaCsvSourceConfig.class,                        // here we comment out the original Source Config
        ExtendedSaCsvSourceConfig.class,                // and replace with our Extended Source Config
        SaAzureCsvSourceConfig.class,
        SaAwsCsvSourceConfig.class,
        SaGoogleCsvSourceConfig.class,

Step 6 - Data Load Controller

Append the list of topics which get registered inside of DataLoadControllerConfig to include our new ‘Custom’ Topic so that our new data can be loaded into our accelerator

    //--SA data
            final List<String> saDataTopics = new ArrayList<>(
                    Arrays.asList(
                            saCsvSource.CSV_TOPIC__CVA_DELTA_SENSITIVITIES,
                            saCsvSource.CSV_TOPIC__HEDGED_DELTA_SENSITIVITIES,
                            saCsvSource.CSV_TOPIC__CVA_VEGA_SENSITIVITIES,
                            saCsvSource.CSV_TOPIC__HEDGED_VEGA_SENSITIVITIES,
                            saCsvSource.CSV_TOPIC__INTEREST_RATE_DELTA_RISK_WEIGHT,
                            saCsvSource.CSV_TOPIC__COUNTERPARTY_CREDIT_SPREAD_DELTA_RISK_WEIGHT,
                            saCsvSource.CSV_TOPIC__REFERENCE_CREDIT_DELTA_RISK_WEIGHT,
                            saCsvSource.CSV_TOPIC__EQUITY_DELTA_RISK_WEIGHT,
                            //jdbcSourceConfig.JDBC_TOPIC__EQUITY_DELTA_RISK_WEIGHT,//Register JDBC Topic
                            saCsvSource.CSV_TOPIC__COMMODITY_DELTA_RISK_WEIGHT,
                            saCsvSource.CSV_TOPIC__INTEREST_RATE_DELTA_CORRELATION,
                            saCsvSource.CSV_TOPIC__COUNTERPARTY_CREDIT_SPREAD_DELTA_CROSS_BUCKET_CORRELATION,
                            saCsvSource.CSV_TOPIC__REFERENCE_CREDIT_CROSS_BUCKET_CORRELATION,
                            saCsvSource.CSV_TOPIC__SA_CVA_VEGA_RISK_WEIGHT,
                            "Custom"                                                    // here we add one more element to the saDataTopics List Custom
                            ));

            controller.registerTopicAlias(TOPIC__SA_DATA, saDataTopics);
}

To Expose fields as Hierarchies or Measures please see example in Adding a Hierarchy and Adding a new Measure.

search.js