Adding New Data Loading or Unloading Topics

This page provides a description of how to load a new file into a new store within the FRTB Accelerator. The techniques employed are generic examples that can be extended, adapted and repeated for any use case that we need. In all cases, we make minimal changes to the Accelerator Reference Implementation.

 

Step 1 - Move data to the relevant directory

For this example, we want to bring any custom data to an appropriate folder inside of:

/frtb-starter/src/test/resources/data within the /2018-09-26 folder and name our file Custom.csv

AsOfDate	RiskClass	CustomProjected
2018-09-26	Commodity	-6640.633693
2018-09-26	GIRR	14020.37649
2018-09-26	CSR Sec CTP	8386.767854
2018-09-26	CSR Sec non-CTP	19218.3336
2018-09-26	Commodity	-2460.048584
2018-09-26	Equity	8274.302903
2018-09-26	FX	2150.845785
2018-09-26	GIRR	17537.8908
2018-09-26	CSR non-Sec	-25353.39868
2018-09-26	DRC Sec non-CTP	11319.08548
2018-09-26	FX	25977.18728
2018-09-26	Commodity	11714.89133
2018-09-26	Equity	-19844.11309
2018-09-26	FX	8906.302165
2018-09-26	GIRR	19617.16455
2018-09-26	DRC non-Sec	17134.37517

Step 2 - Define the datastore we want to populate

To add a new datastore, we will need to navigate to the addModifications() method inside of the DatastoreCustomisationsConfig class.

frtb-starter/src/main/java/com/activeviam/frtb/starter/cfg/impl/

 // 1. This will be all the fields you wish to store in your cube for that particular store.
IStoreDescription storeDescription = new StoreDescriptionBuilder()
    .withStoreName(CUSTOM_STORE)
    .withField(AS_OF_DATE, ILiteralType.LOCAL_DATE).asKeyField()
    .withField(RISK_CLASS, ILiteralType.STRING).asKeyField()
    .withField("CustomProjected", ILiteralType.DOUBLE)
    // You can customize your datastore with any number of fields, as well as define partitioning, duplicate key handlers, and indexing configurations
    .build();

    configurator.addStore(FRTBConstants.FRTB_SCHEMA, storeDescription);

Here, we can also add if needed:

 IReferenceDescription referenceDescription = configurator.referenceBuilder(FRTBConstants.FRTB_SCHEMA)
    .fromStore(SADatastoreConfig.TRADE_BASE_STORE_NAME)
    .toStore(CUSTOM_STORE)
    .withName(SADatastoreConfig.TRADE_BASE_STORE_NAME + "To" + CUSTOM_STORE)
    .withMapping(SADatastoreConfig.TRADE_BASE_STORE_RISK_CLASS, SADatastoreConfig.TRADE_BASE_STORE_RISK_CLASS)
    .withMapping(SADatastoreConfig.AS_OF_DATE, SADatastoreConfig.AS_OF_DATE)
    .build();

    configurator.addReference(FRTBConstants.FRTB_SCHEMA, referenceDescription);

See DatastoreHelper for more information.

Step 3 - Create channel parameters for our data loading topic

The channel parameters for our store include store name, topic name, and file pattern.

Ensure that a file pattern property is defined within frtb-starter/src/main/resources/frtb-data-load.properties

custom.file-pattern=**/*Custom*.csv

To add channel parameters, we will have to extend the SourceConfig class.

public class ExtendedSourceConfig  extends SourceConfig {

    public static final String FILE_PATTERN_PROP = "custom.file-pattern";
    public static final String TOPIC_CUSTOM = "Custom_TOPIC";

    @Bean
    @DependsOn({"startManager"})
    public java.util.List<ChannelParameters> sourceConfigStores() {

        List<ChannelParameters> channelParametersList = super.sourceConfigStores();
        channelParametersList.add(new ChannelParameters(CUSTOM_STORE, TOPIC_CUSTOM, FILE_PATTERN_PROP));
        return channelParametersList;

    }
}

Step 4 - Ensure that FRTBConfig takes our extended class

Navigate to the application config class FRTBConfig/frtb-starter/src/main/java/com/activeviam/frtb/starter/cfg/impl/

Replace SourceConfig with ExtendSourceConfig

@Configuration  
@Import(value = {  
...  
    // SourceConfig.class,
    ExtendedSourceConfig.class,
...  
})                      

 

Step 5 - Ensure that our topic is included within InitiaDataLoadConfig

InitialDataLoadConfig orchestrates our Topics' execution.

public Void initialDataLoad() throws IOException {
    //...
        //In this case it makes sense to include a scope for our files.
        for(final LocalDate date: initialLoadAsOfDates()){

            final Map<String, Object> fetchScope = new HashMap<>();
            fetchScope.put(DataLoadControllerConfig.SCOPE__AS_OF_DATE, date.toString());
            controller.execute(new DataLoadControllerRequest(FRTBLoadDataTxControllerTask.PLUGIN_KEY,
            Arrays.asList(ExtendedSourceConfig.TOPIC_CUSTOM), fetchScope));
    //...
}
}

Suggested further reading

Enriching File Fields by Adding Column Calculators, Adding New Cube Hierarchies, Adding New Cube Measures, and Add and Load a New Column to Existing File.