Adding New Data Loading or Unloading Topics
This page provides a description of how to load a new file into a new store within the FRTB Accelerator. The techniques employed are generic examples that can be extended, adapted and repeated for any use case that we need. In all cases, we make minimal changes to the Accelerator Reference Implementation.
Step 1 - Move data to the relevant directory
For this example, we want to bring any custom data to an appropriate folder inside of:
/frtb-starter/src/test/resources/data
within the /2018-09-26
folder
and name our file Custom.csv
AsOfDate | RiskClass | CustomProjected |
---|---|---|
2018-09-26 | Commodity | -6640.633693 |
2018-09-26 | GIRR | 14020.37649 |
2018-09-26 | CSR Sec CTP | 8386.767854 |
2018-09-26 | CSR Sec non-CTP | 19218.3336 |
2018-09-26 | Commodity | -2460.048584 |
2018-09-26 | Equity | 8274.302903 |
2018-09-26 | FX | 2150.845785 |
2018-09-26 | GIRR | 17537.8908 |
2018-09-26 | CSR non-Sec | -25353.39868 |
2018-09-26 | DRC Sec non-CTP | 11319.08548 |
2018-09-26 | FX | 25977.18728 |
2018-09-26 | Commodity | 11714.89133 |
2018-09-26 | Equity | -19844.11309 |
2018-09-26 | FX | 8906.302165 |
2018-09-26 | GIRR | 19617.16455 |
2018-09-26 | DRC non-Sec | 17134.37517 |
Step 2 - Define the datastore we want to populate
To add a new datastore, we will need to navigate to the
addModifications()
method inside of the
DatastoreCustomisationsConfig
class.
frtb-starter/src/main/java/com/activeviam/frtb/starter/cfg/impl/
// 1. This will be all the fields you wish to store in your cube for that particular store.
IStoreDescription storeDescription = new StoreDescriptionBuilder()
.withStoreName(CUSTOM_STORE)
.withField(AS_OF_DATE, ILiteralType.LOCAL_DATE).asKeyField()
.withField(RISK_CLASS, ILiteralType.STRING).asKeyField()
.withField("CustomProjected", ILiteralType.DOUBLE)
// You can customize your datastore with any number of fields, as well as define partitioning, duplicate key handlers, and indexing configurations
.build();
configurator.addStore(FRTBConstants.FRTB_SCHEMA, storeDescription);
Here, we can also add if needed:
IReferenceDescription referenceDescription = configurator.referenceBuilder(FRTBConstants.FRTB_SCHEMA)
.fromStore(SADatastoreConfig.TRADE_BASE_STORE_NAME)
.toStore(CUSTOM_STORE)
.withName(SADatastoreConfig.TRADE_BASE_STORE_NAME + "To" + CUSTOM_STORE)
.withMapping(SADatastoreConfig.TRADE_BASE_STORE_RISK_CLASS, SADatastoreConfig.TRADE_BASE_STORE_RISK_CLASS)
.withMapping(SADatastoreConfig.AS_OF_DATE, SADatastoreConfig.AS_OF_DATE)
.build();
configurator.addReference(FRTBConstants.FRTB_SCHEMA, referenceDescription);
See DatastoreHelper for more information.
Step 3 - Create channel parameters for our data loading topic
The channel parameters for our store include store name, topic name, and file pattern.
Ensure that a file pattern property is defined within
frtb-starter/src/main/resources/frtb-data-load.properties
custom.file-pattern=**/*Custom*.csv
To add channel parameters, we will have to extend the SourceConfig
class.
public class ExtendedSourceConfig extends SourceConfig {
public static final String FILE_PATTERN_PROP = "custom.file-pattern";
public static final String TOPIC_CUSTOM = "Custom_TOPIC";
@Bean
@DependsOn({"startManager"})
public java.util.List<ChannelParameters> sourceConfigStores() {
List<ChannelParameters> channelParametersList = super.sourceConfigStores();
channelParametersList.add(new ChannelParameters(CUSTOM_STORE, TOPIC_CUSTOM, FILE_PATTERN_PROP));
return channelParametersList;
}
}
Step 4 - Ensure that FRTBConfig takes our extended class
Navigate to the application config class
FRTBConfig/frtb-starter/src/main/java/com/activeviam/frtb/starter/cfg/impl/
Replace SourceConfig
with ExtendSourceConfig
@Configuration
@Import(value = {
...
// SourceConfig.class,
ExtendedSourceConfig.class,
...
})
Step 5 - Ensure that our topic is included within InitiaDataLoadConfig
InitialDataLoadConfig
orchestrates our Topics' execution.
public Void initialDataLoad() throws IOException {
//...
//In this case it makes sense to include a scope for our files.
for(final LocalDate date: initialLoadAsOfDates()){
final Map<String, Object> fetchScope = new HashMap<>();
fetchScope.put(DataLoadControllerConfig.SCOPE__AS_OF_DATE, date.toString());
controller.execute(new DataLoadControllerRequest(FRTBLoadDataTxControllerTask.PLUGIN_KEY,
Arrays.asList(ExtendedSourceConfig.TOPIC_CUSTOM), fetchScope));
//...
}
}
Suggested further reading
Enriching File Fields by Adding Column Calculators, Adding New Cube Hierarchies, Adding New Cube Measures, and Add and Load a New Column to Existing File.