Adding New Data Loading or Unloading Topics

Step 1 - Move data to the relevant directory

For this example, we want to bring any custom data to an appropriate folder inside of:

/frtb-starter/src/test/resources/data within the /2018-09-26 folder and name our file Custom.csv

AsOfDate	RiskClass	CustomProjected
2018-09-26	Commodity	-6640.633693
2018-09-26	GIRR	14020.37649
2018-09-26	CSR Sec CTP	8386.767854
2018-09-26	CSR Sec non-CTP	19218.3336
2018-09-26	Commodity	-2460.048584
2018-09-26	Equity	8274.302903
2018-09-26	FX	2150.845785
2018-09-26	GIRR	17537.8908
2018-09-26	CSR non-Sec	-25353.39868
2018-09-26	DRC Sec non-CTP	11319.08548
2018-09-26	FX	25977.18728
2018-09-26	Commodity	11714.89133
2018-09-26	Equity	-19844.11309
2018-09-26	FX	8906.302165
2018-09-26	GIRR	19617.16455
2018-09-26	DRC non-Sec	17134.37517

Step 2 - Define the datastore we want to populate

To add a new datastore, we will need to navigate to the addModifications() method inside of the DatastoreCustomisationsConfig class.

frtb-starter/src/main/java/com/activeviam/frtb/starter/cfg/impl/

 // 1. This will be all the fields you wish to store in your cube for that particular store.
IStoreDescription storeDescription = new StoreDescriptionBuilder()
	.withStoreName(CUSTOM_STORE)
	.withField(AS_OF_DATE, ILiteralType.LOCAL_DATE).asKeyField()
	.withField(RISK_CLASS, ILiteralType.STRING).asKeyField()
	.withField("CustomProjected", ILiteralType.DOUBLE)
	// You can customize your datastore with any number of fields, as well as define partitioning, duplicate key handlers, and indexing configurations
	.build();

	configurator.addStore(FRTBConstants.FRTB_SCHEMA, storeDescription);

Here, we can also add if needed:

 IReferenceDescription referenceDescription = configurator.referenceBuilder(FRTBConstants.FRTB_SCHEMA)
	.fromStore(SADatastoreConfig.TRADE_BASE_STORE_NAME)
	.toStore(CUSTOM_STORE)
	.withName(SADatastoreConfig.TRADE_BASE_STORE_NAME + "To" + CUSTOM_STORE)
	.withMapping(SADatastoreConfig.TRADE_BASE_STORE_RISK_CLASS, SADatastoreConfig.TRADE_BASE_STORE_RISK_CLASS)
	.withMapping(SADatastoreConfig.AS_OF_DATE, SADatastoreConfig.AS_OF_DATE)
	.build();

	configurator.addReference(FRTBConstants.FRTB_SCHEMA, referenceDescription);

See DatastoreHelper for more information.

Step 3 - Create channel parameters for our data loading topic

The channel parameters for our store include store name, topic name, and file pattern.

Ensure that a file pattern property is defined within frtb-starter/src/main/resources/frtb-data-load.properties

custom.file-pattern=**/*Custom*.csv

To add channel parameters, we will have to extend the SourceConfig class.

public class ExtendedSourceConfig  extends SourceConfig {

	public static final String FILE_PATTERN_PROP = "custom.file-pattern";
	public static final String TOPIC_CUSTOM = "Custom_TOPIC";

	@Bean
	@DependsOn({"startManager"})
	public java.util.List<ChannelParameters> sourceConfigStores() {

		List<ChannelParameters> channelParametersList = super.sourceConfigStores();
		channelParametersList.add(new ChannelParameters(CUSTOM_STORE, TOPIC_CUSTOM, FILE_PATTERN_PROP));
		return channelParametersList;

	}
}

Step 4 - Ensure that FRTBConfig takes our extended class

Navigate to the application config class FRTBConfig/frtb-starter/src/main/java/com/activeviam/frtb/starter/cfg/impl/

Replace SourceConfig with ExtendSourceConfig

@Configuration  
@Import(value = {  
...  
	// SourceConfig.class,
	ExtendedSourceConfig.class,
...  
})

Step 5 - Ensure that our topic is included within InitiaDataLoadConfig

InitialDataLoadConfig orchestrates our Topics' execution.

public Void initialDataLoad() throws IOException {
	//...
		//In this case it makes sense to include a scope for our files.
		for(final LocalDate date: initialLoadAsOfDates()){

			final Map<String, Object> fetchScope = new HashMap<>();
			fetchScope.put(DataLoadControllerConfig.SCOPE__AS_OF_DATE, date.toString());
			controller.execute(new DataLoadControllerRequest(FRTBLoadDataTxControllerTask.PLUGIN_KEY,
			Arrays.asList(ExtendedSourceConfig.TOPIC_CUSTOM), fetchScope));
	//...
}
}