☰
Navigation :

Adding New Data Loading or Unloading Topics

This page provides a description of how to load a new file into a new store within the FRTB Accelerator. The techniques employed are generic examples that can be extended, adapted and repeated for any use case that are needed. In all cases, we make minimal changes to the Accelerator Reference Implementation.

 

Step 1 - Move data to the relevant directory

In this step, we want to bring any custom data into an appropriate folder inside:

/frtb-starter/src/test/resources/data/

For this example we will be bringing our file named Custom.csv within the 2018-09-26 folder and setting up our topic configuration so that we can load the same format file for each AsOfDate. Below is a sample file for reference.

AsOfDate RiskClass CustomProjected
2018-09-26 Commodity -6640.633693
2018-09-26 GIRR 14020.37649
2018-09-26 CSR Sec CTP 8386.767854
2018-09-26 CSR Sec non-CTP 19218.3336
2018-09-26 Equity 8274.302903
2018-09-26 FX 2150.845785
2018-09-26 CSR non-Sec -25353.39868
2018-09-26 DRC Sec non-CTP 11319.08548
2018-09-26 DRC non-Sec 17134.37517

Step 2 - Create the datastore definition we want to populate

To add a new datastore, we will need to navigate to the addModifications() method inside of the DatastoreCustomisationsConfig class.

frtb-starter/src/main/java/com/activeviam/frtb/starter/cfg/impl/

In this method we can append the following to create our datastore description.

	@Override
	public void addModifications(IDatastoreConfigurator configurator) {
		mergingDuplicateKeyHandlerCustomisations(configurator);
		customizationForDrc(configurator);
		if (extraDatastoreConfigs != null) {
			extraDatastoreConfigs.forEach(config -> config.accept(configurator));
		}
		// Here we add our modification
		// In this method we can append the following to create our datastore description.
		IStoreDescription storeDescription = StoreDescription.builder()
				.withStoreName(CUSTOM_STORE)
				.withField(StoreFieldNames.AS_OF_DATE, ILiteralType.LOCAL_DATE).asKeyField()
				.withField(StoreFieldNames.RISK_CLASS, ILiteralType.STRING).asKeyField()
				.withField("CustomProjected", ILiteralType.DOUBLE)
				// You can customize your datastore with any number of fields, as well as define partitioning, duplicate key handlers, and indexing configurations
				.build();

		configurator.addStore(FRTBConstants.FRTB_SCHEMA, storeDescription);

		// Here, we can also add a reference if needed:
		IReferenceDescription referenceDescription = configurator.referenceBuilder(FRTBConstants.FRTB_SCHEMA)
				.fromStore(SADatastoreConfig.TRADE_BASE_STORE_NAME)
				.toStore(CUSTOM_STORE)
				.withName(SADatastoreConfig.TRADE_BASE_STORE_NAME + "To" + CUSTOM_STORE)
				.withMapping(SADatastoreConfig.TRADE_BASE_STORE_RISK_CLASS, SADatastoreConfig.TRADE_BASE_STORE_RISK_CLASS)
				.withMapping(SADatastoreConfig.AS_OF_DATE, SADatastoreConfig.AS_OF_DATE)
				.build();

		configurator.addReference(FRTBConstants.FRTB_SCHEMA, referenceDescription);
	}

If we want to organize our customizations into a separate class for instance ClientCustomisations , we can create a new class as outlined here.

See Datastore Helper documentation for more information on potential datastore modifications.

Step 3 - Create ChannelParameters for our data loading topic

  1. Create a class with a type parameter <I> and which autowires the desired source configuration class
  2. Create the ChannelParameters bean which includes the store name, topic name, and file pattern. We must qualify our bean with a unique qualifier. To wire ChannelParameters bean properly, we must use the appropriate ChannelParameters Bean Annotation. For more information, see ChannelParameters Bean Annotations.
  3. Ensure that a file pattern property is defined within frtb-starter/src/main/resources/frtb-data-load.properties as custom.file-pattern=**/*Custom*.csv. Wildcards can be appended as needed, for instance to be able to include Custom_1.csv.
public class CustomChannelParameters<I> {

	@Autowired
	ASourceConfig<I> sourceConfig;

	private static final String CUSTOM_STORE = "CustomStore";
	public static final String FILE_PATTERN_PROP = "custom.file-pattern";
	public static final String TOPIC_CUSTOM = "Custom_TOPIC";
    public static final String SP_QUALIFIER__CUSTOM_STORE_CHANNEL_PARAMETER = "customStoreChannelParameter";

	@SourceConfigChannelParametersBean
	@Qualifier(SP_QUALIFIER__CUSTOM_CHANNEL_PARAMETER)
	public ACSVSourceConfig.ChannelParameters customStoreChannelParameter() {
		return sourceConfig.new ChannelParameters(TOPIC_CUSTOM, CUSTOM_STORE, FILE_PATTERN_PROP);
	}
}

Step 4 - Include our new topic in initialDataLoad

Find InitialDataLoadConfig in frtb-starter/src/main/java/com/activeviam/frtb/starter/cfg/impl/ and modify initialDataLoad method to include our topic as desired. In this example, we want to load our new file for each asOfDate.

public class InitialDataLoadConfig {
	@Bean
	@DependsOn(value = { "startManager" })
	public Void initialDataLoad(
			@Qualifier("initialLoadAsOfDates") List<LocalDate> initialLoadAsOfDates,
			@Qualifier("initialLoadHistory") boolean initialLoadHistory,
			@Qualifier("initialLoadConfiguration") boolean initialLoadConfiguration
	) throws IOException {
		/* ... */
		for (final LocalDate date : initialLoadAsOfDates) {
			/* ... */
			controller.execute(
					new DataLoadControllerRequest(
							LoadDataTxControllerTask.PLUGIN_KEY,
							Arrays.asList(CustomChannelParameters.TOPIC_CUSTOM),
							fetchScope)
			);
		}
		/* ... */
	}
}

Step 5 - Ensure FRTBConfig picks up our modifications

To ensure that our customizations are picked up, be sure to include CustomChannelParameters and ClientCustomisations classes in FRTBConfig located in /frtb-starter/src/main/java/com/activeviam/frtb/starter/cfg/impl/

Suggested further reading

Enriching File Fields by Adding Column Calculators

Adding New Cube Hierarchies

Adding New Cube Measures

Add and Load a New Column to Existing File