Adding and populating a new store

This page provides a description of how to load a new file into a new store within Atoti FRTB. The techniques employed are generic examples that can be extended, adapted and repeated for any use case that is needed. In all cases, minimal changes are made to the Reference Implementation by externalizing the customizations into configuration classes.

Step 1 - Data Preparation

In this step, any custom data is brought into the appropriate folder. For this example the files are added to /sample-data/data/ which is picked up by the csv-source.dataset property. The file CustomProjection.csv is also placed into the dated subdirectory 2018-09-26 and the DLC configuration is set up so that these files can be loaded by dated subdirectory. The file below will be placed into /sample-data/data/2018-09-26:

AsOfDate	RiskClass	CustomProjected
2018-09-26	Commodity	-6640.633693
2018-09-26	GIRR	14020.37649
2018-09-26	CSR Sec CTP	8386.767854
2018-09-26	CSR Sec non-CTP	19218.3336
2018-09-26	Equity	8274.302903
2018-09-26	FX	2150.845785
2018-09-26	CSR non-Sec	-25353.39868
2018-09-26	DRC Sec non-CTP	11319.08548
2018-09-26	DRC non-Sec	17134.37517

Step 2 - Datastore Modifications

For this step, the Datastore Helper is used. To add a new store, define a new IStoreDescription and add it via DatastoreConfiguratorConsumer. Optionally, references can also be updated by defining a IReferenceDescription and adding it via the DatastoreConfiguratorConsumer. In the example below, the IStoreDescription and IReferenceDescription are defined and the datastore schema is updated via a DatastoreConfiguratorConsumer in a new Spring configuration class CustomProjectionStoreConfiguration:

@Configuration
public static class CustomProjectionStoreConfiguration {

	/**
	 * Define our custom store
	 */
	@Bean
	public IStoreDescription customProjectionStoreDescription() {
		return StartBuilding.store()
				.withStoreName("CustomProjectionStore")
				.withField("CustomProjection_AsOfDate", DatastoreConfig.STORE_DATE_FIELD_FORMAT).asKeyField()
				.withField("CustomProjection_RiskClass", ILiteralType.STRING).asKeyField()
				.withField("CustomProjected", ILiteralType.DOUBLE)
				.build();
	}

	/**
	 * Define a reference from the SA Cube's base store to our CustomProjectionStore store
	 */
	@Bean
	public IReferenceDescription customProjectionReferenceDescription() {
		var joinName = DatastoreConfig.ConcreteDatastoreConfig
				.joinName(
						SADatastoreConfig.SA_SENSITIVITIES_STORE_NAME,
						"CustomProjectionStore"
				);
		return StartBuilding.reference()
				.fromStore(SADatastoreConfig.SA_SENSITIVITIES_STORE_NAME)
				.toStore("CustomProjectionStore")
				.withName(joinName)
				.withMapping(SADatastoreConfig.AS_OF_DATE, "CustomProjection_AsOfDate")
				.withMapping(SADatastoreConfig.SA_SENSITIVITIES_STORE_RISK_CLASS, "CustomProjection_RiskClass")
				.build();
	}

	/**
	 * Add our custom Store and Reference into the overall Datastore Schema
	 */
	@Bean
	@Order(DatastoreCustomisationSpringBeanOrder.PROJECT)
	public DatastoreConfiguratorConsumer datastoreConfiguratorConsumer(
			IStoreDescription customStoreDescription,
			IReferenceDescription customStoreReference){
		// Add our custom Store & Reference into the datastore schema:
		return datastoreConfigurator -> datastoreConfigurator
				.addStore(customStoreDescription)
				.addReference(customStoreReference);
	}
}

This configuration class must be added into FRTBConfig’s list of Spring configuration dependencies inside the @Import annotation. For more information on potential datastore modifications, see the Datastore Helper documentation

Step 3 - DLC Configuration

For this step, the Data Load Controller is used.

Define a Custom Source

To load custom data, create a custom Source Description, or use an existing one. In the example below, a custom Source Description is created which is configured to accept an AsOfDate scope. The scope value will change which dated subdirectory is loaded from. The topic that is added in the next section is explicitly included so that the source does not implicitly pick up all existing CSV topics.

@Configuration
public static class CustomSourceConfig {

	/**
	 * Define a Source to load our custom topics through.
	 */
	@Bean
	LocalCsvSourceDescription customLocalCsvSourceDescription(CsvSourceDatasetConfigurationProperties csvSourceDataset) {
		return LocalCsvSourceDescription.builder("CustomSource", csvSourceDataset.getDataset() + AS_OF_DATE_SUBDIR)
				// Provide all custom topic names:
				.topicsToInclude(Set.of("CustomProjectionStore"))
				.build();
	}
}

This configuration class must be added into FRTBConfig’s list of Spring configuration dependencies inside the @Import annotation.

Define a Custom Topic Description

Next, define the Topic Description to load CustomProjection.csv into CustomProjectionStore. The following topic configuration example configures a topic called “CustomProjectionStore”, which loads files that match the file pattern {**/,}CustomProjection*.csv. This configuration does not specify a target for the data, nor the data format.

If you do not name your topic the same as your store, you will have to explicitly specify the target in the channel of the topic.

The target is implicitly the Atoti table named “CustomProjectionStore”.
The format (column order and how input fields are parsed) comes implicitly from the columns and types of the targeted Atoti table, in this case “CustomProjectionStore”.

@Configuration
public static class CustomTopicsConfig {
	/**
	 * Define the topic to load our Custom Projection file.
	 */
	@Bean
	@Qualifier
	public CsvTopicDescription customProjectionTopicDescription() {
		// Create the Topic Description:
		return CsvTopicDescription.builder("CustomProjectionStore", "glob:{**/,}CustomProjection*.csv")
				.build();
	}
}

This configuration class must be added into FRTBConfig’s list of Spring configuration dependencies inside the @Import annotation.

Step 4 - Executing the Load

The Topic can now be loaded via the DLC. A configuration can be added to load the topic at application startup by defining a Spring ApplicationRunner bean to execute either before or after the application’s default initial data loading phase. When the application starts, there are two ApplicationRunner beans that perform the initial load of Configuration data and Dated data. Their order is defined via the StartupSpringBeanOrder integer constants:

StartupSpringBeanOrder.INITIAL_CONFIGURATION_DATA_LOAD: Default value is 10.
StartupSpringBeanOrder.INITIAL_DATA_LOAD: Default value is 20.

Loading can occur before or after the INITIAL_CONFIGURATION_DATA_LOAD or INITIAL_DATA_LOAD by adding or subtracting 1 to either ordering phase. Here, a configuration is defined to load after the default initial data loading phase by adding 1 to the INITIAL_DATA_LOAD bean order:

@Configuration
public static class LoadCustomTopicOnInitialLoadConfig {

	@Autowired IDataLoadControllerService dataLoadControllerService;

	@Bean
	// Run before (-) / after (+) the default initial dated data load
	@Order(StartupSpringBeanOrder.INITIAL_DATA_LOAD + 1)
	@ConditionalOnInMemoryDatabase
	public ApplicationRunner customTopicInitialLoad() {
		return args -> {
			// Execute loading of our topic:
			var requestResult = dataLoadControllerService.execute(
					DlcLoadRequest.builder()
							.topics("CustomProjectionStore")
							.scope(DlcScope.of(Map.of(SCOPE__AS_OF_DATE, "2018-09-26")))
							.build()
			);
		};
	}
}

This configuration class must be added into FrtbApplicationConfig’s list of Spring configuration dependencies inside the @Import annotation.

Step 5 - Ensure FrtbApplicationConfig picks up the modifications

For the customizations to be picked up, all added Spring @Configuration classes must be included in FrtbApplicationConfig located in /frtb-application/src/main/java/com/activeviam/frtb/application/config/.

When using DirectQuery

When using DirectQuery, Step 2 (that is, create the datastore definition to populate) must be performed and the datastore configured correctly. Then the remote database must be modified to include the new Table / Fields. When doing so, the naming convention of the Name Mapper must be followed. The database only needs to contain the new fields and/or stores. When the application starts, the datastore description will be converted into a database description. The remote database needs to contain the modifications before the application is started.

Adding a Field with DirectQuery

When adding a new field, it must be added to the datastore description as outlined in Step 2 and the field must also exist in the database in the correct table. Then when the application starts, the datastore description (containing the custom field) will be converted into a DirectQuery database description and the field can be used for Hierarchies or treated the same as other DirectQuery fields.

When using custom fields with DirectQuery it is important to remember that datastore queries and getByKey queries are not recommended due to the performance overhead.

Adding a new Table with DirectQuery

When adding a new table, all that is required is to complete Step 2 and ensure the table is defined on the remote database. Then the Migrator will convert the in-memory store description into a remote DirectQuery table description automatically. This is done when the remote DirectQuery schema is created inside the DirectQueryApplicationConfig.schema() method.

The DirectQuery and in-memory Atoti servers must be configured identically. There must not be any hierarchies in the DirectQuery data node that do not exist in the in-memory one.

​Step 1 - Data Preparation

​Step 2 - Datastore Modifications

​Step 3 - DLC Configuration

​Define a Custom Source

​Define a Custom Topic Description

​Step 4 - Executing the Load

​Step 5 - Ensure FrtbApplicationConfig picks up the modifications

​When using DirectQuery

​Adding a Field with DirectQuery

​Adding a new Table with DirectQuery

​Suggested further reading

Step 1 - Data Preparation

Step 2 - Datastore Modifications

Step 3 - DLC Configuration

Define a Custom Source

Define a Custom Topic Description

Step 4 - Executing the Load

Step 5 - Ensure FrtbApplicationConfig picks up the modifications

When using DirectQuery

Adding a Field with DirectQuery

Adding a new Table with DirectQuery

Suggested further reading