Adding New Data Loading or Unloading Topics

This page provides a description of how to load a new file into a new store within Atoti FRTB. The techniques employed are generic examples that can be extended, adapted and repeated for any use case that are needed. In all cases, we make minimal changes to the Reference Implementation.

Step 1 - Move data to the relevant directory

In this step, we want to bring any custom data into an appropriate folder inside:

/frtb-starter/src/test/resources/data/

For this example we will be bringing our file named Custom.csv within the 2018-09-26 folder and setting up our topic configuration so that we can load the same format file for each AsOfDate. Below is a sample file for reference.

Step 2 - Create the datastore definition we want to populate

To add a new datastore, we will need to navigate to the addModifications() method inside of the DatastoreCustomisationsConfig class.

frtb-starter/src/main/java/com/activeviam/frtb/starter/cfg/impl/

In this method we can append the following to create our datastore description.

	@Override
	public void addModifications(IDatastoreConfigurator configurator) {
		mergingDuplicateKeyHandlerCustomisations(configurator);
		customizationForDrc(configurator);
		if (extraDatastoreConfigs != null) {
			extraDatastoreConfigs.forEach(config -> config.accept(configurator));
		}
		// Here we add our modification
		// In this method we can append the following to create our datastore description.
		IStoreDescription storeDescription = StoreDescription.builder()
				.withStoreName(CUSTOM_STORE)
				.withField(StoreFieldNames.AS_OF_DATE, ILiteralType.LOCAL_DATE).asKeyField()
				.withField(StoreFieldNames.RISK_CLASS, ILiteralType.STRING).asKeyField()
				.withField("CustomProjected", ILiteralType.DOUBLE)
				// You can customize your datastore with any number of fields, as well as define partitioning, duplicate key handlers, and indexing configurations
				.build();

		configurator.addStore(FRTBConstants.FRTB_SCHEMA, storeDescription);

		// Here, we can also add a reference if needed:
		IReferenceDescription referenceDescription = configurator.referenceBuilder(FRTBConstants.FRTB_SCHEMA)
				.fromStore(SADatastoreConfig.SA_SENSITIVITIES_STORE_NAME)
				.toStore(CUSTOM_STORE)
				.withName(SADatastoreConfig.SA_SENSITIVITIES_STORE_NAME + "To" + CUSTOM_STORE)
				.withMapping(SADatastoreConfig.SA_SENSITIVITIES_STORE_RISK_CLASS, CUSTOM_STORE_RISK_CLASS)
				.withMapping(SADatastoreConfig.AS_OF_DATE, SADatastoreConfig.AS_OF_DATE)
				.build();

		configurator.addReference(FRTBConstants.FRTB_SCHEMA, referenceDescription);
	}

If we want to organize our customizations into a separate class for instance ClientCustomisations , we can create a new class as outlined here.

See Datastore Helper documentation for more information on potential datastore modifications.

Step 3 - Create ChannelParameters for our data loading topic

1. Create a class which auto wires the desired source configuration class
2. Create the ChannelParameters bean which includes the store name, topic name, and file pattern. We must qualify our bean with a unique qualifier. To wire ChannelParameters bean properly, we must use the appropriate ChannelParameters Bean Annotation. For more information, see ChannelParameters Bean Annotations.
3. Ensure that a file pattern property is defined within frtb-starter/src/main/resources/frtb-data-load.properties as custom.file-pattern=**/*Custom*.csv. Wildcards can be appended as needed, for instance to be able to include Custom_1.csv.

public class CustomChannelParameters {

	@Autowired
	ASourceConfig sourceConfig;

	private static final String CUSTOM_STORE = "CustomStore";
	public static final String FILE_PATTERN_PROP = "custom.file-pattern";
	public static final String TOPIC_CUSTOM = "Custom_TOPIC";
	public static final String SP_QUALIFIER__CUSTOM_STORE_CHANNEL_PARAMETER = "customStoreChannelParameter";

	@SourceConfigChannelParametersBean
	@Qualifier(SP_QUALIFIER__CUSTOM_STORE_CHANNEL_PARAMETER)
	public ACSVSourceConfig.ChannelParameters customStoreChannelParameter() {
		return sourceConfig.channelParametersBuilder(TOPIC_CUSTOM, CUSTOM_STORE, FILE_PATTERN_PROP).build();
	}
}

Step 4 - Include our new topic in initialDataLoad

Find InitialDataLoadConfig in frtb-starter/src/main/java/com/activeviam/frtb/starter/cfg/impl/ and modify initialDataLoad method to include our topic as desired. In this example, we want to load our new file for each asOfDate.

public class InitialDataLoadConfig {
	@Bean
	@DependsOn(value = { "startManager", "initialConfigDataLoad"})
	public Void initialDataLoad(
			@Qualifier("initialLoadAsOfDates") List<LocalDate> initialLoadAsOfDates,
			@Qualifier("initialLoadHistory") boolean initialLoadHistory
	) throws IOException {
		/* ... */
		for (final LocalDate date : initialLoadAsOfDates) {
			/* ... */
			controller.execute(
					new DataLoadControllerRequest(
							LoadDataTxControllerTask.PLUGIN_KEY,
							Arrays.asList(CustomChannelParameters.TOPIC_CUSTOM),
							fetchScope)
			);
		}
		/* ... */
	}
}

Step 5 - Ensure FRTBConfig picks up our modifications

To ensure that our customisations are picked up, be sure to include CustomChannelParameters and ClientCustomisations classes in FRTBConfig located in /frtb-starter/src/main/java/com/activeviam/frtb/starter/cfg/impl/

When using DirectQuery

When using DirectQuery, we must perform Step 2 and configure our datastore correctly. Then we have to modify our remote database to include the new Table / Fields. When doing so we must ensure we follow the naming convention of our Name Mapper.

Our database only needs to contain the new fields and or stores. When the application starts, the Datastore description will be converted into a database description. The remote database needs to contain the modifications before we start the application.

Adding a Field with DirectQuery

When adding a new field, we must add it to the datastore description as outlined in step 2 and also ensure the field exists in our database in the correct table.

Then when the application starts, the datastore description (containing the custom field) will be converted into a DirectQuery database description and the field can be used for Hierarchies or treated the same as other DirectQuery fields.

Adding a new Table with DirectQuery

When adding a new table, we first need to ensure we have properly completed step 2 and have defined our table on our remote database. Once complete, we can then add our new table to our DirectQuery database configuration (class extending ADirectQueryDatabaseSchema).

In this example we want to add a table with the name “MyCustomTable” in our Datastore config.

First we create a CustomDirectQueryDatabaseConfig marked as @Primary which extends the reference DirectQueryDatabaseConfig class. We add our table to the getDirectQueryTableDescriptions() method as follows:

@Primary
public class CustomDirectQueryDatabaseConfig extends DirectQueryDatabaseConfig {

	@Override
	public List<Table> getDirectQueryTableDescriptions() {
		// Get all default tables from the parent
		List<Table> existingTables = super.getDirectQueryTableDescriptions();

		existingTables.add(getMyCustomTable());

		return existingTables;
	}

	public Table getMyCustomTable(){
	    // Convert our datastore description into a database description
		return migrator.migrateToDirectQueryTable(
				getStoreDescriptionFor(
						"MyCustomTable"
				)
		);
	}
}

Then we just add our DirectQueryDatabaseConfig class to our application configuration and the extra table will exist in the Atoti instance.

Suggested further reading

Enriching File Fields by Adding Column Calculators

Adding New Cube Hierarchies

Adding New Cube Measures

Add and Load a New Column to Existing File