This page provides a description of how to load a new file into a new store within . The techniques employed are generic examples that can be extended, adapted and repeated for any use case that are needed. In all cases, we make minimal changes to the Reference Implementation by externalizing our customizations into configuration classes.Documentation Index
Fetch the complete documentation index at: https://docs.activeviam.com/llms.txt
Use this file to discover all available pages before exploring further.
Step 1 - Data Preparation
In this step, we want to bring any custom data into the appropriate folder. For this example we will be adding our files to/sample-data/data/ which is picked up by the csv-source.dataset property.
We will also place our file CustomProjection.csv into the dated subdirectory 2018-09-26 and set up our DLC configuration so that we
can load these files by dated subdirectory.
The file below will be placed into /sample-data/data/2018-09-26:
| AsOfDate | RiskClass | CustomProjected |
|---|---|---|
| 2018-09-26 | Commodity | -6640.633693 |
| 2018-09-26 | GIRR | 14020.37649 |
| 2018-09-26 | CSR Sec CTP | 8386.767854 |
| 2018-09-26 | CSR Sec non-CTP | 19218.3336 |
| 2018-09-26 | Equity | 8274.302903 |
| 2018-09-26 | FX | 2150.845785 |
| 2018-09-26 | CSR non-Sec | -25353.39868 |
| 2018-09-26 | DRC Sec non-CTP | 11319.08548 |
| 2018-09-26 | DRC non-Sec | 17134.37517 |
Step 2 - Datastore Modifications
For this step, we will be using the Datastore Helper. To add a new store, we need to define a newIStoreDescription and add it via DatastoreConfiguratorConsumer.
Optionally, we can also update our references by defining a IReferenceDescription and adding it via the DatastoreConfiguratorConsumer.
In the example below, we define the IStoreDescription, IReferenceDescription and update the datastore schema via a DatastoreConfiguratorConsumer in a
new Spring configuration class CustomProjectionStoreConfiguration:
FRTBConfig’s list of Spring configuration dependencies inside the @Import annotation.
For more information on potential datastore modifications, see the Datastore Helper documentation
Step 3 - DLC Configuration
For this step, we will be using the Data Load Controller.Define a Custom Source
To load custom data, we can create a custom Source Description, or use an existing one. In the example below, we create a custom Source Description which is configured to accept anAsOfDate scope.
The scope value will change which dated subdirectory we load from.
We explicitly include the topic which we will add in the next section so that our source does not implicitly pick up all existing CSV topics.
FRTBConfig’s list of Spring configuration dependencies inside the @Import annotation.
Define a Custom Topic Description
Next, we need to define the Topic Description to loadCustomProjection.csv into CustomProjectionStore.
The following topic configuration example configures a topic called “CustomProjectionStore”,
which loads files that match the file pattern {**/,}CustomProjection*.csv.
This configuration does not specify a target for the data, nor the data format.
If you do not name your topic the same as your store, you will have to explicitly specify the target in the channel of the topic.
- The target is implicitly the Atoti table named “CustomProjectionStore”.
- The format (column order and how input fields are parsed) comes implicitly from the columns and types of the targeted Atoti table, in this case “Trades”.
FRTBConfig’s list of Spring configuration dependencies inside the @Import annotation.
Step 4 - Executing Our Load
We can now load our Topic via the DLC. We can add a configuration to load our topic at application startup by defining a SpringApplicationRunner bean to
execute either before or after the application’s default initial data loading phase.
When the application starts, there are two ApplicationRunner beans that perform the initial load of Configuration data and Dated data. Their order is defined
via the StartupSpringBeanOrder integer constants:
StartupSpringBeanOrder.INITIAL_CONFIGURATION_DATA_LOAD: Default value is10.StartupSpringBeanOrder.INITIAL_DATA_LOAD: Default value is20.
INITIAL_CONFIGURATION_DATA_LOAD or INITIAL_DATA_LOAD by adding or subtracting 1 to either ordering phase.
Here we will define a configuration to load after the default initial data loading phase by adding 1 to the INITIAL_DATA_LOAD bean order:
FrtbApplicationConfig’s list of Spring configuration dependencies inside the @Import annotation.
Step 5 - Ensure FrtbApplicationConfig picks up our modifications
For our customizations to be picked up, we must include all added Spring@Configuration classes in FrtbApplicationConfig located in /frtb-application/src/main/java/com/activeviam/frtb/application/config/.
When using DirectQuery
When using DirectQuery, we must perform Step 2 (that is, create the datastore definition we want to populate) and configure our datastore correctly. Then we have to modify our remote database to include the new Table / Fields. When doing so we must ensure we follow the naming convention of our Name Mapper. Our database only needs to contain the new fields and/or stores. When the application starts, the datastore description will be converted into a database description. The remote database needs to contain the modifications before we start the application.Adding a Field with DirectQuery
When adding a new field, we must add it to the datastore description as outlined in Step 2 (that is, create the datastore definition we want to populate) and also ensure the field exists in our database in the correct table. Then when the application starts, the datastore description (containing the custom field) will be converted into a DirectQuery database description and the field can be used for Hierarchies or treated the same as other DirectQuery fields.When using custom fields with DirectQuery it is important to remember that datastore queries and getByKey queries are not recommended due to the performance
overhead.
Adding a new Table with DirectQuery
When adding a new table, all we need to do is complete Step 2 (that is, create the datastore definition we want to populate) and ensure we have defined our table on our remote database. Then theMigrator will convert our
in-memory store description into a remote DirectQuery table description automatically. This is done when we create our remote DirectQuery schema inside the
DirectQueryApplicationConfig.schema() method.