Adding columns to an existing file and store

This page provides a description of how to add a new column to an existing file and load that column into the Atoti FRTB project. It features an example in which TestField is added to the file Trade\_Attributes.csv. The techniques employed are generic examples that can be extended, adapted and repeated for any use case. In all cases, only minimal changes are made to the Reference Implementation.

Editing the data

For this example, a column is added to the file Trade_Attributes.csv and the data from this column is loaded into the TradeMapping store. In this case, the new column’s header is TestField, and all data in this column is TestData. Below is a portion of the file.

AsOfDate	TradeId	Book	LegalEntity	Notional	Notional Ccy	PresentValue	PVCcy	ResidualRisk	ExoticUnderlying	OtherResidualRiskType	TradeDate	TestField
2018-09-26	EQ_FUT_JBS S.A. c854d9d0	EQ_LARG_EM	ActiveBank US	-71380331.05	EUR	-4207218.46	EUR	N	N	N	2018-09-26	TestData
2018-09-26	IRS_CNY_CPI c866a688	CVA_EU_RATE	ActiveBank EU	10938652.89	EUR	-1619042.59	EUR	N	N	N	2018-09-26	TestData
2018-09-26	EQ_SPOT_Sony c8791bf6	EQ_WAREHOU	ActiveBank US	35271290.98	EUR	-2181687.99	EUR	N	N	N	2018-09-26	TestData
2018-09-26	SWAPTION_JPY.IBOR6M c8896ec0	CVA_AP_RATE	ActiveBank US	52720813.95	EUR	10177813.4	EUR	N	N	N	2018-09-26	TestData
2018-09-26	EQ_SWAP_Cr_dit Agricole c89b077a	CVA_HG_EQ	ActiveBank UK	-87503147.64	EUR	6911885.66	EUR	N	N	N	2018-09-26	TestData
2018-09-26	COM_FUT_dry-bulk route c8ada754	CM_PREC_EX	ActiveBank EU	-77905376.84	EUR	-12973904.23	EUR	N	N	N	2018-09-26	TestData
2018-09-26	IRS_DKK.IBOR6M c8bf1930	CVA_EU_RATE	ActiveBank EU	61826432.15	EUR	-3652040.68	EUR	N	N	N	2018-09-26	TestData

Step 1 - Datastore modifications

Before this new column can be loaded into the cube, the datastore must have a field that can accept this column. To add a new field to an existing datastore, create a DatastoreConfiguratorConsumer bean that adds a field to the existing DatastoreConfig.TRADE_MAPPING_STORE_NAME store.

@Configuration
public static class DatastoreCustomisations {
   /**
    * Add our custom column to the {@link DatastoreConfig#TRADE_MAPPING_STORE_NAME} store.
    */
   @Bean
   @Order(DatastoreCustomisationSpringBeanOrder.PROJECT)
   public DatastoreConfiguratorConsumer datastoreConfiguratorConsumer(){
      // Add our custom Store & Reference into the datastore schema:
      return datastoreConfigurator -> datastoreConfigurator
              .appendField(DatastoreConfig.TRADE_MAPPING_STORE_NAME, new CustomField("TestField", ILiteralType.STRING));
   }
}

See the Datastore Helper documentation for more information on potential datastore modifications.

Step 2 - DLC config

This step uses the Data Load Controller. Next, the topic configuration may have to be updated to expect the new column. If no parser.columns are set for the Topic Description, the columns of the file are coming implicitly from the columns of the Atoti table. If the columns of the file still match the columns of the Atoti table, no other configuration is required. In the case of Trade_Attributes, the columns are being set explicitly, so the new column must be added to the Trade_Attributes topic. Add the snippet below under dlc.csv.topics.Trade_Attributes.parser.columns in frtb-application/src/main/resources/application.yaml, which Spring Boot loads automatically at startup:

dlc:
  csv:
    topics:
      Trade_Attributes:
        parser:
          columns:
            - AsOfDate
            - TradeId
            - Book
            - LegalEntity
            - Notional
            - NotionalCcy
            - PresentValue
            - PVCcy
            - ResidualRisk
            - ExoticUnderlying
            - OtherResidualRiskType
            - TradeDate
            - SensitivityScaleCategory
            - RRAOCategory
            - Inclusion
            - TestField
        channels:
          - target: tradeAttributesPublisher

The file sample-data/application-dlc-in-memory.yaml contains the complete DLC topic configuration used by the reference implementation and is a convenient reference for the exact YAML shape of every topic (file patterns, columns, channels). It is not loaded by default: it lives outside the application classpath and is gated behind the dlc-in-memory Spring profile. To consume it as-is, copy it to frtb-application/src/main/resources/ and start the application with -Dspring.profiles.active=dlc-in-memory. For most customizations, copying the relevant topic block into your application.yaml (as shown above) is simpler.

If you would like to specify the column names in Java, or if you need to include additional processing logic to handle the new column, you can do so by defining DLC Description beans. See DLC - Configuration for details on how to customize different aspects of the data loading.

Step 3 - Ensure FrtbApplicationConfig picks up the modifications

For the customizations to be picked up, all added Spring @Configuration classes must be included in FrtbApplicationConfig located in /frtb-application/src/main/java/com/activeviam/frtb/application/config/.

When using DirectQuery

With DirectQuery, modifying CSV files is not applicable as DirectQuery uses a remote database in place of loading data from CSV files. However, if you would like to add a column to an existing Table with DirectQuery, then see Modifying Datastores - when using DirectQuery.

​Editing the data

​Step 1 - Datastore modifications

​Step 2 - DLC config

​Step 3 - Ensure FrtbApplicationConfig picks up the modifications

​When using DirectQuery

​Suggested further reading

Editing the data

Step 1 - Datastore modifications

Step 2 - DLC config

Step 3 - Ensure FrtbApplicationConfig picks up the modifications

When using DirectQuery

Suggested further reading