Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.activeviam.com/llms.txt

Use this file to discover all available pages before exploring further.

This page provides a description of how to add a new column to an existing file and load that column into the project. It features an example in which we will add ‘TestField’ to the file Trade\_Attributes.csv. The techniques employed are generic examples that can be extended, adapted and repeated for any use case that we need. In all cases we make minimal changes to the Reference Implementation.

​
Editing the Data

For this example, we will add a column to the file Trade_Attributes.csv and load the data from this column into the TradeMapping store. In this case, our new column’s header is ‘TestField’, and all data in this column is ‘TestData’. Below is a portion of the file.
AsOfDateTradeIdBookLegalEntityNotionalNotional CcyPresentValuePVCcyResidualRiskExoticUnderlyingOtherResidualRiskTypeTradeDateSensitivity Scale CategoryRRAO CategoryTestField
2018-09-26EQ_FUT_JBS S.A. c854d9d0EQ_LARG_EMActiveBank US-71380331.05EUR-4207218.46EURNNN2018-09-26TestData
2018-09-26IRS_CNY_CPI c866a688CVA_EU_RATEActiveBank EU10938652.89EUR-1619042.59EURNNN2018-09-26TestData
2018-09-26EQ_SPOT_Sony c8791bf6EQ_WAREHOUActiveBank US35271290.98EUR-2181687.99EURNNN2018-09-26TestData
2018-09-26SWAPTION_JPY.IBOR6M c8896ec0CVA_AP_RATEActiveBank US52720813.95EUR10177813.4EURNNN2018-09-26TestData
2018-09-26EQ_SWAP_Cr_dit Agricole c89b077aCVA_HG_EQActiveBank UK-87503147.64EUR6911885.66EURNNN2018-09-26TestData
2018-09-26COM_FUT_dry-bulk route c8ada754CM_PREC_EXActiveBank EU-77905376.84EUR-12973904.23EURNNN2018-09-26TestData
2018-09-26IRS_DKK.IBOR6M c8bf1930CVA_EU_RATEActiveBank EU61826432.15EUR-3652040.68EURNNN2018-09-26TestData
 

​
Step 1 - Datastore Modifications

Before we can load this new column into our cube, we need to make sure that our datastore has a field that can accept this column. To add a new field to an existing datastore, we will create a DatastoreConfiguratorConsumer bean that will add a field to the existing DatastoreConfig.TRADE_MAPPING_STORE_NAME store. 
@Configuration
public static class DatastoreCustomisations {
   /**
    * Add our custom column to the {@link DatastoreConfig#TRADE_MAPPING_STORE_NAME} store.
    */
   @Qualifier(SP_QUALIFIER__CUSTOMISATIONS)
   @Bean
   public DatastoreConfiguratorConsumer datastoreConfiguratorConsumer(){
      // Add our custom Store & Reference into the datastore schema:
      return datastoreConfigurator -> datastoreConfigurator
              .appendField(DatastoreConfig.TRADE_MAPPING_STORE_NAME, new CustomField("TestField", ILiteralType.STRING));
   }
}
See the Datastore Helper documentation for more information on potential datastore modifications.

​
Step 2 - DLC Config

For this step, we will be using the Data Load Controller. Next, we may have to update the topic configuration to expect our new column. If no parser.columns are set for the Topic Description, the columns of the file are coming implicitly from the columns of the Atoti table. If the columns of the file still match the columns of the Atoti table, no other configuration is required. In the case of Trade_Attributes, the columns are being set explicitly so we must add the new column to the Trade_Attributes topic. Add the snippet below under dlc.csv.topics.Trade_Attributes.parser.columns in frtb-application/src/main/resources/application.yaml, which Spring Boot loads automatically at startup: 
dlc:
  csv:
    topics:
      Trade_Attributes:
        parser:
          columns:
            - AsOfDate
            - TradeId
            - Book
            - LegalEntity
            - Notional
            - NotionalCcy
            - PresentValue
            - PVCcy
            - ResidualRisk
            - ExoticUnderlying
            - OtherResidualRiskType
            - TradeDate
            - SensitivityScaleCategory
            - RRAOCategory
            - Inclusion
            - TestField
        channels:
          - target: tradeAttributesPublisher
The file sample-data/application-dlc-in-memory.yaml contains the complete DLC topic configuration used by the reference implementation and is a convenient reference for the exact YAML shape of every topic (file patterns, columns, channels). It is not loaded by default: it lives outside the application classpath and is gated behind the dlc-in-memory Spring profile. To consume it as-is, copy it to frtb-application/src/main/resources/ and start the application with -Dspring.profiles.active=dlc-in-memory. For most customisations, copying the relevant topic block into your application.yaml (as shown above) is simpler.
If you would like to specify the column names in Java, or if you need to include additional processing logic to handle the new column, you can do so by defining DLC Description beans. See DLC - Configuration for details on how to customize different aspects of the data loading.

​
Step 3 - Ensure FrtbApplicationConfig picks up our modifications

For our customizations to be picked up, we must include all added Spring @Configuration classes in FrtbApplicationConfig located in /frtb-application/src/main/java/com/activeviam/frtb/application/config/.

​
When using DirectQuery

With DirectQuery, modifying CSV files is not applicable as DirectQuery uses a remote database in place of loading data from CSV files. However, if you would like to add a column to an existing Table with DirectQuery, then see Modifying Datastores - when using DirectQuery.

​
Suggested further reading

Enriching File Fields by Adding Column Calculators Adding New Cube Hierarchies Adding New Cube Measures Adding and Populating a New Store