☰
Navigation :

Adding and Loading New Columns to an Existing File

This page provides a description of how to add a new column to an existing file and load that column into the Market Risk Accelerator project. It features an example in which we will add ‘TestField’ to the file Trade_Attributes.csv. The techniques employed are generic examples that can be extended, adapted and repeated for any use case that we need. In all cases we make minimal changes to the Accelerator Reference Implementation.

Editing the Data

For the purposes of this example, we will add a column to the file Trade_Attributes.csv and load the data from this column into the TradeBase store. In this case, our new column’s header is ‘TestField’, and all data in this column is ‘Testdata’. Below is a portion of the file.

AsOfDate TradeId Book LegalEntity Notional Notional Ccy PresentValue PVCcy ResidualRisk ExoticUnderlying OtherResidualRiskType TradeDate TestField
2018-09-26 EQ_FUT_JBS S.A. c854d9d0 EQ_LARG_EM ActiveBank US -71380331.05 EUR -4207218.46 EUR N N N 2018-09-26 Testdata
2018-09-26 IRS_CNY_CPI c866a688 CVA_EU_RATE ActiveBank EU 10938652.89 EUR -1619042.59 EUR N N N 2018-09-26 Testdata
2018-09-26 EQ_SPOT_Sony c8791bf6 EQ_WAREHOU ActiveBank US 35271290.98 EUR -2181687.99 EUR N N N 2018-09-26 Testdata
2018-09-26 SWAPTION_JPY.IBOR6M c8896ec0 CVA_AP_RATE ActiveBank US 52720813.95 EUR 10177813.4 EUR N N N 2018-09-26 Testdata
2018-09-26 EQ_SWAP_Cr_dit Agricole c89b077a CVA_HG_EQ ActiveBank UK -87503147.64 EUR 6911885.66 EUR N N N 2018-09-26 Testdata
2018-09-26 COM_FUT_dry-bulk route c8ada754 CM_PREC_EX ActiveBank EU -77905376.84 EUR -12973904.23 EUR N N N 2018-09-26 Testdata
2018-09-26 IRS_DKK.IBOR6M c8bf1930 CVA_EU_RATE ActiveBank EU 61826432.15 EUR -3652040.68 EUR N N N 2018-09-26 Testdata

Step 1 - Define customizations to datastore

Before we can load this new column into our cube, we need to make sure that our datastore has a field that can accept this column.

To add a new field to an existing store, we will need to create a bean with the signature @Qualifier(SP_QUALIFIER__CUSTOMISATIONS) Consumer<IDatastoreConfigurator> customisation()

    @Bean
    @Qualifier(SP_QUALIFIER__CUSTOMISATIONS)
    public Consumer<IDatastoreConfigurator> myCustomisations() {
        return Customisations::loadCustomisations;
    }
    public void loadCustomisations(IDatastoreConfigurator configurator) {
        configurator.appendField(ASourceConfig.TRADE_BASE_STORE,
                new CustomField(TEST_FIELD, ILiteralType.STRING));
    }

See Customizing the Datastore with the Datastore Helper for more information.

Step 2 - Source config / ETL code

Now that we have modified our store, we need to make sure the ETL is correctly set up so that the field will be properly populated.

To make modifications to our ETL, we will create the class ExtendedSourceConfig.java inside of the directory /risk-starter/src/main/java/com/activeviam/risk/starter/cfg/impl/ that extends the appropriate source configuration that handles the topic we want to modify. In this case, we will extend VaRSourceConfig.java .

  1. Ensure that the columns that are expected by the Source are set properly. (In this case we created a class ExtendedTradeAttributesFileConstants, which is exactly the same as TradeAttributesFileConstants with the addition of our field.)
  2. (Optional) If the tuple publisher needs to be modified, we will also need to modify our topic configuration. In this case the tuple publisher was not modified but was added to the code example below to show how you would do so.
Cube Class
Var-ES VaRSourceConfig
Sensi SensiSourceConfig
PNL PnLSourceConfig 
@Configuration
abstract class ExtendedSourceConfig<I> extends VaRSourceConfig {
    @Autowired
    String csvSourceFilePath;
    /**
     * 1. Ensure that the columns are properly set for the topic in the Source.
     */
    @Override
    public CsvScopedFetchSource<Path> csvSource() {
        CsvScopedFetchSource<Path> csvSource = super.csvSource();
        final String filePath = csvSourceFilePath;
        createAndAddTopics(csvSource, filePath, ExtendedTradeAttributesFileConstants.TOPIC_TRADES,
                FRTBDataLoadConstants.TRADE_ATTRIBUTES_SA_TRADES_FILE_PATTERN, ExtendedTradeAttributesFileConstants.ALL_FIELDS, false, null);
        return  csvSource;
    }
   /**
     * 2. (Optional) If there is a tuple publisher for the store you are modifying then the below method is necessary so that our new tuple
     * publisher will overwrite the old for our topic configuration. In this case CustomTuplePublisher will need to be defined.
     */
    @Override
    @Bean
    protected Map<String, SimpleSourceConfig.ITopicConfig<IFileInfo<Path>>> csvTopicConfigs() {
        Map<String, SimpleSourceConfig.ITopicConfig<IFileInfo<Path>>> topicConfigs = super.csvTopicConfigs();
        topicConfigs.put(ExtendedTradeAttributesFileConstants.TOPIC_TRADES,
                createTopicConfig(ExtendedTradeAttributesFileConstants.TOPIC_TRADES, TRADE_BASE_STORE, new CustomTuplePublisher<>(datastoreConfig.datastore()), false));
        return topicConfigs;
    }
}

Step 3 - Application config

Navigate to the application config class RiskStarterConfig``risk-starter/src/main/java/com/activeviam/risk/starter/cfg/impl/

Replace VaRSourceConfig with ExtendedSourceConfig

@Configuration  
@Import(value = {  
...  
        // SourceConfig.class,
        ExtendedSourceConfig.class,
...  
})

Suggested Further Reading

Enriching File Fields by Adding Column Calculators
Adding a New Cube Hierarchy
Adding a New Cube Measure
Adding a New Data Loading or Unloading Topic

search.js