Navigation :

Adding and Loading New Columns to an Existing File

This page provides a description of how to add a new column to an existing file and load that column into the Market Risk Accelerator project. It features an example in which we will add ‘TestField’ to the file TradeAttributes.csv. The techniques employed are generic examples that can be extended, adapted and repeated for any use case that we need. In all cases we make minimal changes to the Accelerator Reference Implementation.

Editing the Data

For the purposes of this example, we will add a column to the file TradeAttributes.csv and load the data from this column into the TradeAttributes store. In this case, our new column’s header is ‘TestField’, and all data in this column is ‘Testdata’. Below is a portion of the file.

AsOfDate	TradeId	Book	LegalEntity	CounterpartyId	Notional	NotionalCcy	Trader	Sales	InstrumentClass	InstrumentType	InstrumentSubType	TradeDate	MaturityDate	VaRInclusion	TestField
2018-09-28	CDO_Cosco Pacific 358ff806	FI_STRUCT	ActiveBank Singapore	43	89618.90	EUR	John	Mathew	CDO	Non-linear	N/A	2018-04-19	2031-02-13	S	Testdata
2018-09-28	IRS_GBP_CPI 35a38b80	IR_LOW_VOL	ActiveBank US	56	25967.68	EUR	Matt	Chris	IRS	Linear	N/A	2018-06-29	2025-09-10		Testdata
2018-09-28	FX_OPT_CHF 35b1b658	IR_LOW_VOL	ActiveBank US	8	30949.51	EUR	Sam	Matt	FX_OPT	Non-linear	N/A	2018-02-08	2033-06-29		Testdata
2018-09-28	FX_SPOT_EUR 35bfaae2	IR_TRADING	ActiveBank UK	98	21140.64	EUR	Mathew	Chris	FX_SPOT	Spot	N/A	2018-06-22	2022-10-28		Testdata
2018-09-28	EQ_SWAP_Peugeot 35cbec92	EQ_STRUCT	ActiveBank Singapore	58	85350.16	EUR	Mathew	Chris	EQ_SWAP	Linear	N/A	2018-03-16	2033-05-09		Testdata
2018-09-28	COM_OPT_aluminium 35d9a77e	CM_OILGAS	ActiveBank US	73	71069.22	EUR	Mathew	John	COM_OPT	Non-linear	N/A	2018-06-22	2027-06-01		Testdata

Step 1 - Define customizations to datastore

Before we can load this new column into our cube, we need to make sure that our datastore has a field that can accept this column.

To add a new field to an existing store, we will need to create a bean with the signature @Qualifier(SP_QUALIFIER__CUSTOMISATIONS) Consumer<IDatastoreConfigurator> customisation()

    @Bean
    @Qualifier(SP_QUALIFIER__CUSTOMISATIONS)
    public Consumer<IDatastoreConfigurator> myCustomisations() {
        return Customisations::loadCustomisations;
    }
    public void loadCustomisations(IDatastoreConfigurator configurator) {
        configurator.appendField(StoreNames.TRADE_ATTRIBUTES_STORE_NAME,
                new CustomField(TEST_FIELD, ILiteralType.STRING));
    }

See Customizing the Datastore with the Datastore Helper for more information.

Step 2 - Source config / ETL code

Now that we have modified our store, we need to make sure the ETL is correctly set up so that the field will be properly populated. By default, there is no need to modify the ETL as the mapping is generally 1 to 1. If the tuple publisher needs to be modified, we will also need to modify our topic configuration. In this case the tuple publisher was not modified but was added to the code example below to show how you would do so.

To make modifications to our ETL, we will create the class ExtendedSourceConfig.java inside of the directory /risk-activepivot/src/main/java/com/activeviam/risk/ref/cfg/common/impl/source/ that extends the appropriate source configuration that handles the topic we want to modify. In this case, we will extend CommonCsvSourceConfig.java. We also need to change the medium specialized classes CommonAzureCsvSourceConfig and CommonLocalCsvSourceConfig.

Cube	Source Class	Medium classes
VaR-ES	VaRCsvSourceConfig	VaRAzureCsvSourceConfig, VaRLocalCsvSourceConfig
VaR-ES Summary	VaRSummaryCsvSourceConfig	VaRSummaryAzureCsvSourceConfig, VaRSummaryLocalCsvSourceConfig
Sensi Std	AStdSensiCsvSourceConfig	SensiAzureCsvSourceConfig, SensiLocalCsvSourceConfig
Sensi Scalar	AScalarSensiCsvSourceConfig	ScalarSensiAzureCsvSourceConfig, ScalarSensiLocalCsvSourceConfig
Sensi Summary	ASensiImportCsvSourceConfig	SensiImportLocalCsvSourceConfig, SensiImportAzureCsvSourceConfig
PnL	PnLCsvSourceConfig	PnLAzureCsvSourceConfig, PnLLocalCsvSourceConfig
PnL Summary	PnLSummaryCsvSourceConfig	PnLSummaryAzureCsvSourceConfig, PnLSummaryLocalCsvSourceConfig
ALL	CommonCsvSourceConfig	CommonAzureCsvSourceConfig, CommonLocalCsvSourceConfig

    @Configuration
    abstract class ExtendedSourceConfig<I> extends CommonCsvSourceConfig<I> {

       /**
         * (Optional) If there is a tuple publisher for the store you are modifying, then the method below is required so that our new tuple
         * publisher will overwrite the old one for our topic configuration. In this case CustomTuplePublisher will need to be defined.
         */
        @Override
        protected ITuplePublisher<IFileInfo<I>> getTuplePublisher(String topic, CSVMessageChannelFactory<I> csvChannelFactory) {
            return new SensiTradeStoreTuplePublisher<>(datastoreConfig.datastore(), store, tupleFields,
                    findGreekTypeBean, greekDescription, tenorUtil, tenorAndMaturityDefaultValue, moneynessDefaultValue, super.getTuplePublisher(topic, csvChannelFactory));
        }
    }

Step 3 - Application config

Navigate to the application config class CommonSourceConfiguration risk-activepivot/src/main/java/com/activeviam/risk/ref/cfg/common/impl/source/

Replace CommonCsvSourceConfig with ExtendedSourceConfig

@Configuration  
@Import(value = {  
...  
        // SourceConfig.class,
        ExtendedSourceConfig.class,
        ExtendedLocalSourceConfig.class,
        ExtendedAzureSourceConfig.class,
...  
})

Adding and Loading New Columns to an Existing File

Editing the Data

Step 1 - Define customizations to datastore

Step 2 - Source config / ETL code

Step 3 - Application config

Suggested Further Reading