Adding and Loading New Columns to an Existing File
This page provides a description of how to add a new column to an existing file and load that column into the FRTB project. It features an example in which we will add ‘TestField’ to the file Trade_Attributes.csv. The techniques employed are generic examples that can be extended, adapted and repeated for any use case that we need. In all cases we make minimal changes to the Accelerator Reference Implementation.
Editing the Data
For the purposes of this example, we will add a column to the file
Trade_Attributes.csv
and load the data from this column into the
TradeBase
store. In this case, our new column’s header is ‘TestField’,
and all data in this column is ‘Testdata’. Below is a portion of the
file.
AsOfDate | TradeId | Book | LegalEntity | Notional | Notional Ccy | PresentValue | PVCcy | ResidualRisk | ExoticUnderlying | OtherResidualRiskType | TradeDate | TestField |
---|---|---|---|---|---|---|---|---|---|---|---|---|
2018-09-26 | EQ_FUT_JBS S.A. c854d9d0 | EQ_LARG_EM | ActiveBank US | -71380331.05 | EUR | -4207218.46 | EUR | N | N | N | 2018-09-26 | Testdata |
2018-09-26 | IRS_CNY_CPI c866a688 | CVA_EU_RATE | ActiveBank EU | 10938652.89 | EUR | -1619042.59 | EUR | N | N | N | 2018-09-26 | Testdata |
2018-09-26 | EQ_SPOT_Sony c8791bf6 | EQ_WAREHOU | ActiveBank US | 35271290.98 | EUR | -2181687.99 | EUR | N | N | N | 2018-09-26 | Testdata |
2018-09-26 | SWAPTION_JPY.IBOR6M c8896ec0 | CVA_AP_RATE | ActiveBank US | 52720813.95 | EUR | 10177813.4 | EUR | N | N | N | 2018-09-26 | Testdata |
2018-09-26 | EQ_SWAP_Cr_dit Agricole c89b077a | CVA_HG_EQ | ActiveBank UK | -87503147.64 | EUR | 6911885.66 | EUR | N | N | N | 2018-09-26 | Testdata |
2018-09-26 | COM_FUT_dry-bulk route c8ada754 | CM_PREC_EX | ActiveBank EU | -77905376.84 | EUR | -12973904.23 | EUR | N | N | N | 2018-09-26 | Testdata |
2018-09-26 | IRS_DKK.IBOR6M c8bf1930 | CVA_EU_RATE | ActiveBank EU | 61826432.15 | EUR | -3652040.68 | EUR | N | N | N | 2018-09-26 | Testdata |
Step 1 - Define customizations to datastore
Before we can load this new column into our cube, we need to make sure that our datastore has a field that can accept this column.
To add a new field to an existing datastore, we will need to navigate to
the addModifications()
method inside of the
DatastoreCustomisationsConfig
class.
@Override
public void addModifications(IDatastoreConfigurator configurator) {
configurator.appendField(ASourceConfig.TRADE\_BASE\_STORE,
new CustomField(TEST\_FIELD, ILiteralType.STRING));
See [Datastore Helper]((https://artifacts.activeviam.com/documentation/accelerators/tools/dash/2.0.0-AP5.9/online-help/) for more information.
Step 2 - Source config / ETL code
Now that we have modified our store, we need to make sure the ETL is set up properly so that the field will be properly populated.
To make modifications to our ETL, we will create the class
ExtendedSourceCofig.java
inside of the directory
/frtb-starter/src/main/java/com/activeviam/frtb/starter/cfg/impl/
that
extends the appropriate source configuration that handles the topic we
want to modify. In this case, we will extend SourceConfig.java
.
- Ensure that the columns that are expected by the Source are set
properly. (In this case we created a class
ExtendedTradeAttributesFileConstants
, which is exactly the same asTradeAttributesFileConstants
with the addition of our field.) - (Optional) If the tuple publisher needs to be modified, we will also need to modify our topic configuration. In this case the tuplepublisher was not modified but was added to the code example below to show how one would do so.
@Configuration
abstract class ExtendedSourceConfig<I> extends SourceConfig {
@Autowired
String csvSourceFilePath;
/**
* 1. Ensure that the columns are properly set for the topic in the Source.
*/
@Override
public CsvScopedFetchSource<Path> csvSource() {
CsvScopedFetchSource<Path> csvSource = super.csvSource();
final String filePath = csvSourceFilePath;
createAndAddTopics(csvSource, filePath, ExtendedTradeAttributesFileConstants.TOPIC_TRADES,
FRTBDataLoadConstants.TRADE_ATTRIBUTES_SA_TRADES_FILE_PATTERN, ExtendedTradeAttributesFileConstants.ALL_FIELDS, false, null);
return csvSource;
}
/**
* 2. (Optional) If there is a tuple publisher for the store you are modifying then the below method is necessary so that our new tuple
* publisher will overwrite the old for our topic configuration. In this case CustomTuplePublisher will need to be defined.
*/
@Override
@Bean
protected Map<String, SimpleSourceConfig.ITopicConfig<IFileInfo<Path>>> csvTopicConfigs() {
Map<String, SimpleSourceConfig.ITopicConfig<IFileInfo<Path>>> topicConfigs = super.csvTopicConfigs();
topicConfigs.put(ExtendedTradeAttributesFileConstants.TOPIC_TRADES,
createTopicConfig(ExtendedTradeAttributesFileConstants.TOPIC_TRADES, TRADE_BASE_STORE, new CustomTuplePublisher<>(datastoreConfig.datastore()), false));
return topicConfigs;
}
}
Step 3 - Application config
Navigate to the application config class
FRTBConfig/frtb-starter/src/main/java/com/activeviam/frtb/starter/cfg/impl/
Replace SourceConfig
with ExtendSourceConfig
.
/frtb-starter/src/main/java/com/activeviam/frtb/starter/cfg/impl/FRTBConfig.java
@Configuration
@Import(value = {
...
// SourceConfig.class,
ExtendSourceConfig.class,
...
})
Suggested further reading
Enriching File Fields by Adding Column Calculators, Adding New Cube Hierarchies, Adding New Cube Measures, and Adding New Data Loading or UnloadingTopics.