Adding and Loading New Columns to an Existing File

This page provides a description of how to add a new column to an existing file and load that column into the FRTB project. It features an example in which we will add ‘TestField’ to the file Trade_Attributes.csv. The techniques employed are generic examples that can be extended, adapted and repeated for any use case that we need. In all cases we make minimal changes to the Reference Implementation.

Editing the Data

For the purposes of this example, we will add a column to the file Trade_Attributes.csv and load the data from this column into the TradeBase store. In this case, our new column’s header is ‘TestField’, and all data in this column is ‘Testdata’. Below is a portion of the file.

Step 1 - Define customizations to datastore

Before we can load this new column into our cube, we need to make sure that our datastore has a field that can accept this column.

To add a new field to an existing datastore, we will need to navigate to the addModifications() method inside of the DatastoreCustomisationsConfig class.

   @Override
   public void addModifications(IDatastoreConfigurator configurator) {
      mergingDuplicateKeyHandlerCustomisations(configurator);
      customizationForDrc(configurator);
      if (extraDatastoreConfigs != null) {
         extraDatastoreConfigs.forEach(config -> config.accept(configurator));
      }
      // Here we add our modification
      configurator.appendField(ASourceConfig.TRADE_BASE_STORE, new CustomField(TEST_FIELD, ILiteralType.STRING));
   }

If we want to organize our customizations into a separate class for instance ClientCustomisations , we can create a new class as outlined here.

See Datastore Helper documentation for more information on potential datastore modifications.

Step 2 - Create our Replacement ChannelParameters Bean

Now that we have modified our store, we need to make sure the ETL is set up correctly so that the field will be properly populated.

1. Create a class which auto wires the appropriate source configuration class
2. Replace existing ChannelParameters Bean by qualifier. The replacement will be the same except for the addition of our custom field.
3. (Optional) If the tuple publisher needs to be modified, we will create a new one and add it to our ChannelParameters. In this case the tuple publisher was not modified but was added to the code example below to show how one would do so.

To figure out which qualifier to use, and the appropriate source configuration class see Available ChannelParameters Qualifiers.

public class CustomChannelParameters {

   @Autowired
   ASourceConfig sourceConfig;
   @Autowired
   IDatastore datastore;

   public static final String TEST_FIELD = "TestField";

   /**
    * Override existing ChannelParameters Bean to include our extra fields
    */
   @Primary
   @SourceConfigChannelParametersBean
   @Qualifier(SP_QUALIFIER__TRADE_ATTRIBUTES_CHANNEL_PARAMETER)
   public ACSVSourceConfig.ChannelParameters customTradeAttributesChannelParameter(ACSVSourceConfig.ChannelParameters tradeAttributesChannelParameter) {
      List<String> modifiedColumns = tradeAttributesChannelParameter.getFileColumns();
      modifiedColumns.add(TEST_FIELD);
      return tradeAttributesChannelParameter.toBuilder().fileColumns(modifiedColumns).build();
   }

   /**
    * (Optional) A custom tuple publisher can be created and used if necessary
    */
   public ITuplePublisher<IFileInfo<Path>> customTuplePublisher(ACSVSourceConfig.ChannelParameters channelParameters) {
      return new CustomTuplePublisher<>(datastore);
   }
}

Step 3 - Ensure FRTBConfig picks up our modifications

To ensure that our customisations are picked up, be sure to include CustomChannelParameters and ClientCustomisations classes in FRTBConfig located in /frtb-starter/src/main/java/com/activeviam/frtb/starter/cfg/impl/

When using DirectQuery

With DirectQuery, modifying CSV files is not applicable as DirectQuery uses a remote database in place of loading data from CSV files. But if you would like to add a column to an existing Table with DirectQuery, then see Modifying Datastores - When Using DirectQuery.

Suggested further reading

Enriching File Fields by Adding Column Calculators

Adding New Cube Hierarchies

Adding New Cube Measures

Adding New Data Loading or UnloadingTopics

Organizing Datastore Helper Customizations into a Separate Class