Navigation :

Enriching File Fields by Adding Column Calculators

This page provides a description of how we can add a new column calculator. It features an example in which we populate a new field to a new store. To see how this store was populated see Adding a New Data Loading or Unloading Topic. The techniques employed are generic examples that can be extended, adapted and repeated for any use case that we need. In all cases, we make minimal changes to the Accelerator Reference Implementation.

Step 1 - Define Customizations to Data Store

Before we can load this new column into our cube, we need to make sure that our datastore has a field that can accept this column.

To make modifications to our datastore, we will need to create a bean with the signature @Qualifier(SP_QUALIFIER__CUSTOMISATIONS) Consumer<IDatastoreConfigurator> customisation()

In this case we will be adding a field to a store which we populated here.

    @Bean
    @Qualifier(SP_QUALIFIER__CUSTOMISATIONS)
    public Consumer<IDatastoreConfigurator> myCustomisations() {
        return Customisations::loadCustomisations;
    }

In the case that we want to enrich the data of an existing store we can just add the field to that existing store:

    public void loadCustomisations(IDatastoreConfigurator configurator) {
        configurator.appendField(EXISTING_STORE, new CustomField(CALCULATED_COLUMN_NAME, ILiteralType.STRING));
    }

Read more about Datastore Helper. For details about the specific MRA implementation see Datastore Helper implementation within MRA.

Step 2 - Define Column Calculator

We want to define the logic of our column calculator.

Below we have an example of a simple Column Calculator that returns a value Negative when the Custom Projected value is less than 0, and Non-Negative otherwise. We will define our custom column calculator inside of:

/risk-starter/src/main/java/com/activeviam/risk/starter/cfg/impl/

public class CustomColumnCalculator implements IColumnCalculator<ILineReader> {
    private String columnName;
    public CustomColumnCalculator(String columnName) {
        this.columnName = columnName;
    }
    @Override
    public String getColumnName() {
        return this.columnName;
    }
    @Override
    public Object compute(IColumnCalculationContext<ILineReader> context) {
        if ((Double) context.getValue(CustomStoreFileConstants.CUSTOM_PROJECTED) < 0) {
            return "Negative";
        }
        return "Non-Negative";
    }
}

note

Column Calculators can be used to add columns from simple operations across any number of columns that are present in your dataset. A common use case for column calculators is to parse data out of the file name. Below is how you would access the file name:

`final String filename = context.getContext().getCurrentFile().getName();`

More metadata is available through the context.

Step 3 - Include Column Calculator in Our Channel Parameters

For our column calculator to be picked up, we need to include it into our channel parameters.

To add channel parameters we will have to extend the corresponding SourceConfig class.

Cube	Class
Var-ES	VaRSourceConfig
Sensi	SensiSourceConfig
PNL	PnLSourceConfig

@Configuration
abstract class ExtendedSourceConfig<I> extends VaRSourceConfig {
    public static final String CUSTOM_STORE = "Custom";
    public static final String FILE_PATTERN_PROP = "custom.file.pattern";
    public static final String TOPIC_CUSTOM = "Custom";
    public static final String CALCULATED_COLUMN_NAME= "calculated";
    @Bean
    @DependsOn({"startManager"})
    public List<ChannelParameters> sourceConfigStores() {
        List<ChannelParameters> channelParametersList = super.sourceConfigStores();
        channelParametersList.add(new ChannelParameters(CUSTOM_STORE, TOPIC_CUSTOM, FILE_PATTERN_PROP, "N/A", null, Arrays.asList(new CustomColumnCalculator(CALCULATED_COLUMN_NAME)), null));
        return channelParametersList;
    }
}

Step 4 - Application Config

Navigate to the application config class RiskStarterConfig``risk-starter/src/main/java/com/activeviam/risk/starter/cfg/impl/

Replace VaRSourceConfig with ExtendedSourceConfig

@Configuration  
@Import(value = {  
...  
        // SourceConfig.class,
        ExtendedSourceConfig.class,
...  
})