Enriching File Fields by Adding Column Calculators
This page provides a description of how we can add a new column calculator. It features an example in which we populate a new field to a new store. To see how this store was populated, see Adding New Data Loading or Unloading Topics. The techniques employed are generic examples that can be extended, adapted and repeated for any use case that we need. In all cases, we make minimal changes to the Accelerator Reference Implementation.
Step 1 - Define customizations to datastore
Before we can load this new column into our cube, we need to make sure that our datastore has a field that can accept this column.
To make modifications to our datastore, we will need to navigate to the
addModifications() method inside of the
DatastoreCustomisationsConfig.java class.
frtb-starter/src/main/java/com/activeviam/frtb/starter/cfg/impl/
In this case we will be adding a field to a store that we populated in Adding New Data Loading or Unloading Topics.
IStoreDescription storeDescription = new StoreDescriptionBuilder()
.withStoreName(CUSTOM_STORE)
.withField(AS_OF_DATE, ILiteralType.LOCAL_DATE).asKeyField()
.withField(RISK_CLASS, ILiteralType.STRING).asKeyField()
.withField("CustomProjected", ILiteralType.DOUBLE)
// we create the column which will be populated by our column calculator
.withField(CALCULATED_COLUMN_NAME)
.build();
configurator.addStore(FRTBConstants.FRTB_SCHEMA, storeDescription);
To enrich the data of an existing store, we can add the field to that existing store:
configurator.appendField(EXISTING_STORE, new CustomField(CALCULATED_COLUMN_NAME, ILiteralType.STRING))
See DatastoreHelper for more information.
Step 2 - Define column calculator
We want to define the logic of our column calculator.
Below we have an example of a simple Column Calculator that returns a value Negative when the Custom Projected value is less than 0, and Non-Negative otherwise. We will define our custom column calculator inside of:
/frtb-starter/src/main/java/com/activeviam/frtb/starter/cfg/impl/
public class CustomColumnCalculator implements IColumnCalculator<ILineReader> {
private String columnName;
public CustomColumnCalculator(String columnName) {
this.columnName = columnName;
}
@Override
public String getColumnName() {
return this.columnName;
}
@Override
public Object compute(IColumnCalculationContext<ILineReader> context) {
if ((Double) context.getValue(CustomStoreFileConstants.CUSTOM_PROJECTED) < 0) {
return "Negative";
}
return "Non-Negative";
}
}
Column Calculators can be used to add columns from simple operations across any number of columns that are present in your dataset. A common use case for column calculators is to parse data out of the file name. Below is how you would access the file name. More metadata is available through the context.
final String filename = context.getContext().getCurrentFile().getName();
Step 3 - Include column calculator in our channel parameters
For our column calculator to be picked up, we need to include it in our channel parameters.
To add channel parameters, we will have to extend the SourceConfig class.
@Configuration
abstract class ExtendedSourceConfig<I> extends SourceConfig {
public static final String CUSTOM_STORE = "Custom";
public static final String FILE_PATTERN_PROP = "custom.file.pattern";
public static final String TOPIC_CUSTOM = "Custom";
public static final String CALCULATED_COLUMN_NAME= "calculated";
@Bean
@DependsOn({"startManager"})
public List<ChannelParameters> sourceConfigStores() {
List<ChannelParameters> channelParametersList = super.sourceConfigStores();
channelParametersList.add(new ChannelParameters(CUSTOM_STORE, TOPIC_CUSTOM, FILE_PATTERN_PROP, "N/A", null, Arrays.asList(new CustomColumnCalculator(CALCULATED_COLUMN_NAME)), null));
return channelParametersList;
}
}
Step 4 - Application config
Navigate to the application config class
FRTBConfig/frtb-starter/src/main/java/com/activeviam/frtb/starter/cfg/impl
Replace SourceConfig with ExtendedSourceConfig
@Configuration
@Import(value = {
...
// SourceConfig.class,
ExtendedSourceConfig.class,
...
}