Navigation :

Column Calculators

This page provides a description of how we can add a new column calculator. It features an example in which we populate a new field to SaCvaVegaRiskWeight store. The techniques employed are generic examples that can be extended, adapted and repeated for any use case that we need. In all cases, we make minimal changes to the Reference Implementation.

Step 1 - Define Customisations to Data Store via Datastore Helper

The first thing we want to do is to add a field that we will populate with our column calculator, in this case we will be exploring a few examples. Define data store customizations inside of loadcustomisations() method inside DatastoreCustomisations class located /cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/DatastoreCustomisations.java

See more customizations achievable with Datastore Helper.

package com.activeviam.cvarc.starter.cfg.impl;

import com.activeviam.tools.datastore.IDatastoreCustomisations;
import com.activeviam.tools.datastore.impl.CustomField;
import com.activeviam.tools.datastore.impl.DatastoreHelper;
import com.qfs.literal.ILiteralType;

public class DatastoreCustomisations implements IDatastoreCustomisations {

    /**
     * Defines the transformations to be performed on the default stores, using {@link DatastoreHelper}.
     */
    @Override
    public void loadCustomisations() {
        //Define transformations here.

        DatastoreHelper.appendField("SaCvaVegaRiskWeight", new CustomField("FileName", ILiteralType.STRING));
        DatastoreHelper.appendField("SaCvaVegaRiskWeight", new CustomField("Math", ILiteralType.DOUBLE));

    }
}

Step 2 - Define Column Calculator

We want to define the logic of our column calculator.

Bellow we have some examples of Column Calculators that we will define inside of /cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/

File Name Column Calculator

package com.activeviam.cvarc.starter.cfg.impl;

import com.qfs.msg.csv.ILineReader;
import com.qfs.msg.csv.translator.impl.AColumnCalculator;

public class FileNameColCal extends AColumnCalculator<ILineReader> {

    public FileNameColCal(String columnName) {
        super(columnName);
    }

    @Override
    public Object compute(IColumnCalculationContext<ILineReader> context) {
        final String filename = context.getContext().getCurrentFile().getName();
        return filename;
    }
}

Mathematical Operation Between Two Existing Columns Column Calculator

package com.activeviam.cvarc.starter.cfg.impl;

import com.qfs.msg.csv.ILineReader;
import com.qfs.msg.csv.translator.impl.AColumnCalculator;

public class MathColCal extends AColumnCalculator<ILineReader> {

    public MathColCal(String columnName) {
        super(columnName);
    }

    @Override
    public Object compute(IColumnCalculationContext<ILineReader> context) {
        final double RiskWeight = (double) context.getValue("RiskWeight");
        final double ParameterC = (double) context.getValue("ParameterC");
        return RiskWeight + ParameterC;
    }
}

Step 3 - Include Column Calculator in Channel Factory

For our column calculator to be picked up we need to include it into our Channel Factory.

package com.activeviam.cvarc.starter.cfg.impl;

import com.activeviam.cvarc.sa.ref.cfg.impl.SaCsvSourceConfig;
import com.qfs.msg.IColumnCalculator;
import com.qfs.msg.csv.ILineReader;
import com.qfs.source.impl.CSVMessageChannelFactory;
import java.nio.file.Path;
import java.util.ArrayList;
import java.util.List;

public class ExtendedSaCsvSourceConfig extends SaCsvSourceConfig {


@Override
public CSVMessageChannelFactory<Path> bulkLoadCsvMessageChannelFactory(String sourceName) {

    final CSVMessageChannelFactory<Path> csvMessageChannelFactory = new CSVMessageChannelFactory<>(
            bulkLoadCsvSource(sourceName), datastoreConfig.datastore());

    final List<IColumnCalculator<ILineReader>> csvCalculatedColumns = new ArrayList<>();
    csvCalculatedColumns.add(new FileNameColCal("FileName"));                                    // Here we pass the desired field name we wish to populate
    csvCalculatedColumns.add(new MathColCal("Math"));
    csvMessageChannelFactory.setCalculatedColumns("SaCvaVegaRiskWeight", "SaCvaVegaRiskWeight", csvCalculatedColumns);    // setCalculatedColumns takes params: Topic, Store, calculatedColumns

    return csvMessageChannelFactory;
}

@Override
    public CSVMessageChannelFactory<Path> incrLoadCsvMessageChannelFactory(String sourceName) {

    final CSVMessageChannelFactory<Path> csvMessageChannelFactory = new CSVMessageChannelFactory<>(
            incrLoadCsvSource(sourceName), datastoreConfig.datastore());

    final List<IColumnCalculator<ILineReader>> csvCalculatedColumns = new ArrayList<>();
    csvCalculatedColumns.add(new FileNameColCal("FileName"));                                    // Here we pass the desired field name we wish to populate
    csvCalculatedColumns.add(new MathColCal("Math"));
    csvMessageChannelFactory.setCalculatedColumns("SaCvaVegaRiskWeight", "SaCvaVegaRiskWeight", csvCalculatedColumns);   // setCalculatedColumns takes params: Topic, Store, calculatedColumns

    return csvMessageChannelFactory;
    }
}

Step 4 - Application Config

Replace SaCsvSourceConfig with ExtendedSaCsvSourceConfig so that our changes can be picked up in ApplicationConfig located in cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/ApplicationConfig.java

        // data sourcing
        SharedCsvSourceConfig.class,
        SharedAzureCsvSourceConfig.class,
        SharedAwsCsvSourceConfig.class,
        SharedGoogleCsvSourceConfig.class,

//      SaCsvSourceConfig.class,                        // here we comment out the original Source Config
        ExtendedSaCsvSourceConfig.class,                // and replace with our Extended Source Config
        SaAzureCsvSourceConfig.class,
        SaAwsCsvSourceConfig.class,
        SaGoogleCsvSourceConfig.class,