☰
Navigation :

Adding a Field

This page provides a description of how to add new fields to CVARC project. It features an example in which we will add ‘TestField’ to Atoti CVA Risk Capital. We will also explore additional steps we need to take for adding to stores which are populated via a Tuple Publisher. The techniques employed are generic examples that can be extended, adapted and repeated for any use case that we need. In all cases, we make minimal changes to the Reference Implementation.

Editing the Data

For this example we have modified files which are currently populating the Vega store (files which match a naming scheme of sa-cva-cva-vega-sensitivities.csv) by adding a column to them, in this case our new column’s header is ‘TestField’, below is an example of the contents of one such file. (This new column was added to all files matching such naming scheme).

AsOfDate NettingSetId RiskClass RiskFactorId Sensitivity SensitivityCcy ReferenceName ExplicitBucket BucketSuffix TestField
2018-12-05 ActiveBank EU_10 commodity whey_Implied volatility -12352.9 EUR whey Data1
2018-12-05 ActiveBank EU_101 interest rate CHF.IBOR3M_Implied volatility 7896.9 EUR CHF.IBOR3M Data2
2018-12-05 ActiveBank EU_12 foreign exchange CAD_ATM Implied volatility 5068.05 EUR CAD Data3
2018-12-05 ActiveBank EU_12 interest rate CHF.IBOR3M_Implied volatility 4782.93 EUR CHF.IBOR3M Data4
2018-12-05 ActiveBank EU_12 commodity wheat_Implied volatility 32805.05 EUR wheat Data5

Step 1 - Define Customisations to Data Store via Datastore Helper

It would be possible to add these attributes to several stores, but for simplicity in this example we will just add the field to the Vega Store. Define data store customizations inside of loadcustomisations() method inside DatastoreCustomisations class located /cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/DatastoreCustomisations.java

See more customizations achievable with Datastore Helper.

public class DatastoreCustomisations implements IDatastoreCustomisations {
    @Override
    public void loadCustomisations() {
        DatastoreHelper.appendField("Vega", new CustomField("TestField", ILiteralType.STRING));  // Append "Vega" store with a new field "TestField" of type STRING
    }
}

Step 2 - Source Config / ETL Code

Modify the appropriate CsvSourceConfig in this case SaCsvSourceConfig by creating a new class inside of /cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/ which extends SaCsvSourceConfig

  1. Modify the method which generates a list for the column names.
  2. Keep in mind if there are any tuple publisher which manipulate data before it’s published to the store. If so, override those methods and modify them to make sure our new field is published.
@Configuration
public abstract class ExtendedSaCsvSourceConfig extends SaCsvSourceConfig {

    //1. Modify column names
    @Override
    protected List<String> generateCvaVegaCsvFields() {
        List<String> modifiedCsvFields = super.generateCvaVegaCsvFields();
        modifiedCsvFields.add("TestField");
        return modifiedCsvFields;
    }


    /**
     * 2. If there is a tuple publisher for the store you are modifying then the below method is necessary so that our new tuple
     * publisher which we will be used. In this case ExtendedVegaCVASensitivitiesPublisherWrapper will need to be defined.
     */
    @Override
    public Map<String, com.quartetfs.fwk.impl.Pair<List<String>, ITuplePublisher>> createTopicChannelsMap() {

        Map<String, com.quartetfs.fwk.impl.Pair<List<String>, ITuplePublisher>> extendedTopicChannels = super.createTopicChannelsMap();

        ITuplePublisher vegaCvaPublisher = new ExtendedVegaCVASensitivitiesPublisherWrapper(datastoreConfig.datastore(), generateCvaVegaToBaseStoreMapping());

        extendedTopicChannels.put(CSV_TOPIC__CVA_VEGA_SENSITIVITIES, new Pair(vegaCvaPublisher.getTargetStores(),vegaCvaPublisher));

        return extendedTopicChannels;
    }
}

Step 3 - Source Config cont. Tuple Publisher

Below is an example of the modified publisher method ExtendedVegaCVASensitivitiesPublisherWrapper, here we have copied the logic in VegaCVASensitivitiesPublisherWrapper and we have added handling of ‘TestField’ by appending our subTuple.

public class ExtendedVegaCVASensitivitiesPublisherWrapper extends VegaCVASensitivitiesPublisherWrapper {
    protected RiskFactorCatalogUtils riskFactorCatalogUtils = new RiskFactorCatalogUtils();
    protected static Logger logger = LoggerFactory.getLogger(VegaCVASensitivitiesPublisherWrapper.class.getName());
    /**
     * @param dataStore           {@link IDatastore}
     * @param saBaseHeaderMapping Mapping of the base store field names corresponding to the index
     *                            which they are found at inside the incoming sensitivity file.
     */
    public ExtendedVegaCVASensitivitiesPublisherWrapper(IDatastore dataStore, HashMap<String, Integer> saBaseHeaderMapping) {
        super(dataStore, saBaseHeaderMapping);
        // Instantiate SaBaseStore Publisher
        this.saBaseStorePublisher = new SaBaseStorePublisher(dataStore, SAStoreNames.SA_BASE_STORE_NAME,
                SharedDataStoreNames.VEGA);
    }
    @Override
    public void publish(List<Object[]> tuples) {
        List<Object[]> storeTuples = new ArrayList<>();
        tuples.stream().forEach(tuple -> {
            LocalDate asOfDate = (LocalDate)tuple[0];
            String riskClass = ((String)tuple[3]).toLowerCase();

            Object[] subTuple = new Object[] { tuple[0], // AsOfDate
                    tuple[1], // NettingSetId
                    null, // TradeId
                    ((String) tuple[3]).toLowerCase(), // RiskClass
                    tuple[4], // RiskFactorId
                    RiskMeasureEnum.Vega.toString().toLowerCase(), // RiskMeasure
                    tuple[6], // Sensitivities
                    tuple[7], // SensitivitiesCcy
                    null,
                    null,
                    null,
                    null,
                    null,
                    tuple[13], // NettingSetTradeId
                    tuple[14] // This tuple was added to the subtuple so that our TestField will be published to the store
            };
            // publish to SA Base store
            storeTuples.add(subTuple);
        });
        dataStore.getTransactionManager().addAll(SAStoreNames.VEGA_STORE_NAME, storeTuples);
    }
}

Step 4 - Application Config

Replace SaCsvSourceConfig with ExtendedSaCsvSourceConfig so that our changes can be picked up in ApplicationConfig located in cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/ApplicationConfig.java

        // data sourcing
        SharedCsvSourceConfig.class,
        SharedAzureCsvSourceConfig.class,
        SharedAwsCsvSourceConfig.class,
        SharedGoogleCsvSourceConfig.class,

//      SaCsvSourceConfig.class,                        // here we comment out the original Source Config
        ExtendedSaCsvSourceConfig.class,                // and replace with our Extended Source Config
        SaAzureCsvSourceConfig.class,
        SaAwsCsvSourceConfig.class,
        SaGoogleCsvSourceConfig.class,

To Expose fields as Hierarchies or Measures please see example in Adding a Hierarchy and Adding a new Measure.