Adding a File
This page provides a description of how to add new files to Atoti CVA Risk Capital in a new datastore. The techniques employed are generic examples that can be extended, adapted, and repeated for any use case that we need. In all cases, we make minimal changes to the Reference Implementation.
Step 1 - Moving the Data
Move any custom data to an appropriate folder in
/cvarc-starter/src/test/resources/data-samples/data
.
This example moves the data (shown below) into the /2018-12-05
folder.
123_Custom_1.csv
AsOfDate | RiskClass | CustomProjected | UselessData |
---|---|---|---|
2018-12-05 | commodity | 340000 | x |
2018-12-05 | interest rate | 2000 | a |
987_Custom_2.csv
AsOfDate | RiskClass | CustomProjected | UselessData |
---|---|---|---|
2018-12-05 | foreign exchange | 12341234 | z |
2018-12-05 | equity | 222222222 | f |
note
UselessData
should be ignored.
Step 2 - Define Customisations
Append the Datastore Model Config by creating a class that extends the appropriate Datastore Model Config, in this case SADatastoreModelConfig
:
- Within this class, define the name of the new store and the fields it will contain, by making a method that adds a store builder to the IDatastoreConfigurator.
- Add a store reference to the configurator by using the Reference builder.
- Call the method building our new store in createStores(IDatastoreConfigurator configurator) method.
@Configuration
public class ExtendedSADatastoreModelConfig extends SADatastoreModelConfig {
// 1. This will be all the fields you wish to store in your cube for that particular store.
public static void customStoreDescription(IDatastoreConfigurator configurator){
configurator.addStore("Schema", StartBuilding.store()
.withStoreName("Custom")
.withField("AsOfDate", LOCAL_DATE).asKeyField()
.withField("RiskClass", STRING).asKeyField()
//asKeyField() denotes a field(s) that can uniquely identify data, key(s) should not resolve to multiple peices of data
.withField("CustomProjected", DOUBLE)
//You can have unlimited numbers of Fields by stringing .withField methods before the .build() method
// We wont want to add any fields we want to ignore here
.build());
}
// 2. appended reference list
@Override
public static void createReferences(IDatastoreConfigurator configurator){
configurator
.addReference("Schema", configurator.referenceBuilder(CVARCParameters.CVA_SA_SCHEMA_NAME)
.fromStore(SAStoreNames.SA_BASE_STORE_NAME)
.toStore(SAStoreNames.SA_CARVED_NETTING_SET_STORE_NAME)
.withName(SAStoreNames.SA_BASE_TO_CARVED_NETTING_SET_REF)
.withMapping(SAStoreNames.SA_BASE_AS_OF_DATE, SAStoreNames.SA_CARVED_NETTING_SET_AS_OF_DATE).build());
}
public static void createStores(IDatastoreConfigurator configurator) {
customStoreDescription(configurator);
}
}
Step 3 - Application Config
Replace SADatastoreModelConfig
with ExtendedSADatastoreModelConfig
so that your changes can be picked up in the ApplicationConfig
class located in
/cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/ApplicationConfig.java
@Configuration
@Import(value = {
// data model
BADatastoreModelConfig.class,
// SADatastoreModelConfig.class, // here we comment out our Datastore Model Config
ExtendedSADatastoreModelConfig.class, // and in its place write our extended one
DatastoreConfig.class,
OlapModelConfig.class,
Replace SADatastoreModelConfig with ExtendedSADatastoreModelConfig so that your changes can be picked up in DatastoreConfig class located in cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/DatastoreConfig.java
@Configuration
@Profile("!" + CVARCEnvConstants.SP_PROFILE__QUERY_NODE)
@Import(value = {
BADatastoreModelConfig.class,
// SADatastoreModelConfig.class, // here we comment out the original Datastore Model Config
ExtendedSADatastoreModelConfig.class, // and in its place write our extended one
})
Step 4 - Source Config / ETL Code
Modify the appropriate csv source config. In this case SaCsvSourceConfig
, by creating a new class inside of /cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/
, which extends SaCsvSourceConfig
.
- Define the csv fields of your file.
- Modify your bulkload and incremental load.
- Lastly, append the topic channels map.
public abstract class ExtendedSaCsvSourceConfig extends SaCsvSourceConfig {
// 1. Define CSV fields
protected List<String> generateCustomCsvFields(){
List<String> customCvaFields = new ArrayList<>();
customCvaFields.add("AsOfDate");
customCvaFields.add("RiskClass");
customCvaFields.add("CustomProjected");
customCvaFields.add("UselessData");
return customCvaFields;
}
// 2. defining topic for bulk load and incremental load
@Override
public CsvScopedFetchSource bulkLoadCsvSource(String sourceName) {
final CsvScopedFetchSource<Path> source = super.bulkLoadCsvSource(sourceName);
source.addTopic(
createDirectoryTopic(
source, "Custom", "./src/test/resources/data-samples" , "/data",
"glob:**/*Custom*.csv", source.createParserConfiguration(generateCustomCsvFields(), 1)
// the int 1 is passed into the createParserConfiguration method to denote that our topic should skip the first line of our source
)
);
return source;
}
@Override
public CsvScopedFetchSource incrLoadCsvSource(String sourceName) {
final CsvScopedFetchSource<Path> source = super.incrLoadCsvSource(sourceName);
source.addTopic(
createDirectoryTopic(
source, "Incremental-Custom", "./src/test/resources/data-samples", "/data",
"glob:**/*Custom*.csv", source.createParserConfiguration(generateCustomCsvFields(), 1)
)
);
return source;
}
// 3. Appending Topic Channel Map
@Override
public Map<String, Pair<List<String>, ITuplePublisher>> createTopicChannelsMap() {
Map<String, Pair<List<String>, ITuplePublisher>> extendedTopicChannels = super.createTopicChannelsMap();
extendedTopicChannels.put("Custom", new Pair(Arrays.asList("Custom"),
new TuplePublisher<IFileInfo<Path>>(datastoreConfig.datastore(),
"Custom")));
return extendedTopicChannels;
}
}
Step 5 - Application Config
Replace SaCsvSourceConfig
with ExtendedSaCsvSourceConfig
, so that your changes can be picked up in ApplicationConfig
located in
cvarc-starter/src/main/java/com/activeviam/cvarc/starter/cfg/impl/ApplicationConfig.java
// data sourcing
SharedCsvSourceConfig.class,
SharedAzureCsvSourceConfig.class,
SharedAwsCsvSourceConfig.class,
SharedGoogleCsvSourceConfig.class,
// SaCsvSourceConfig.class, // here we comment out the original Source Config
ExtendedSaCsvSourceConfig.class, // and replace with our Extended Source Config
SaAzureCsvSourceConfig.class,
SaAwsCsvSourceConfig.class,
SaGoogleCsvSourceConfig.class,
Step 6 - Data Load Controller
Append the list of topics that get registered in DataLoadControllerConfig
to include the new ‘Custom’ Topic so that your new data can be loaded into the Solution.
//--SA data
final List<String> saDataTopics = new ArrayList<>(
Arrays.asList(
saCsvSource.CSV_TOPIC__CVA_DELTA_SENSITIVITIES,
saCsvSource.CSV_TOPIC__HEDGED_DELTA_SENSITIVITIES,
saCsvSource.CSV_TOPIC__CVA_VEGA_SENSITIVITIES,
saCsvSource.CSV_TOPIC__HEDGED_VEGA_SENSITIVITIES,
saCsvSource.CSV_TOPIC__INTEREST_RATE_DELTA_RISK_WEIGHT,
saCsvSource.CSV_TOPIC__COUNTERPARTY_CREDIT_SPREAD_DELTA_RISK_WEIGHT,
saCsvSource.CSV_TOPIC__REFERENCE_CREDIT_DELTA_RISK_WEIGHT,
saCsvSource.CSV_TOPIC__EQUITY_DELTA_RISK_WEIGHT,
//jdbcSourceConfig.JDBC_TOPIC__EQUITY_DELTA_RISK_WEIGHT,//Register JDBC Topic
saCsvSource.CSV_TOPIC__COMMODITY_DELTA_RISK_WEIGHT,
saCsvSource.CSV_TOPIC__INTEREST_RATE_DELTA_CORRELATION,
saCsvSource.CSV_TOPIC__COUNTERPARTY_CREDIT_SPREAD_DELTA_CROSS_BUCKET_CORRELATION,
saCsvSource.CSV_TOPIC__REFERENCE_CREDIT_CROSS_BUCKET_CORRELATION,
saCsvSource.CSV_TOPIC__SA_CVA_VEGA_RISK_WEIGHT,
"Custom" // here we add one more element to the saDataTopics List Custom
));
controller.registerTopicAlias(TOPIC__SA_DATA, saDataTopics);
}
To expose fields as hierarchies or measures, see the examples in Adding a Hierarchy and Adding New Cube Measures.