Configuring sources using Spring Beans and properties
This page describes the source topic descriptions defined in the application, and provides an example of adding customization.
Source Topic Descriptions
The Data Load Controller is used to define the source topics. The following descriptions are defined:
Name | File pattern | Columns (if explicitly defined)* | Target description name | Custom fields | Configuration class |
---|---|---|---|---|---|
Countries | glob:**Countries*.csv |
CommonStoreCsvLoadConfig | |||
CounterpartyParentChild | glob:**CounterpartyParentChild*.csv |
CommonStoreCsvLoadConfig | |||
Counterparties | glob:**Counterparties*.csv |
CommonStoreCsvLoadConfig | |||
TradeAttributes | glob:**TradeAttributes*.csv |
CommonStoreCsvLoadConfig | |||
RiskFactorsCatalogue | glob:**RiskFactorsCatalog*.csv |
CommonStoreCsvLoadConfig | |||
Scenarios | glob:**Scenarios*.csv |
CommonStoreCsvLoadConfig | |||
LegalEntityParentChild | glob:**LegalEntityParentChild*.csv |
CommonStoreCsvLoadConfig | |||
BookParentChild | glob:**BookParentChild*.csv |
CommonStoreCsvLoadConfig | |||
MarketShifts | glob:**MarketShifts*.csv |
CommonStoreCsvLoadConfig | |||
PnL | glob:**PLPC*.csv |
AsOfDate TradeId Daily Monthly Yearly Lifetime Type PLDriver IsFullReval Ccy MarketDataSet Bucket |
PnL | TradeKey | PnLCsvLoadConfig |
DynamicTenors | glob:**DynamicTenors*.csv |
TenorLabel NumberOfDays SensitivityName TenorSet |
DynamicTenors | TenorIndices | SensiCsvLoadConfig |
DynamicMaturities | glob:**DynamicMaturities*.csv |
MaturityLabel NumberOfDays SensitivityName MaturitySet |
DynamicMaturities | MaturityIndices | SensiCsvLoadConfig |
DynamicMoneyness | glob:**DynamicMoneyness*.csv |
MoneynessLabel Shift SensitivityName MoneynessSet |
DynamicMoneyness | MoneynessIndices | SensiCsvLoadConfig |
CorrelationMarketData | glob:**Correlation_Market_Data*.csv |
SensiCsvLoadConfig | |||
DividendMarketData | glob:**Dividends*.csv |
SensiCsvLoadConfig | |||
SplitRatioMarketData | glob:**SplitRatio*.csv |
SensiCsvLoadConfig | |||
SensiLadders | glob:**LadderDefinition*.csv |
SensiCsvLoadConfig | |||
Delta | glob:**DeltaSensitivities*.csv |
AsOfDate TradeId SensitivityName RiskClass MarketDataSet RiskFactorId TenorLabel TenorDate MaturityLabel MaturityDate Moneyness Value Ladder Ccy |
Delta | TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed | SensiCsvLoadConfig |
Correlation | glob:**CorrelationSensitivities*.csv |
AsOfDate TradeId SensitivityName RiskClass MarketDataSet RiskFactorId RiskFactorId2 TenorLabel TenorDate MaturityLabel MaturityDate Moneyness Value Ladder Ccy |
Correlation | TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed | SensiCsvLoadConfig |
CrossGamma | glob:**CrossGammaSensitivities*.csv |
AsOfDate TradeId SensitivityName RiskClass MarketDataSet RiskFactor RiskFactor2 TenorLabel TenorDate MaturityLabel MaturityDate Moneyness Value Ladder Ccy |
CrossGamma | TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed | SensiCsvLoadConfig |
Volga | glob:**VolgaSensitivities*.csv |
AsOfDate TradeId SensitivityName RiskClass MarketDataSet RiskFactorId TenorLabel TenorDate MaturityLabel MaturityDate Moneyness Value Ladder Ccy |
Volga | TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed | SensiCsvLoadConfig |
Vanna | glob:**VannaSensitivities*.csv |
AsOfDate TradeId SensitivityName RiskClass MarketDataSet RiskFactorId RiskFactorId2 TenorLabel TenorDate MaturityLabel MaturityDate Moneyness Value Ladder Ccy |
Vanna | TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed | SensiCsvLoadConfig |
Vega | glob:**VegaSensitivities*.csv |
AsOfDate TradeId SensitivityName RiskClass MarketDataSet RiskFactorId TenorLabel TenorDate MaturityLabel MaturityDate Moneyness Value Ladder Ccy |
Vega | TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed | SensiCsvLoadConfig |
Theta | glob:**ThetaSensitivities*.csv |
AsOfDate TradeId SensitivityName RiskClass MarketDataSet RiskFactorId TenorLabel TenorDate MaturityLabel MaturityDate Moneyness Value Ladder Ccy |
Theta | TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed | SensiCsvLoadConfig |
Gamma | regex:(?i).*Gamma(?<!CrossGamma)Sensitivities.*\.csv.* |
AsOfDate TradeId SensitivityName RiskClass MarketDataSet RiskFactorId TenorLabel TenorDate MaturityLabel MaturityDate Moneyness Value Ladder Ccy |
Gamma | TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed | SensiCsvLoadConfig |
SpotMarketData | glob:**Spot_Market_Data*.csv |
SpotMarketDataCsvLoadConfig | |||
SurfaceMarketData | glob:**Surface_Market_Data*.csv |
SurfaceMarketDataCsvLoadConfig | |||
CurveMarketData | glob:**Curve_Market_Data*.csv |
CurveMarketDataCsvLoadConfig | |||
CubeMarketData | glob:**Cube_Market_Data*.csv |
CubeMarketDataCsvLoadConfig | |||
FxRateMarketData | glob:**FX_Rate_Market_Data*.csv |
FxRateMarketDataCsvLoadConfig | |||
TradePnLs | glob:**TradePnLs*.csv |
AsOfDate TradeId ScenarioSet CalculationId MarketDataSet RiskFactor RiskClass SensitivityName Ccy MTM PnLVector |
TradePnLs | TradeKey | VaRCsvLoadConfig |
SensiBaseStore | glob:**Sensitivity Cube*.csv* |
SensiSummaryCsvLoadConfig | |||
BaseStore | glob:**VaR-ES Cube*.csv* |
VaRSummaryCsvLoadConfig | |||
PnLBaseStore | glob:**PLCube*.csv* |
PnLSummaryCsvLoadConfig |
*If no fields are present in the configuration, the datastore’s target store fields are taken into account.
Example
For the purposes of this example, we will add a column to the file
DynamicTenors.csv
and load the data from this column into the
DynamicTenors
store. In this case, our new column’s header is “TestField”,
and all data in this column is “Testdata”. Here’s how the input file looks like:
TenorSet | TenorLabel | SensitivityName | NumberOfDays | TestField |
---|---|---|---|---|
DEFAULT | N/A | N/A | 0.0 | Testdata |
DEFAULT | 0.25Y | N/A | 90.0 | Testdata |
DEFAULT | 0.5Y | N/A | 180.0 | Testdata |
DEFAULT | 1Y | N/A | 360.0 | Testdata |
DEFAULT | 2Y | N/A | 720.0 | Testdata |
DEFAULT | 3Y | N/A | 1080.0 | Testdata |
DEFAULT | 5Y | N/A | 1800.0 | Testdata |
DEFAULT | 10Y | N/A | 3600.0 | Testdata |
DEFAULT | 15Y | N/A | 5400.0 | Testdata |
DEFAULT | 20Y | N/A | 7200.0 | Testdata |
REDUCED | N/A | N/A | 0.0 | Testdata |
REDUCED | 0.5Y | N/A | 180.0 | Testdata |
REDUCED | 1Y | N/A | 360.0 | Testdata |
REDUCED | 2Y | N/A | 720.0 | Testdata |
REDUCED | 30Y | N/A | 10800.0 | Testdata |
DECADE | 10Y | N/A | 3600.0 | Testdata |
DECADE | 20Y | N/A | 7200.0 | Testdata |
We will also attach a column calculator that multiplies the NumberOfDays by 2 into a new column, as well as a tuple publisher that will add a value into the SensitivityName field.
Step 1 - Defining customizations to datastore
Before we can load these new columns into our cube, we need to make sure that our datastore has fields that can accept them.
To add new fields to an existing store, we need to create a DatastoreConfiguratorConsumer
bean, which appends the required fields to the DynamicTenors
store:
public static String DOUBLE_NUMBER_OF_DAYS = "DoubleNumberOfDays";
public static String TEST_FIELD = "TestData";
@Bean
public DatastoreConfiguratorConsumer addFieldToStore() {
return configurator -> {
configurator.appendFields(DYNAMIC_TENOR_STORE_NAME,
List.of(
new CustomField(DOUBLE_NUMBER_OF_DAYS, ILiteralType.DOUBLE),
new CustomField(TEST_FIELD, ILiteralType.STRING)
)
);
};
}
See Customizing the Datastore with the Datastore Helper for more information.
Step 2 - Implementing the column calculator and the tuple publisher
Now that we have modified our store, we need to make sure the ETL is correctly set up, so that the field will be properly populated.
In most cases, the topic configuration uses the store fields as the expected file columns, which would require no further configuration.
For the DynamicTenors
store used in this example, file columns are defined explicitly, so any new columns are not auto-configured.
The DynamicTenors store also has a previously defined column calculator creating indices for each of the tenors.
To define the column calculator, we create the following bean, containing the required logic:
@Bean
public CustomFieldDescription dynamicTenorsColumnCalculator() {
return CustomFieldDescription.of("MyCustomDynamicTenorColCalculator",
scope -> new LambdaCalculator(DOUBLE_NUMBER_OF_DAYS,
context -> {
Double originalValue = (Double) context.getValue(StoreFieldConstants.TENOR_NUMBER_OF_DAYS);
if (originalValue == null) {
return null;
}
return 2.0 * originalValue;
}
)
);
}
For the tuple publisher, a second bean is created, defining the publishing logic:
@Bean
public TargetDescription dynamicTenorsTarget(IDatastore datastore) {
return TargetDescription.of("MyCustomDynamicTenors", DYNAMIC_TENOR_STORE_NAME,
scope ->
new ITuplePublisher<>() {
@Override public void publish(IStoreMessage<?, ?> message, List<Object[]> tuples) {
// We use the expected index for the SensitivityName field here, while in a complete implementation the index would be an instance variable.
tuples.forEach(tuple -> tuple[2] = "TestValue");
datastore.getTransactionManager().addAll(StoreConstants.DYNAMIC_TENOR_STORE_NAME, tuples);
}
@Override
public Collection<String> getTargetStores() {
return Collections.singleton(DYNAMIC_TENOR_STORE_NAME);
}
});
}
Step 3 - Data Load Controller Properties
We add the following properties to our existing properties,to define:
- the list of columns read from the input file,
- the calculated column,
- the tuple publisher.
csv:
topics:
DynamicTenors:
file-pattern: "glob:**DynamicTenors**.csv"
parser:
columns:
- TenorLabel
- NumberOfDays
- SensitivityName
- TenorSet
- TestData
channels:
- target: MyCustomDynamicTenors
custom-fields:
- MyCustomDynamicTenorColCalculator
- TenorIndices
Step 4 - Configuration classes
You can include the Beans detailed above in a single class within Atoti Market Risk.
@Configuration
public class CustomizationsConfig {
public static String DOUBLE_NUMBER_OF_DAYS = "DoubleNumberOfDays";
public static String TEST_FIELD = "TestData";
@Bean
public DatastoreConfiguratorConsumer addFieldToStore() {
return configurator -> {
configurator.appendFields(DYNAMIC_TENOR_STORE_NAME,
List.of(
new CustomField(DOUBLE_NUMBER_OF_DAYS, ILiteralType.DOUBLE),
new CustomField(TEST_FIELD, ILiteralType.STRING)
)
);
};
}
@Bean
public CustomFieldDescription dynamicTenorsColumnCalculator() {
return CustomFieldDescription.of("MyCustomDynamicTenorColCalculator",
scope -> new LambdaCalculator(DOUBLE_NUMBER_OF_DAYS,
context -> {
Double originalValue = (Double) context.getValue(StoreFieldConstants.TENOR_NUMBER_OF_DAYS);
if (originalValue == null) {
return null;
}
return 2.0 * originalValue;
}
)
);
}
@Bean
public TargetDescription dynamicTenorsTarget(IDatastore datastore) {
return TargetDescription.of("MyCustomDynamicTenors", DYNAMIC_TENOR_STORE_NAME,
scope ->
new ITuplePublisher<>() {
@Override public void publish(IStoreMessage<?, ?> message, List<Object[]> tuples) {
// We use the expected index for the SensitivityName field here, while in a complete implementation the index would be an instance variable.
tuples.forEach(tuple -> tuple[2] = "TestValue");
datastore.getTransactionManager().addAll(StoreConstants.DYNAMIC_TENOR_STORE_NAME, tuples);
}
@Override
public Collection<String> getTargetStores() {
return Collections.singleton(DYNAMIC_TENOR_STORE_NAME);
}
});
}
}
Then, include this configuration class in the @Import
annotation of the MarketRiskConfig
class:
@Import(value = {
...
CustomizationsConfig.class
})
public class MarketRiskConfig {
...
}
Importing this customization class and modifying the input file will result in the following data being loaded into the store:
TenorSet | TenorLabel | SensitivityName | NumberOfDays | TenorIndices | DoubleNumberOfDays | TestData |
---|---|---|---|---|---|---|
DEFAULT | N/A | TestValue | 0.0 | 0 | 0.0 | Testdata |
DEFAULT | 0.25Y | TestValue | 90.0 | 1 | 180.0 | Testdata |
DEFAULT | 0.5Y | TestValue | 180.0 | 2 | 360.0 | Testdata |
DEFAULT | 1Y | TestValue | 360.0 | 3 | 720.0 | Testdata |
DEFAULT | 2Y | TestValue | 720.0 | 4 | 1440.0 | Testdata |
DEFAULT | 3Y | TestValue | 1080.0 | 5 | 2160.0 | Testdata |
DEFAULT | 5Y | TestValue | 1800.0 | 6 | 3600.0 | Testdata |
DEFAULT | 10Y | TestValue | 3600.0 | 7 | 7200.0 | Testdata |
DEFAULT | 15Y | TestValue | 5400.0 | 8 | 10800.0 | Testdata |
DEFAULT | 20Y | TestValue | 7200.0 | 9 | 14400.0 | Testdata |
REDUCED | N/A | TestValue | 0.0 | 0 | 0.0 | Testdata |
REDUCED | 0.5Y | TestValue | 180.0 | 1 | 360.0 | Testdata |
REDUCED | 1Y | TestValue | 360.0 | 2 | 720.0 | Testdata |
REDUCED | 2Y | TestValue | 720.0 | 3 | 1440.0 | Testdata |
REDUCED | 30Y | TestValue | 10800.0 | 4 | 21600.0 | Testdata |
DECADE | 10Y | TestValue | 3600.0 | 0 | 7200.0 | Testdata |
DECADE | 20Y | TestValue | 7200.0 | 1 | 14400.0 | Testdata |
The ATableFormatTuplePublisher
For use cases where the destination table doesn’t match the input file format, we provide the ATableFormatTuplePublisher
.
Any extension of this publisher can be constructed with the file tuple containing a subset of the table fields, or with an explicit mapping between table fields and file columns.
Example
For the backwards-compatible Market Data API sources, the stores should be loaded from the deprecated file format. For SpotMarketData
, when loading from the MarketData.csv file, the following applies:
File Field | Table Field | Behavior |
---|---|---|
AsOfDate | AsOfDate | Loaded as-is |
MarketDataSet | MarketDataSet | Loaded as-is |
RiskFactorId | InstrumentId | Mapped by the publisher |
Quote | Quote | Loaded as-is |
TenorLabels | - | Must be empty |
MaturityLabels | - | Must be empty |
MoneynessLabels | - | Must be empty |
TenorDates | - | Must be empty |
MaturityDates | - | Must be empty |
Nominal | - | Must be empty |
The configuration for this behavior is:
new ATableFormatTuplePublisher<>(datastore, Map.of(INSTRUMENT_ID, RISK_FACTOR)) {
@Override
public void publish(IStoreMessage<? extends I, ?> message, List<Object[]> tuples) {
Queue<Object[]> storeTuples = new ConcurrentLinkedQueue<>();
tuples.forEach(
tuple -> {
if (checkNull(tmmProps, message, tuple, TENOR_LABELS, TENOR_DATES, MATURITY_LABELS, MATURITY_DATES, MONEYNESS_LABELS,
NOMINAL)) {
Queue<Object[]> extractedScalarTuples = new ConcurrentLinkedQueue<>();
devectorizer.extractScalarTuples(message, tuple, extractedScalarTuples);
extractedScalarTuples.forEach(
scalarTuple -> storeTuples.add(createTupleForTable(message, SPOT_MARKET_DATA_STORE, scalarTuple))
);
}
}
);
datastore.getTransactionManager().addAll(SPOT_MARKET_DATA_STORE, storeTuples);
}
@Override
public Collection<String> getTargetStores() {
return Set.of(SPOT_MARKET_DATA_STORE);
}
}
Within the createTupleForTable
method, the publisher iterates over the table fields, attempting to retrieve a value in the input tuple by the mapped field (or directly if no mapping is available).
If no value is available in the input tuple, or no matching field is found, the value will be empty. If a tuple field’s value isn’t requested by the publisher, it is ignored.
Suggested Further Reading
- Adding cube hierarchies
- Configuring measures using Spring Beans
- Configuring schema selections using Spring Beans
- Adding a new KPI
- Adding data loading or unloading topics