Configuring sources using Spring Beans and properties

Atoti Market Risk defines source topic descriptions that control how data is loaded into each store. The example below demonstrates how to add a custom source using Spring Beans and properties.

Source Topic Descriptions

The Data Load Controller is used to define the source topics. The following descriptions are defined:

Name	File pattern	Columns (if explicitly defined)*	Target description name	Custom fields	Configuration class
Countries	`glob:*Countries.csv`				CommonStoreCsvLoadConfig
CounterpartyParentChild	`glob:*CounterpartyParentChild.csv`				CommonStoreCsvLoadConfig
Counterparties	`glob:*Counterparties.csv`				CommonStoreCsvLoadConfig
TradeAttributes	`glob:*TradeAttributes.csv`				CommonStoreCsvLoadConfig
RiskFactorsCatalogue	`glob:*RiskFactorsCatalog.csv`				CommonStoreCsvLoadConfig
Scenarios	`glob:*Scenarios.csv`				CommonStoreCsvLoadConfig
LegalEntityParentChild	`glob:*LegalEntityParentChild.csv`				CommonStoreCsvLoadConfig
BookParentChild	`glob:*BookParentChild.csv`				CommonStoreCsvLoadConfig
MarketShifts	`glob:*MarketShifts.csv`				CommonStoreCsvLoadConfig
PnL	`glob:*PLPC.csv`	AsOfDate TradeId Daily Monthly Yearly Lifetime Type PLDriver IsFullReval Ccy MarketDataSet Bucket	PnL	TradeKey	PnLCsvLoadConfig
DynamicTenors	`glob:*DynamicTenors.csv`	TenorLabel NumberOfDays SensitivityName TenorSet	DynamicTenors	TenorIndices	SensiCsvLoadConfig
DynamicMaturities	`glob:*DynamicMaturities.csv`	MaturityLabel NumberOfDays SensitivityName MaturitySet	DynamicMaturities	MaturityIndices	SensiCsvLoadConfig
DynamicMoneyness	`glob:*DynamicMoneyness.csv`	MoneynessLabel Shift SensitivityName MoneynessSet	DynamicMoneyness	MoneynessIndices	SensiCsvLoadConfig
CorrelationMarketData	`glob:*Correlation_Market_Data.csv`				SensiCsvLoadConfig
DividendMarketData	`glob:*Dividends.csv`				SensiCsvLoadConfig
SplitRatioMarketData	`glob:*SplitRatio.csv`				SensiCsvLoadConfig
SensiLadders	`glob:*LadderDefinition.csv`				SensiCsvLoadConfig
Delta	`glob:*DeltaSensitivities.csv`	AsOfDate TradeId SensitivityName RiskClass MarketDataSet RiskFactorId TenorLabel TenorDate MaturityLabel MaturityDate Moneyness Value Ladder Ccy	Delta	TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed	SensiCsvLoadConfig
Correlation	`glob:*CorrelationSensitivities.csv`	AsOfDate TradeId SensitivityName RiskClass MarketDataSet RiskFactorId RiskFactorId2 TenorLabel TenorDate MaturityLabel MaturityDate Moneyness Value Ladder Ccy	Correlation	TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed	SensiCsvLoadConfig
CrossGamma	`glob:*CrossGammaSensitivities.csv`	AsOfDate TradeId SensitivityName RiskClass MarketDataSet RiskFactor RiskFactor2 TenorLabel TenorDate MaturityLabel MaturityDate Moneyness Value Ladder Ccy	CrossGamma	TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed	SensiCsvLoadConfig
Volga	`glob:*VolgaSensitivities.csv`	AsOfDate TradeId SensitivityName RiskClass MarketDataSet RiskFactorId TenorLabel TenorDate MaturityLabel MaturityDate Moneyness Value Ladder Ccy	Volga	TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed	SensiCsvLoadConfig
Vanna	`glob:*VannaSensitivities.csv`	AsOfDate TradeId SensitivityName RiskClass MarketDataSet RiskFactorId RiskFactorId2 TenorLabel TenorDate MaturityLabel MaturityDate Moneyness Value Ladder Ccy	Vanna	TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed	SensiCsvLoadConfig
Vega	`glob:*VegaSensitivities.csv`	AsOfDate TradeId SensitivityName RiskClass MarketDataSet RiskFactorId TenorLabel TenorDate MaturityLabel MaturityDate Moneyness Value Ladder Ccy	Vega	TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed	SensiCsvLoadConfig
Theta	`glob:*ThetaSensitivities.csv`	AsOfDate TradeId SensitivityName RiskClass MarketDataSet RiskFactorId TenorLabel TenorDate MaturityLabel MaturityDate Moneyness Value Ladder Ccy	Theta	TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed	SensiCsvLoadConfig
Gamma	`regex:(?i).Gamma(?<!CrossGamma)Sensitivities.\.csv.*`	AsOfDate TradeId SensitivityName RiskClass MarketDataSet RiskFactorId TenorLabel TenorDate MaturityLabel MaturityDate Moneyness Value Ladder Ccy	Gamma	TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed	SensiCsvLoadConfig
SpotMarketData	`glob:*Spot_Market_Data.csv`				SpotMarketDataCsvLoadConfig
SurfaceMarketData	`glob:*Surface_Market_Data.csv`				SurfaceMarketDataCsvLoadConfig
CurveMarketData	`glob:*Curve_Market_Data.csv`				CurveMarketDataCsvLoadConfig
CubeMarketData	`glob:*Cube_Market_Data.csv`				CubeMarketDataCsvLoadConfig
FxRateMarketData	`glob:*FX_Rate_Market_Data.csv`				FxRateMarketDataCsvLoadConfig
TradePnLs	`glob:*TradePnLs.csv`	AsOfDate TradeId ScenarioSet CalculationId MarketDataSet RiskFactor RiskClass SensitivityName Ccy MTM PnLVector	TradePnLs	TradeKey	VaRCsvLoadConfig
SensiBaseStore	`glob:*Sensitivity Cube.csv*`				SensiSummaryCsvLoadConfig
BaseStore	`glob:*VaR-ES Cube.csv*`				VaRSummaryCsvLoadConfig
PnLBaseStore	`glob:*PLCube.csv*`				PnLSummaryCsvLoadConfig

*If no fields are present in the configuration, the datastore’s target store fields are taken into account.

Example

For the purposes of this example, we will add a column to the file DynamicTenors.csv and load the data from this column into the DynamicTenors store. In this case, our new column’s header is “TestField”, and all data in this column is “Testdata”. Here’s how the input file looks like:

TenorSet	TenorLabel	SensitivityName	NumberOfDays	TestField
DEFAULT	N/A	N/A	0.0	Testdata
DEFAULT	0.25Y	N/A	90.0	Testdata
DEFAULT	0.5Y	N/A	180.0	Testdata
DEFAULT	1Y	N/A	360.0	Testdata
DEFAULT	2Y	N/A	720.0	Testdata
DEFAULT	3Y	N/A	1080.0	Testdata
DEFAULT	5Y	N/A	1800.0	Testdata
DEFAULT	10Y	N/A	3600.0	Testdata
DEFAULT	15Y	N/A	5400.0	Testdata
DEFAULT	20Y	N/A	7200.0	Testdata
REDUCED	N/A	N/A	0.0	Testdata
REDUCED	0.5Y	N/A	180.0	Testdata
REDUCED	1Y	N/A	360.0	Testdata
REDUCED	2Y	N/A	720.0	Testdata
REDUCED	30Y	N/A	10800.0	Testdata
DECADE	10Y	N/A	3600.0	Testdata
DECADE	20Y	N/A	7200.0	Testdata

We will also attach a column calculator that multiplies the NumberOfDays by 2 into a new column, as well as a tuple publisher that will add a value into the SensitivityName field.

Step 1 - Defining customizations to datastore

Before we can load these new columns into our cube, we need to make sure that our datastore has fields that can accept them. To add new fields to an existing store, we need to create a DatastoreConfiguratorConsumer bean, which appends the required fields to the DynamicTenors store:

public static String DOUBLE_NUMBER_OF_DAYS = "DoubleNumberOfDays";
public static String TEST_FIELD = "TestData";

@Bean
public DatastoreConfiguratorConsumer addFieldToStore() {
    return configurator -> {
        configurator.appendFields(DYNAMIC_TENOR_STORE_NAME,
                List.of(
                        new CustomField(DOUBLE_NUMBER_OF_DAYS, ILiteralType.DOUBLE),
                        new CustomField(TEST_FIELD, ILiteralType.STRING)
                )
        );
    };
}

See Customizing the Datastore with the Datastore Helper for more information.

Step 2 - Implementing the column calculator and the tuple publisher

Now that we have modified our store, we need to make sure the ETL is correctly set up, so that the field will be properly populated. In most cases, the topic configuration uses the store fields as the expected file columns, which would require no further configuration. For the DynamicTenors store used in this example, file columns are defined explicitly, so any new columns are not auto-configured. The DynamicTenors store also has a previously defined column calculator creating indices for each of the tenors. To define the column calculator, we create the following bean, containing the required logic:

@Bean
public CustomFieldDescription dynamicTenorsColumnCalculator() {
    return CustomFieldDescription.of("MyCustomDynamicTenorColCalculator",
            scope -> new GenericLambdaCalculator<>(DOUBLE_NUMBER_OF_DAYS,
                    context -> {
                        Double originalValue = (Double) context.getValue(StoreFieldConstants.TENOR_NUMBER_OF_DAYS);
                        if (originalValue == null) {
                            return null;
                        }
                        return 2.0 * originalValue;
                    }
            )
    );
}

For the tuple publisher, a second bean is created, defining the publishing logic:

@Bean
public TargetDescription dynamicTenorsTarget(IDatastore datastore) {
    return TargetDescription.of("MyCustomDynamicTenors", DYNAMIC_TENOR_STORE_NAME,
            scope ->
                    new ITuplePublisher<>() {

                        @Override public void publish(IStoreMessage<?, ?> message, List<Object[]> tuples) {
                            // We use the expected index for the SensitivityName field here, while in a complete implementation the index would be an instance variable.
                            tuples.forEach(tuple -> tuple[2] = "TestValue");
                            datastore.getTransactionManager().addAll(StoreConstants.DYNAMIC_TENOR_STORE_NAME, tuples);
                        }

                        @Override
                        public Collection<String> getTargetStores() {
                            return Collections.singleton(DYNAMIC_TENOR_STORE_NAME);
                        }

                    });
}

Step 3 - Update the topic description

We now need to modify the DynamicTenors topic to include the column calculator and tuple publisher. This can either be done with properties, or in Java.

Properties

We add the following properties to our existing properties,to define:

the list of columns read from the input file,
the calculated column,
the tuple publisher.

  csv:
    topics:
      DynamicTenors:
        file-pattern: "glob:**DynamicTenors**.csv"
        parser:
          columns:
            - TenorLabel
            - NumberOfDays
            - SensitivityName
            - TenorSet
            - TestData
        channels:
            - target: MyCustomDynamicTenors
              custom-fields:
                - MyCustomDynamicTenorColCalculator
                - TenorIndices

Java

We create a new CsvTopicConfiguration Spring bean that will override the existing topic bean provided in Atoti Market Risk. We inject into the bean method:

Our previously defined target and custom field. These will be used to create the channel,
The existing tenorIndicesCalculator, defined in Atoti Market Risk. This will be used to create the channel,
The existing topic bean. This enables the default parser and file pattern to be reused.

The bean is annotated with @Order(1). This ensures it takes precedence over existing topics in Atoti Market Risk.

@Bean
@Order(1)
public CsvTopicDescription customDynamicTenorsTopic(
        TargetDescription dynamicTenorsTarget,
        CustomFieldDescription dynamicTenorsColumnCalculator,
        CustomFieldDescription tenorIndicesCalculator,
        CsvTopicDescription dynamicTenorsCsvLoadTopic) {
    ChannelDescription
            .builder(dynamicTenorsTarget)
            .customFields(Set.of(dynamicTenorsColumnCalculator, tenorIndicesCalculator))
            .build();
    return CsvTopicDescription
            .builder(StoreConstants.DYNAMIC_TENOR_STORE_NAME, dynamicTenorsCsvLoadTopic.filePattern())
            .parser(dynamicTenorsCsvLoadTopic.parser())
            .channel(dynamicTenorsChannel(StoreConstants.DYNAMIC_TENOR_STORE_NAME)).build();
}

Step 4 - Configuration classes

You can include the Beans detailed above in a single class within Atoti Market Risk.

@Configuration
public class CustomizationsConfig {

    public static String DOUBLE_NUMBER_OF_DAYS = "DoubleNumberOfDays";
    public static String TEST_FIELD = "TestData";

    @Bean
    public DatastoreConfiguratorConsumer addFieldToStore() {
        return configurator -> {
            configurator.appendFields(DYNAMIC_TENOR_STORE_NAME,
                    List.of(
                            new CustomField(DOUBLE_NUMBER_OF_DAYS, ILiteralType.DOUBLE),
                            new CustomField(TEST_FIELD, ILiteralType.STRING)
                    )
            );
        };
    }

    @Bean
    public CustomFieldDescription dynamicTenorsColumnCalculator() {
        return CustomFieldDescription.of("MyCustomDynamicTenorColCalculator",
                scope -> new GenericLambdaCalculator<>(DOUBLE_NUMBER_OF_DAYS,
                        context -> {
                            Double originalValue = (Double) context.getValue(StoreFieldConstants.TENOR_NUMBER_OF_DAYS);
                            if (originalValue == null) {
                                return null;
                            }
                            return 2.0 * originalValue;
                        }
                )
        );
    }

    @Bean
    public TargetDescription dynamicTenorsTarget(IDatastore datastore) {
        return TargetDescription.of("MyCustomDynamicTenors", DYNAMIC_TENOR_STORE_NAME,
                scope ->
                        new ITuplePublisher<>() {

                            @Override public void publish(IStoreMessage<?, ?> message, List<Object[]> tuples) {
                                // We use the expected index for the SensitivityName field here, while in a complete implementation the index would be an instance variable.
                                tuples.forEach(tuple -> tuple[2] = "TestValue");
                                datastore.getTransactionManager().addAll(StoreConstants.DYNAMIC_TENOR_STORE_NAME, tuples);
                            }

                            @Override
                            public Collection<String> getTargetStores() {
                                return Collections.singleton(DYNAMIC_TENOR_STORE_NAME);
                            }

                        });
    }

    // This bean is not required if you are using properties to define the topic.
    @Bean
    public CsvTopicDescription customDynamicTenorsTopic(
            TargetDescription dynamicTenorsTarget,
            CustomFieldDescription dynamicTenorsColumnCalculator,
            CustomFieldDescription tenorIndicesCalculator,
            CsvTopicDescription dynamicTenorsCsvLoadTopic) {
        ChannelDescription
                .builder(dynamicTenorsTarget)
                .customFields(Set.of(dynamicTenorsColumnCalculator, tenorIndicesCalculator))
                .build();
        return CsvTopicDescription
                .builder(StoreConstants.DYNAMIC_TENOR_STORE_NAME, dynamicTenorsCsvLoadTopic.filePattern())
                .parser(dynamicTenorsCsvLoadTopic.parser())
                .channel(dynamicTenorsChannel(StoreConstants.DYNAMIC_TENOR_STORE_NAME)).build();
    }
}

Then, include this configuration class in the @Import annotation of the MarketRiskConfig class:

@Import(value = {
        ...
		CustomizationsConfig.class

})
public class MarketRiskConfig {
    ...
}

Importing this customization class and modifying the input file will result in the following data being loaded into the store:

TenorSet	TenorLabel	SensitivityName	NumberOfDays	TenorIndices	DoubleNumberOfDays	TestData
DEFAULT	N/A	TestValue	0.0	0	0.0	Testdata
DEFAULT	0.25Y	TestValue	90.0	1	180.0	Testdata
DEFAULT	0.5Y	TestValue	180.0	2	360.0	Testdata
DEFAULT	1Y	TestValue	360.0	3	720.0	Testdata
DEFAULT	2Y	TestValue	720.0	4	1440.0	Testdata
DEFAULT	3Y	TestValue	1080.0	5	2160.0	Testdata
DEFAULT	5Y	TestValue	1800.0	6	3600.0	Testdata
DEFAULT	10Y	TestValue	3600.0	7	7200.0	Testdata
DEFAULT	15Y	TestValue	5400.0	8	10800.0	Testdata
DEFAULT	20Y	TestValue	7200.0	9	14400.0	Testdata
REDUCED	N/A	TestValue	0.0	0	0.0	Testdata
REDUCED	0.5Y	TestValue	180.0	1	360.0	Testdata
REDUCED	1Y	TestValue	360.0	2	720.0	Testdata
REDUCED	2Y	TestValue	720.0	3	1440.0	Testdata
REDUCED	30Y	TestValue	10800.0	4	21600.0	Testdata
DECADE	10Y	TestValue	3600.0	0	7200.0	Testdata
DECADE	20Y	TestValue	7200.0	1	14400.0	Testdata

The configuration for this behavior is:

new ATableFormatTuplePublisher<>(datastore, Map.of(INSTRUMENT_ID, RISK_FACTOR)) {
  @Override
  public void publish(IStoreMessage<? extends I, ?> message, List<Object[]> tuples) {
      Queue<Object[]> storeTuples = new ConcurrentLinkedQueue<>();
      tuples.forEach(
              tuple -> {
                  if (checkNull(tmmProps, message, tuple, TENOR_LABELS, TENOR_DATES, MATURITY_LABELS, MATURITY_DATES, MONEYNESS_LABELS,
                          NOMINAL)) {
                      Queue<Object[]> extractedScalarTuples = new ConcurrentLinkedQueue<>();
                      devectorizer.extractScalarTuples(message, tuple, extractedScalarTuples);
                      extractedScalarTuples.forEach(
                              scalarTuple -> storeTuples.add(createTupleForTable(message, SPOT_MARKET_DATA_STORE, scalarTuple))
                      );
                  }
              }
      );
      datastore.getTransactionManager().addAll(SPOT_MARKET_DATA_STORE, storeTuples);
  }

  @Override
  public Collection<String> getTargetStores() {
      return Set.of(SPOT_MARKET_DATA_STORE);
  }
}

Within the createTupleForTable method, the publisher iterates over the table fields, attempting to retrieve a value in the input tuple by the mapped field (or directly if no mapping is available). If no value is available in the input tuple, or no matching field is found, the value will be empty. If a tuple field’s value isn’t requested by the publisher, it is ignored.

​Source Topic Descriptions

​Example

​Step 1 - Defining customizations to datastore

​Step 2 - Implementing the column calculator and the tuple publisher

​Step 3 - Update the topic description

​Properties

​Java

​Step 4 - Configuration classes

​Suggested Further Reading