Configuring sources using Spring Beans and properties

This page describes the source topic descriptions defined in the application, and provides an example of adding customization.

Source Topic Descriptions

The Data Load Controller is used to define the source topics. The following descriptions are defined:

Name File pattern Columns (if explicitly defined)* Target description name Custom fields Configuration class
Countries glob:**Countries*.csv CommonStoreCsvLoadConfig
CounterpartyParentChild glob:**CounterpartyParentChild*.csv CommonStoreCsvLoadConfig
Counterparties glob:**Counterparties*.csv CommonStoreCsvLoadConfig
TradeAttributes glob:**TradeAttributes*.csv CommonStoreCsvLoadConfig
RiskFactorsCatalogue glob:**RiskFactorsCatalog*.csv CommonStoreCsvLoadConfig
Scenarios glob:**Scenarios*.csv CommonStoreCsvLoadConfig
LegalEntityParentChild glob:**LegalEntityParentChild*.csv CommonStoreCsvLoadConfig
BookParentChild glob:**BookParentChild*.csv CommonStoreCsvLoadConfig
MarketShifts glob:**MarketShifts*.csv CommonStoreCsvLoadConfig
PnL glob:**PLPC*.csv AsOfDate
TradeId
Daily
Monthly
Yearly
Lifetime
Type
PLDriver
IsFullReval
Ccy
MarketDataSet
Bucket
PnL TradeKey PnLCsvLoadConfig
DynamicTenors glob:**DynamicTenors*.csv TenorLabel
NumberOfDays
SensitivityName
TenorSet
DynamicTenors TenorIndices SensiCsvLoadConfig
DynamicMaturities glob:**DynamicMaturities*.csv MaturityLabel
NumberOfDays
SensitivityName
MaturitySet
DynamicMaturities MaturityIndices SensiCsvLoadConfig
DynamicMoneyness glob:**DynamicMoneyness*.csv MoneynessLabel
Shift
SensitivityName
MoneynessSet
DynamicMoneyness MoneynessIndices SensiCsvLoadConfig
CorrelationMarketData glob:**Correlation_Market_Data*.csv SensiCsvLoadConfig
DividendMarketData glob:**Dividends*.csv SensiCsvLoadConfig
SplitRatioMarketData glob:**SplitRatio*.csv SensiCsvLoadConfig
SensiLadders glob:**LadderDefinition*.csv SensiCsvLoadConfig
Delta glob:**DeltaSensitivities*.csv AsOfDate
TradeId
SensitivityName
RiskClass
MarketDataSet
RiskFactorId
TenorLabel
TenorDate
MaturityLabel
MaturityDate
Moneyness
Value
Ladder
Ccy
Delta TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed SensiCsvLoadConfig
Correlation glob:**CorrelationSensitivities*.csv AsOfDate
TradeId
SensitivityName
RiskClass
MarketDataSet
RiskFactorId
RiskFactorId2
TenorLabel
TenorDate
MaturityLabel
MaturityDate
Moneyness
Value
Ladder
Ccy
Correlation TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed SensiCsvLoadConfig
CrossGamma glob:**CrossGammaSensitivities*.csv AsOfDate
TradeId
SensitivityName
RiskClass
MarketDataSet
RiskFactor
RiskFactor2
TenorLabel
TenorDate
MaturityLabel
MaturityDate
Moneyness
Value
Ladder
Ccy
CrossGamma TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed SensiCsvLoadConfig
Volga glob:**VolgaSensitivities*.csv AsOfDate
TradeId
SensitivityName
RiskClass
MarketDataSet
RiskFactorId
TenorLabel
TenorDate
MaturityLabel
MaturityDate
Moneyness
Value
Ladder
Ccy
Volga TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed SensiCsvLoadConfig
Vanna glob:**VannaSensitivities*.csv AsOfDate
TradeId
SensitivityName
RiskClass
MarketDataSet
RiskFactorId
RiskFactorId2
TenorLabel
TenorDate
MaturityLabel
MaturityDate
Moneyness
Value
Ladder
Ccy
Vanna TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed SensiCsvLoadConfig
Vega glob:**VegaSensitivities*.csv AsOfDate
TradeId
SensitivityName
RiskClass
MarketDataSet
RiskFactorId
TenorLabel
TenorDate
MaturityLabel
MaturityDate
Moneyness
Value
Ladder
Ccy
Vega TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed SensiCsvLoadConfig
Theta glob:**ThetaSensitivities*.csv AsOfDate
TradeId
SensitivityName
RiskClass
MarketDataSet
RiskFactorId
TenorLabel
TenorDate
MaturityLabel
MaturityDate
Moneyness
Value
Ladder
Ccy
Theta TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed SensiCsvLoadConfig
Gamma regex:(?i).*Gamma(?<!CrossGamma)Sensitivities.*\.csv.* AsOfDate
TradeId
SensitivityName
RiskClass
MarketDataSet
RiskFactorId
TenorLabel
TenorDate
MaturityLabel
MaturityDate
Moneyness
Value
Ladder
Ccy
Gamma TradeKey + 6 anonymous custom field descriptions to handle vectorized inputs if needed SensiCsvLoadConfig
SpotMarketData glob:**Spot_Market_Data*.csv SpotMarketDataCsvLoadConfig
SurfaceMarketData glob:**Surface_Market_Data*.csv SurfaceMarketDataCsvLoadConfig
CurveMarketData glob:**Curve_Market_Data*.csv CurveMarketDataCsvLoadConfig
CubeMarketData glob:**Cube_Market_Data*.csv CubeMarketDataCsvLoadConfig
FxRateMarketData glob:**FX_Rate_Market_Data*.csv FxRateMarketDataCsvLoadConfig
TradePnLs glob:**TradePnLs*.csv AsOfDate
TradeId
ScenarioSet
CalculationId
MarketDataSet
RiskFactor
RiskClass
SensitivityName
Ccy
MTM
PnLVector
TradePnLs TradeKey VaRCsvLoadConfig
SensiBaseStore glob:**Sensitivity Cube*.csv* SensiSummaryCsvLoadConfig
BaseStore glob:**VaR-ES Cube*.csv* VaRSummaryCsvLoadConfig
PnLBaseStore glob:**PLCube*.csv* PnLSummaryCsvLoadConfig

*If no fields are present in the configuration, the datastore’s target store fields are taken into account.

Example

For the purposes of this example, we will add a column to the file DynamicTenors.csv and load the data from this column into the DynamicTenors store. In this case, our new column’s header is “TestField”, and all data in this column is “Testdata”. Here’s how the input file looks like:

TenorSet TenorLabel SensitivityName NumberOfDays TestField
DEFAULT N/A N/A 0.0 Testdata
DEFAULT 0.25Y N/A 90.0 Testdata
DEFAULT 0.5Y N/A 180.0 Testdata
DEFAULT 1Y N/A 360.0 Testdata
DEFAULT 2Y N/A 720.0 Testdata
DEFAULT 3Y N/A 1080.0 Testdata
DEFAULT 5Y N/A 1800.0 Testdata
DEFAULT 10Y N/A 3600.0 Testdata
DEFAULT 15Y N/A 5400.0 Testdata
DEFAULT 20Y N/A 7200.0 Testdata
REDUCED N/A N/A 0.0 Testdata
REDUCED 0.5Y N/A 180.0 Testdata
REDUCED 1Y N/A 360.0 Testdata
REDUCED 2Y N/A 720.0 Testdata
REDUCED 30Y N/A 10800.0 Testdata
DECADE 10Y N/A 3600.0 Testdata
DECADE 20Y N/A 7200.0 Testdata

We will also attach a column calculator that multiplies the NumberOfDays by 2 into a new column, as well as a tuple publisher that will add a value into the SensitivityName field.

Step 1 - Defining customizations to datastore

Before we can load these new columns into our cube, we need to make sure that our datastore has fields that can accept them.

To add new fields to an existing store, we need to create a DatastoreConfiguratorConsumer bean, which appends the required fields to the DynamicTenors store:

public static String DOUBLE_NUMBER_OF_DAYS = "DoubleNumberOfDays";
public static String TEST_FIELD = "TestData";

@Bean
public DatastoreConfiguratorConsumer addFieldToStore() {
    return configurator -> {
        configurator.appendFields(DYNAMIC_TENOR_STORE_NAME,
                List.of(
                        new CustomField(DOUBLE_NUMBER_OF_DAYS, ILiteralType.DOUBLE),
                        new CustomField(TEST_FIELD, ILiteralType.STRING)
                )
        );
    };
}

See Customizing the Datastore with the Datastore Helper for more information.

Step 2 - Implementing the column calculator and the tuple publisher

Now that we have modified our store, we need to make sure the ETL is correctly set up, so that the field will be properly populated.

In most cases, the topic configuration uses the store fields as the expected file columns, which would require no further configuration.

For the DynamicTenors store used in this example, file columns are defined explicitly, so any new columns are not auto-configured. The DynamicTenors store also has a previously defined column calculator creating indices for each of the tenors.

To define the column calculator, we create the following bean, containing the required logic:

@Bean
public CustomFieldDescription dynamicTenorsColumnCalculator() {
    return CustomFieldDescription.of("MyCustomDynamicTenorColCalculator",
            scope -> new LambdaCalculator(DOUBLE_NUMBER_OF_DAYS,
                    context -> {
                        Double originalValue = (Double) context.getValue(StoreFieldConstants.TENOR_NUMBER_OF_DAYS);
                        if (originalValue == null) {
                            return null;
                        }
                        return 2.0 * originalValue;
                    }
            )
    );
}

For the tuple publisher, a second bean is created, defining the publishing logic:

@Bean
public TargetDescription dynamicTenorsTarget(IDatastore datastore) {
    return TargetDescription.of("MyCustomDynamicTenors", DYNAMIC_TENOR_STORE_NAME,
            scope ->
                    new ITuplePublisher<>() {

                        @Override public void publish(IStoreMessage<?, ?> message, List<Object[]> tuples) {
                            // We use the expected index for the SensitivityName field here, while in a complete implementation the index would be an instance variable.
                            tuples.forEach(tuple -> tuple[2] = "TestValue");
                            datastore.getTransactionManager().addAll(StoreConstants.DYNAMIC_TENOR_STORE_NAME, tuples);
                        }

                        @Override
                        public Collection<String> getTargetStores() {
                            return Collections.singleton(DYNAMIC_TENOR_STORE_NAME);
                        }

                    });
}

Step 3 - Data Load Controller Properties

We add the following properties to our existing properties,to define:

  • the list of columns read from the input file,
  • the calculated column,
  • the tuple publisher.
  csv:
    topics:
      DynamicTenors:
        file-pattern: "glob:**DynamicTenors**.csv"
        parser:
          columns:
            - TenorLabel
            - NumberOfDays
            - SensitivityName
            - TenorSet
            - TestData
        channels:
            - target: MyCustomDynamicTenors
              custom-fields:
                - MyCustomDynamicTenorColCalculator
                - TenorIndices

Step 4 - Configuration classes

You can include the Beans detailed above in a single class within Atoti Market Risk.

@Configuration
public class CustomizationsConfig {

    public static String DOUBLE_NUMBER_OF_DAYS = "DoubleNumberOfDays";
    public static String TEST_FIELD = "TestData";

    @Bean
    public DatastoreConfiguratorConsumer addFieldToStore() {
        return configurator -> {
            configurator.appendFields(DYNAMIC_TENOR_STORE_NAME,
                    List.of(
                            new CustomField(DOUBLE_NUMBER_OF_DAYS, ILiteralType.DOUBLE),
                            new CustomField(TEST_FIELD, ILiteralType.STRING)
                    )
            );
        };
    }

    @Bean
    public CustomFieldDescription dynamicTenorsColumnCalculator() {
        return CustomFieldDescription.of("MyCustomDynamicTenorColCalculator",
                scope -> new LambdaCalculator(DOUBLE_NUMBER_OF_DAYS,
                        context -> {
                            Double originalValue = (Double) context.getValue(StoreFieldConstants.TENOR_NUMBER_OF_DAYS);
                            if (originalValue == null) {
                                return null;
                            }
                            return 2.0 * originalValue;
                        }
                )
        );
    }

    @Bean
    public TargetDescription dynamicTenorsTarget(IDatastore datastore) {
        return TargetDescription.of("MyCustomDynamicTenors", DYNAMIC_TENOR_STORE_NAME,
                scope ->
                        new ITuplePublisher<>() {

                            @Override public void publish(IStoreMessage<?, ?> message, List<Object[]> tuples) {
                                // We use the expected index for the SensitivityName field here, while in a complete implementation the index would be an instance variable.
                                tuples.forEach(tuple -> tuple[2] = "TestValue");
                                datastore.getTransactionManager().addAll(StoreConstants.DYNAMIC_TENOR_STORE_NAME, tuples);
                            }

                            @Override
                            public Collection<String> getTargetStores() {
                                return Collections.singleton(DYNAMIC_TENOR_STORE_NAME);
                            }

                        });
    }

}

Then, include this configuration class in the @Import annotation of the MarketRiskConfig class:

@Import(value = {
        ...
		CustomizationsConfig.class

})
public class MarketRiskConfig {
    ...
}

Importing this customization class and modifying the input file will result in the following data being loaded into the store:

TenorSet TenorLabel SensitivityName NumberOfDays TenorIndices DoubleNumberOfDays TestData
DEFAULT N/A TestValue 0.0 0 0.0 Testdata
DEFAULT 0.25Y TestValue 90.0 1 180.0 Testdata
DEFAULT 0.5Y TestValue 180.0 2 360.0 Testdata
DEFAULT 1Y TestValue 360.0 3 720.0 Testdata
DEFAULT 2Y TestValue 720.0 4 1440.0 Testdata
DEFAULT 3Y TestValue 1080.0 5 2160.0 Testdata
DEFAULT 5Y TestValue 1800.0 6 3600.0 Testdata
DEFAULT 10Y TestValue 3600.0 7 7200.0 Testdata
DEFAULT 15Y TestValue 5400.0 8 10800.0 Testdata
DEFAULT 20Y TestValue 7200.0 9 14400.0 Testdata
REDUCED N/A TestValue 0.0 0 0.0 Testdata
REDUCED 0.5Y TestValue 180.0 1 360.0 Testdata
REDUCED 1Y TestValue 360.0 2 720.0 Testdata
REDUCED 2Y TestValue 720.0 3 1440.0 Testdata
REDUCED 30Y TestValue 10800.0 4 21600.0 Testdata
DECADE 10Y TestValue 3600.0 0 7200.0 Testdata
DECADE 20Y TestValue 7200.0 1 14400.0 Testdata

The ATableFormatTuplePublisher

For use cases where the destination table doesn’t match the input file format, we provide the ATableFormatTuplePublisher.

Any extension of this publisher can be constructed with the file tuple containing a subset of the table fields, or with an explicit mapping between table fields and file columns.

Example

For the backwards-compatible Market Data API sources, the stores should be loaded from the deprecated file format. For SpotMarketData, when loading from the MarketData.csv file, the following applies:

File Field Table Field Behavior
AsOfDate AsOfDate Loaded as-is
MarketDataSet MarketDataSet Loaded as-is
RiskFactorId InstrumentId Mapped by the publisher
Quote Quote Loaded as-is
TenorLabels - Must be empty
MaturityLabels - Must be empty
MoneynessLabels - Must be empty
TenorDates - Must be empty
MaturityDates - Must be empty
Nominal - Must be empty

The configuration for this behavior is:

new ATableFormatTuplePublisher<>(datastore, Map.of(INSTRUMENT_ID, RISK_FACTOR)) {
  @Override
  public void publish(IStoreMessage<? extends I, ?> message, List<Object[]> tuples) {
      Queue<Object[]> storeTuples = new ConcurrentLinkedQueue<>();
      tuples.forEach(
              tuple -> {
                  if (checkNull(tmmProps, message, tuple, TENOR_LABELS, TENOR_DATES, MATURITY_LABELS, MATURITY_DATES, MONEYNESS_LABELS,
                          NOMINAL)) {
                      Queue<Object[]> extractedScalarTuples = new ConcurrentLinkedQueue<>();
                      devectorizer.extractScalarTuples(message, tuple, extractedScalarTuples);
                      extractedScalarTuples.forEach(
                              scalarTuple -> storeTuples.add(createTupleForTable(message, SPOT_MARKET_DATA_STORE, scalarTuple))
                      );
                  }
              }
      );
      datastore.getTransactionManager().addAll(SPOT_MARKET_DATA_STORE, storeTuples);
  }
  
  @Override
  public Collection<String> getTargetStores() {
      return Set.of(SPOT_MARKET_DATA_STORE);
  }
}

Within the createTupleForTable method, the publisher iterates over the table fields, attempting to retrieve a value in the input tuple by the mapped field (or directly if no mapping is available).

If no value is available in the input tuple, or no matching field is found, the value will be empty. If a tuple field’s value isn’t requested by the publisher, it is ignored.

Suggested Further Reading