Configuring sources using Spring Beans
This page describes the extension points introduced in MR 3.1.0 that allow you to customize the CSV sources through Spring Beans.
It also provides an example of adding customization using the new extension points.
Concept
A source topic is configured through two parameter objects.
Parameter object | Fields | Details |
---|---|---|
CsvTopicColumns |
String topic List<String> columns Map<String, List<IColumnCalculator<ILineReader>>> calculators |
For a given topic, this object contains the list of columns expected in the file, and a list of column calculators for each store targeted by the source. |
CsvTopicPublisher<I> |
String topic ITuplePublisher<IFileInfo<I>> publisher |
For a given topic, this object contains the publisher that will be used by the source. |
Customization Beans
are defined as generator Function
objects that accept the map of previously defined parameter objects and generate a list of new parameter objects.
These function Beans
are then used in the creation of provider objects that respect the order in which the Beans
are defined.
The provider objects apply the Functions
on a copy of the internal Map
of previously defined parameters, to prevent direct operations on the Map
.
The internal source configuration uses the same Bean
mechanism, and we advise using the @Order
annotation for custom Beans
, with values higher than 10.
The Provider
objects that the generator Beans
are wired into are also exposed as Beans
.
While the default behavior should suffice for most customization requirements, you can replace a particular Provider
object by creating a qualified bean of the same type and marking it as @Primary
.
Configuration Bean types
The following Bean types are available:
Bean type | Details |
---|---|
Function<Map<String, CsvTopicColumns>, List<CsvTopicColumns>> |
A Function accepting the previously defined CsvTopicColumns object per topic, and generating a List of new CsvTopicColumns objects. |
Function<Map<String, CsvTopicPublisher<I>>, List<CsvTopicPublisher<I>>> |
A Function accepting the previously defined CsvTopicPublisher object per topic, and generating a List of new CsvTopicPublisher objects. |
CsvColumnsProvider |
A provider of CsvTopicColumns objects, retrievable per topic or as a collection. |
CsvPublisherProvider<I> |
A provider of CsvTopicPublisher objects, retrievable per topic. |
Available Qualifiers
The customization Beans
and the Provider
objects they are used in, are annotated with a @Qualifier
, explicitly defining the source to which the customization applies.
All @Qualifier
constants are defined in the SpringConstants
class.
Constant | Qualifier | Type | Source |
---|---|---|---|
SP_QUALIFIER__SENSI_TOPIC_COLUMNS | “sensi-topic-columns” | Columns generator Function |
Sensitivity |
SP_QUALIFIER__SENSI_SUMMARY_TOPIC_COLUMNS | “sensi-summary-topic-columns” | Columns generator Function |
Sensitivity Summary |
SP_QUALIFIER__VAR_TOPIC_COLUMNS | “var-topic-columns” | Columns generator Function |
VaR |
SP_QUALIFIER__VAR_SUMMARY_TOPIC_COLUMNS | “var-summary-topic-columns” | Columns generator Function |
VaR Summary |
SP_QUALIFIER__PNL_TOPIC_COLUMNS | “pnl-topic-columns” | Columns generator Function |
PnL |
SP_QUALIFIER__PNL_SUMMARY_TOPIC_COLUMNS | “pnl-summary-topic-columns” | Columns generator Function |
PnL Summary |
SP_QUALIFIER__ADJUSTMENTS_TOPIC_COLUMNS | “adjustments-topic-columns” | Columns generator Function |
Adjustments |
SP_QUALIFIER__COMMON_TOPIC_COLUMNS | “common-topic-columns” | Columns generator Function |
Common |
SP_QUALIFIER__SENSI_TOPIC_PUBLISHERS | “sensi-topic-publishers” | Publisher generator Function |
Sensitivity |
SP_QUALIFIER__SENSI_SUMMARY_TOPIC_PUBLISHERS | “sensi-summary-topic-publishers” | Publisher generator Function |
Sensitivity Summary |
SP_QUALIFIER__VAR_TOPIC_PUBLISHERS | “var-topic-publishers” | Publisher generator Function |
VaR |
SP_QUALIFIER__VAR_SUMMARY_TOPIC_PUBLISHERS | “var-summary-topic-publishers” | Publisher generator Function |
VaR Summary |
SP_QUALIFIER__PNL_TOPIC_PUBLISHERS | “pnl-topic-publishers” | Publisher generator Function |
PnL |
SP_QUALIFIER__PNL_SUMMARY_TOPIC_PUBLISHERS | “pnl-summary-topic-publishers” | Publisher generator Function |
PnL Summary |
SP_QUALIFIER__ADJUSTMENTS_TOPIC_PUBLISHERS | “adjustments-topic-publishers” | Publisher generator Function |
Adjustments |
SP_QUALIFIER__COMMON_TOPIC_PUBLISHERS | “common-topic-publishers” | Publisher generator Function |
Common |
SP_QUALIFIER__SENSI_TOPIC_COLUMNS_PROVIDER | “sensi-topic-columns-provider” | Columns Provider |
Sensitivity |
SP_QUALIFIER__SENSI_SUMMARY_TOPIC_COLUMNS_PROVIDER | “sensi-summary-topic-columns-provider” | Columns Provider |
Sensitivity Summary |
SP_QUALIFIER__VAR_TOPIC_COLUMNS_PROVIDER | “var-topic-columns-provider” | Columns Provider |
VaR |
SP_QUALIFIER__VAR_SUMMARY_TOPIC_COLUMNS_PROVIDER | “var-summary-topic-columns-provider” | Columns Provider |
VaR Summary |
SP_QUALIFIER__PNL_TOPIC_COLUMNS_PROVIDER | “pnl-topic-columns-provider” | Columns Provider |
PnL |
SP_QUALIFIER__PNL_SUMMARY_TOPIC_COLUMNS_PROVIDER | “pnl-summary-topic-columns-provider” | Columns Provider |
PnL Summary |
SP_QUALIFIER__ADJUSTMENTS_TOPIC_COLUMNS_PROVIDER | “adjustments-topic-columns-provider” | Columns Provider |
Adjustments |
SP_QUALIFIER__COMMON_TOPIC_COLUMNS_PROVIDER | “common-topic-columns-provider” | Columns Provider |
Common |
SP_QUALIFIER__SENSI_TOPIC_PUBLISHERS_PROVIDER | “sensi-topic-publishers-provider” | Publisher Provider |
Sensitivity |
SP_QUALIFIER__SENSI_SUMMARY_TOPIC_PUBLISHERS_PROVIDER | “sensi-summary-topic-publishers-provider” | Publisher Provider |
Sensitivity Summary |
SP_QUALIFIER__VAR_TOPIC_PUBLISHERS_PROVIDER | “var-topic-publishers-provider” | Publisher Provider |
VaR |
SP_QUALIFIER__VAR_SUMMARY_TOPIC_PUBLISHERS_PROVIDER | “var-summary-topic-publishers-provider” | Publisher Provider |
VaR Summary |
SP_QUALIFIER__PNL_TOPIC_PUBLISHERS_PROVIDER | “pnl-topic-publishers-provider” | Publisher Provider |
PnL |
SP_QUALIFIER__PNL_SUMMARY_TOPIC_PUBLISHERS_PROVIDER | “pnl-summary-topic-publishers-provider” | Publisher Provider |
PnL Summary |
SP_QUALIFIER__ADJUSTMENTS_TOPIC_PUBLISHERS_PROVIDER | “adjustments-topic-publishers-provider” | Publisher Provider |
Adjustments |
SP_QUALIFIER__COMMON_TOPIC_PUBLISHERS_PROVIDER | “common-topic-publishers-provider” | Publisher Provider |
Common |
Example
For the purposes of this example, we will add a column to the file
StaticTenors.csv
and load the data from this column into the
Tenors
store. In this case, our new column’s header is ‘TestField’,
and all data in this column is ‘Testdata’. Below are the intended contents of the input file.
TenorLabels | NumberOfDays | SensitivityName | TestField |
---|---|---|---|
N/A | 0 | Testdata | |
0.25Y | 90 | Testdata | |
0.5Y | 180 | Testdata | |
1Y | 360 | Testdata | |
2Y | 720 | Testdata | |
3Y | 1080 | Testdata | |
4Y | 1440 | Testdata | |
5Y | 1800 | Testdata | |
10Y | 3600 | Testdata | |
15Y | 5400 | Testdata | |
20Y | 7200 | Testdata | |
30Y | 10080 | Testdata |
We will also attach a column calculator that multiplies the ‘NumberOfDays’ by 2 into a new column, as well as a tuple publisher which will add a value into the ‘SensitivityName’ field.
Step 1 - Define customizations to datastore
Before we can load these new columns into our cube, we need to make sure that our datastore has fields that can accept them.
To add new fields to an existing store, we will need to create a bean with the signature
@Qualifier(SP_QUALIFIER__CUSTOMISATIONS)
Consumer<IDatastoreConfigurator> addFieldToStore()`
which appends the required fields to the Tenors
store:
@Bean
@Qualifier(SP_QUALIFIER__CUSTOMISATIONS)
public Consumer<IDatastoreConfigurator> addFieldToStore() {
return configurator -> {
configurator.appendFields(SensiDatastoreDescriptionConfig.SCHEMA, StoreNames.TENOR_STORE_NAME,
List.of(
new CustomField(DOUBLE_NUMBER_OF_DAYS, ILiteralType.DOUBLE),
new CustomField(TEST_FIELD, ILiteralType.STRING)
)
);
};
}
See Customizing the Datastore with the Datastore Helper for more information.
Step 2 - Configuring the source
Now that we have modified our store, we need to make sure the ETL is correctly set up so that the field will be properly populated.
In some cases, the topic configuration will use the store fields as the expected file columns, which would require no further configuration.
For the Tenors
store used in this example, file columns are defined explicitly, and thus any new columns are not autoconfigured.
The Tenors store also has a previously defined column calculator creating indices for each of the tenors.
To add the new columns and the column calculator, we will create a bean with the signature
@Qualifier(SP_QUALIFIER__SENSI_TOPIC_COLUMNS)
@Order(10)
public Function<Map<String, CsvTopicColumns>, List<CsvTopicColumns>> addFieldAndCalculator()
containing the required logic:
@Bean
@Qualifier(SP_QUALIFIER__SENSI_TOPIC_COLUMNS)
@Order(10)
public Function<Map<String, CsvTopicColumns>, List<CsvTopicColumns>> addFieldAndCalculator() {
return previousConfig -> {
// We retrieve the previous columns of the topic for the Tenors store.
CsvTopicColumns prevColumns = previousConfig.get(StoreNames.TENOR_STORE_NAME);
List<String> newColumns = new ArrayList<>(prevColumns.getColumns());
newColumns.add(TEST_FIELD);
// We retrieve the previous calculators defined for the store.
// While in this instance the topic name and the store are the same, calculators can be defined for multiple stores within a topic.
Map<String, List<IColumnCalculator<ILineReader>>> topicCalculators = new HashMap<>(prevColumns.getCalculators());
List<IColumnCalculator<ILineReader>> storeCalculators = new ArrayList<>(topicCalculators.get(StoreNames.TENOR_STORE_NAME));
storeCalculators.add(new IColumnCalculator<>() {
@Override public String getColumnName() {
return DOUBLE_NUMBER_OF_DAYS;
}
@Override public Object compute(IColumnCalculationContext<ILineReader> context) {
return ((Double) context.getValue(StoreFieldNames.TENOR_NUMBER_OF_DAYS)) * 2.0;
}
});
topicCalculators.put(StoreNames.TENOR_STORE_NAME, storeCalculators);
// We create a new CsvTopicColumns object with the new list of columns and the previously defined column calculators.
return List.of(new CsvTopicColumns(
StoreNames.TENOR_STORE_NAME,
newColumns,
topicCalculators
));
};
}
For the tuple publisher, a second bean is created with the signature
@Qualifier(SP_QUALIFIER__SENSI_TOPIC_PUBLISHERS)
@Order(10)
public Function<Map<String, CsvTopicPublisher<Path>>, List<CsvTopicPublisher<Path>>> addPublisher()
defining the publishing logic:
@Bean
@Qualifier(SP_QUALIFIER__SENSI_TOPIC_PUBLISHERS)
@Order(10)
public Function<Map<String, CsvTopicPublisher<Path>>, List<CsvTopicPublisher<Path>>> addPublisher(
@Autowired IDatastore datastore
) {
return previousConfig -> List.of(
new CsvTopicPublisher<>(
StoreNames.TENOR_STORE_NAME,
new ITuplePublisher<IFileInfo<Path>>() {
@Override
public void publish(IStoreMessage<? extends IFileInfo<Path>, ?> message, List<Object[]> tuples) {
// We use the expected index for the SensitivityName field here, while in a complete implementation the index would be an instance variable.
tuples.forEach(tuple -> tuple[1] = "TestValue");
datastore.getTransactionManager().addAll(StoreNames.TENOR_STORE_NAME, tuples);
}
@Override
public Collection<String> getTargetStores() {
return Collections.singleton(StoreNames.TENOR_STORE_NAME);
}
}
)
);
}
Step 3 - Configuration classes
The Beans detailed above can be included in a single class within the MR application.
To prevent the customizations being applied to the scalar sensitivities application mode, annotate the class with @Conditional(ConfigurationVectorised.class)
@Configuration
@Conditional(ConfigurationVectorised.class)
public class CustomizationsConfig {
private static final String TEST_FIELD = "TestData";
private static final String DOUBLE_NUMBER_OF_DAYS = "DoubleNumberOfDays";
@Bean
@Qualifier(SP_QUALIFIER__CUSTOMISATIONS)
public Consumer<IDatastoreConfigurator> addFieldToStore() {
return configurator -> {
configurator.appendFields(SensiDatastoreDescriptionConfig.SCHEMA, StoreNames.TENOR_STORE_NAME,
List.of(
new CustomField(DOUBLE_NUMBER_OF_DAYS, ILiteralType.DOUBLE),
new CustomField(TEST_FIELD, ILiteralType.STRING)
)
);
};
}
@Bean
@Qualifier(SP_QUALIFIER__SENSI_TOPIC_COLUMNS)
@Order(10)
public Function<Map<String, CsvTopicColumns>, List<CsvTopicColumns>> addFieldAndCalculator() {
return previousConfig -> {
// We retrieve the previous columns of the topic for the Tenors store.
CsvTopicColumns prevColumns = previousConfig.get(StoreNames.TENOR_STORE_NAME);
List<String> newColumns = new ArrayList<>(prevColumns.getColumns());
newColumns.add(TEST_FIELD);
// We retrieve the previous calculators defined for the store. The Tenors store has a column calculator setting indices for each tenor.
// While in this instance the topic name and the store are the same, calculators can be defined for multiple stores within a topic.
Map<String, List<IColumnCalculator<ILineReader>>> topicCalculators = new HashMap<>(prevColumns.getCalculators());
List<IColumnCalculator<ILineReader>> storeCalculators = new ArrayList<>(topicCalculators.get(StoreNames.TENOR_STORE_NAME));
storeCalculators.add(new IColumnCalculator<>() {
@Override public String getColumnName() {
return DOUBLE_NUMBER_OF_DAYS;
}
@Override public Object compute(IColumnCalculationContext<ILineReader> context) {
return ((Double) context.getValue(StoreFieldNames.TENOR_NUMBER_OF_DAYS)) * 2.0;
}
});
topicCalculators.put(StoreNames.TENOR_STORE_NAME, storeCalculators);
// We create a new CsvTopicColumns object with the new list of columns and the previously defined column calculators.
return List.of(new CsvTopicColumns(
StoreNames.TENOR_STORE_NAME,
newColumns,
topicCalculators
));
};
}
@Bean
@Qualifier(SP_QUALIFIER__SENSI_TOPIC_PUBLISHERS)
@Order(10)
public Function<Map<String, CsvTopicPublisher<Path>>, List<CsvTopicPublisher<Path>>> addPublisher(
@Autowired IDatastore datastore
) {
return previousConfig -> List.of(
new CsvTopicPublisher<>(
StoreNames.TENOR_STORE_NAME,
new ITuplePublisher<IFileInfo<Path>>() {
@Override
public void publish(IStoreMessage<? extends IFileInfo<Path>, ?> message, List<Object[]> tuples) {
// We use the expected index for the SensitivityName field here, while in a complete implementation the index would be an instance variable.
tuples.forEach(tuple -> tuple[1] = "TestValue");
datastore.getTransactionManager().addAll(StoreNames.TENOR_STORE_NAME, tuples);
}
@Override
public Collection<String> getTargetStores() {
return Collections.singleton(StoreNames.TENOR_STORE_NAME);
}
}
)
);
}
}
This configuration class should then be included in the @Import
annotation of the MarketRiskConfig
class:
@Import(value = {
...
CustomizationsConfig.class
})
public class MarketRiskConfig {
...
}
Importing this customization class and modifying the input file will result in the following data being loaded into the store:
TenorLabels | SensitivityName | NumberOfDays | TenorIndices | DoubleNumberOfDays | TestData |
---|---|---|---|---|---|
N/A | TestValue | 0.0 | 0 | 0.0 | Testdata |
0.25Y | TestValue | 90.0 | 1 | 180.0 | Testdata |
0.5Y | TestValue | 180.0 | 2 | 360.0 | Testdata |
1Y | TestValue | 360.0 | 3 | 720.0 | Testdata |
2Y | TestValue | 720.0 | 4 | 1440.0 | Testdata |
3Y | TestValue | 1080.0 | 5 | 2160.0 | Testdata |
4Y | TestValue | 1440.0 | 6 | 2880.0 | Testdata |
5Y | TestValue | 1800.0 | 7 | 3600.0 | Testdata |
10Y | TestValue | 3600.0 | 8 | 7200.0 | Testdata |
15Y | TestValue | 5400.0 | 9 | 10800.0 | Testdata |
20Y | TestValue | 7200.0 | 10 | 14400.0 | Testdata |
30Y | TestValue | 10080.0 | 11 | 20160.0 | Testdata |
Helper methods
To simplify the customization process for less complex use-cases, a set of helper methods are included in the Provider
classes.
All methods are static and return a generator Function
that can be used directly in the customization Bean
.
The generated List
will contain a single parameter object containing the defined customizations.
Method | Details | Provider |
---|---|---|
addColumnsFunction(String topic, List<String> columns) |
Adds a list of columns to a topic. Preserves the previously defined calculators. | CsvColumnsProvider |
addCalculatorsFunction(String topic, List<IColumnCalculator<ILineReader>> calculators) |
Adds a list of calculators to a topic, using the topic name as the store name. Preserves the previously defined columns. | CsvColumnsProvider |
addCalculatorsFunction(String topic, Map<String, List<IColumnCalculator<ILineReader>>> calculators) |
Adds a map of calculators per store name to a topic. Preserves the previously defined columns. | CsvColumnsProvider |
addColumnsAndCalcsFunction(String topic, List<String> columns, List<IColumnCalculator<ILineReader>> calculators) |
Adds a list of columns and a list of calculators to a topic, using the topic name as the store name. | CsvColumnsProvider |
addColumnsAndCalcsFunction(String topic, List<String> columns, Map<String, List<IColumnCalculator<ILineReader>>> calculators) |
Adds a list of columns and a map of calculators per store name to a topic. | CsvColumnsProvider |
addPublisher(String topic, ITuplePublisher<IFileInfo> publisher) |
Adds a tuple publisher to a topic. | CsvPublisherProvider |
Suggested Further Reading
- Adding cube hierarchies
- Configuring measures using Spring Beans
- Configuring schema selections using Spring Beans
- Adding a new KPI
- Adding data loading or unloading topics