Handling distributed dates
AsOfDate is the distributing level used in Atoti Market Risk. This means that when using horizontal distribution, Atoti Server typically expects each data node to contain a subset of the available AsOfDates. The query node will direct queries to the appropriate data node based on the AsOfDate requested.
However, some use-cases and operational workflows may require your database to contain all available dates. In-memory data nodes contain a subset of these dates (typically the most recent dates). This means there is an overlap of dates between the in-memory data node and the DirectQuery data node. This can be resolved in one of two ways:
Enable data overlap
The simplest solution is to enable the data overlap feature of Atoti Server. In this mode, it is permitted for multiple data nodes to contain the same AsOfDate. You must enable data overlap on the query node by setting the following property:
mr:
combined:
distribution:
enable-data-duplication: true
The query node will choose one of the underlying data nodes to handle the query. By default, this will be the node with the fewest AsOfDates (typically the in-memory node), but you can specify which node should be preferred in the case of overlapping dates by setting a priority for each node. Each of the cubes has a specific property:
mr:
pnl:
distribution:
cube-priority: <set priority here. Relative to other nodes of the same type>
pnl-summary:
distribution:
cube-priority: <set priority here. Relative to other nodes of the same type>
sensi:
distribution:
cube-priority: <set priority here. Relative to other nodes of the same type>
sensi-summary:
distribution:
cube-priority: <set priority here. Relative to other nodes of the same type>
var:
distribution:
cube-priority: <set priority here. Relative to other nodes of the same type>
var-summary:
distribution:
cube-priority: <set priority here. Relative to other nodes of the same type>
Filter dates to include
When using DirectQuery, you may want to only include a subset of data available on the remote database. You can configure this through a custom bean.
The SP_QUALIFIER__DIRECT_QUERY_DATES_INITIAL_LOAD_FILTER_CONDITION and SP_QUALIFIER__DIRECT_QUERY_DATES_ACTIVEPIVOT_INSTANCE_DESCRIPTION_FILTER_CONDITION
beans are used to tell Atoti Market Risk which dates to include in (or exclude from) the DirectQuery data node cube.
Two beans are necessary because one bean is an instance of com.qfs.condition.ICondition and the other is an instance of com.quartetfs.fwk.filtering.ICondition.
The filtering logic of both beans needs to be identical.
These beans can be entirely customized to create a condition on which dates to include in the DirectQuery data node.
Default implementation
The default implementation of the SP_QUALIFIER__DIRECT_QUERY_DATES_INITIAL_LOAD_FILTER_CONDITION and
SP_QUALIFIER__DIRECT_QUERY_DATES_ACTIVEPIVOT_INSTANCE_DESCRIPTION_FILTER_CONDITION beans excludes all dates that were provided to the
mr.data-load.initial-business-dates property. This is so that when running with an in-memory data node, the DirectQuery and in-memory node don’t contain any of
the same AsOfDates.
The default beans are defined in InitialDataDirectQueryConfig as:
@Bean
@Qualifier(SP_QUALIFIER__DIRECT_QUERY_DATES_INITIAL_LOAD_FILTER_CONDITION)
public com.qfs.condition.ICondition filterAsOfDates(){
if(initialLoadAsOfDates == null){
return null;
}
return not(in(FieldPath.of(StoreFieldConstants.AS_OF_DATE), initialLoadAsOfDates));
}
@Bean
@Qualifier(SP_QUALIFIER__DIRECT_QUERY_DATES_ACTIVEPIVOT_INSTANCE_DESCRIPTION_FILTER_CONDITION)
public com.quartetfs.fwk.filtering.ICondition filterAsOfDatesFwk(){
if(initialLoadAsOfDates == null){
return null;
}
List<com.quartetfs.fwk.filtering.ICondition> conditions = initialLoadAsOfDates.stream()
.map(date -> new SubCondition(
"AsOfDate",
new NotCondition(new EqualCondition(date))
)
).collect(Collectors.toList());
return new AndCondition(conditions);
}
Custom implementations
To create a custom implementation, you must create your own @Primary @Bean with a @Qualifier of
SP_QUALIFIER__DIRECT_QUERY_DATES_INITIAL_LOAD_FILTER_CONDITION and SP_QUALIFIER__DIRECT_QUERY_DATES_ACTIVEPIVOT_INSTANCE_DESCRIPTION_FILTER_CONDITION
that returns an instance of com.qfs.condition.ICondition and an instance of com.quartetfs.fwk.filtering.ICondition respectively.
This conditions can be thought of as what to include in the DirectQuery data node.
In the instance of com.quartetfs.fwk.filtering.ICondition, each condition included must be a SubCondition, where the AsOfDate is used as the matching field.
Examples
Here are some simple examples of how to configure this bean.
Include all available data in DirectQuery
If we wanted to include all data available on our remote database, we simply return null and there will be no condition on what data can be loaded into the DirectQuery data node:
@Bean
@Qualifier(SP_QUALIFIER__DIRECT_QUERY_DATES_INITIAL_LOAD_FILTER_CONDITION)
public com.qfs.condition.ICondition filterAsOfDates(){
return null;
}
@Bean
@Qualifier(SP_QUALIFIER__DIRECT_QUERY_DATES_ACTIVEPIVOT_INSTANCE_DESCRIPTION_FILTER_CONDITION)
public com.quartetfs.fwk.filtering.ICondition filterAsOfDatesFwk(){
return null;
}
Only include specific dates
@Bean
@Primary
@Qualifier(SP_QUALIFIER__DIRECT_QUERY_DATES_INITIAL_LOAD_FILTER_CONDITION)
public com.qfs.condition.ICondition filterAsOfDates(){
// Restrict DirectQuery node to INCLUDE the specific dates:
List<LocalDate> asOfDatesToLoad = Arrays.asList(LocalDate.of(2020, 1, 1),
LocalDate.of(2020, 1, 2),
LocalDate.of(2020, 1, 3));
return not(in(FieldPath.of(StoreFieldConstants.AS_OF_DATE), asOfDatesToLoad));
}
@Bean
@Primary
@Qualifier(SP_QUALIFIER__DIRECT_QUERY_DATES_ACTIVEPIVOT_INSTANCE_DESCRIPTION_FILTER_CONDITION)
public com.quartetfs.fwk.filtering.ICondition customFilterCondition(){
// Restrict DirectQuery node to INCLUDE the specific dates:
List<SubCondition> includeConditions = new ArrayList<>();
includeConditions.add(new SubCondition(AS_OF_DATE, new EqualCondition(LocalDate.of(2020, 1, 1))));
includeConditions.add(new SubCondition(AS_OF_DATE, new EqualCondition(LocalDate.of(2020, 1, 2))));
includeConditions.add(new SubCondition(AS_OF_DATE, new EqualCondition(LocalDate.of(2020, 1, 3))));
return new OrCondition(includeConditions);
}