> ## Documentation Index
> Fetch the complete documentation index at: https://docs.activeviam.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Handling distributed dates

AsOfDate is the distributing level used in Atoti Market Risk. This means that when using horizontal distribution, Atoti Server typically expects
each data node to contain a subset of the available AsOfDates. The query node will direct queries to the appropriate data node based on the AsOfDate requested.

However, some use-cases and operational workflows may require your database to contain all available dates. In-memory data nodes contain a subset of these dates
(typically the most recent dates). This means there is an overlap of dates between the in-memory data node and the DirectQuery data node. This can be resolved in
one of two ways:

* [Enable data overlap](#enable-data-overlap)
* [Filter dates to include](#filter-dates-to-include)

## Enable data overlap

The simplest solution is to enable the [data overlap feature](https://docs.activeviam.com/products/atoti/server/6.1.11/docs/distributed/data_overlap/) of Atoti Server. In this
mode, it is permitted for multiple data nodes to contain the same AsOfDate. You must enable data overlap on the query node by setting the following property:

```yaml theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
mr:
  combined:
   distribution:
    enable-data-duplication: true
```

The query node will choose one of the underlying data nodes to handle the query. By default, this will be the node with the fewest AsOfDates (typically the
in-memory node), but you can specify which node should be preferred in the case of overlapping dates by setting a priority for each node. Each of the cubes
has a specific property:

```yaml theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
mr:
  pnl:
    distribution:
      cube-priority: <set priority here. Relative to other nodes of the same type>
  pnl-summary:
    distribution:
      cube-priority: <set priority here. Relative to other nodes of the same type>
  sensi:
    distribution:
      cube-priority: <set priority here. Relative to other nodes of the same type>
  sensi-summary:
    distribution:
      cube-priority: <set priority here. Relative to other nodes of the same type>
  var:
    distribution:
      cube-priority: <set priority here. Relative to other nodes of the same type>
  var-summary:
    distribution:
      cube-priority: <set priority here. Relative to other nodes of the same type>
```

## Filter dates to include

When using DirectQuery, you may want to only include a subset of data available on the remote database. You can configure this through a custom bean.

The `SP_QUALIFIER__DIRECT_QUERY_DATES_INITIAL_LOAD_FILTER_CONDITION` and `SP_QUALIFIER__DIRECT_QUERY_DATES_ACTIVEPIVOT_INSTANCE_DESCRIPTION_FILTER_CONDITION`
beans are used to tell Atoti Market Risk which dates to include in (or exclude from) the DirectQuery data node cube.
Two beans are necessary because one bean is an instance of `com.qfs.condition.ICondition` and the other is an instance of `com.quartetfs.fwk.filtering.ICondition`.
The filtering logic of both beans needs to be identical.
These beans can be entirely customized to create a condition on which dates to include in the DirectQuery data node.

### Default implementation

The default implementation of the `SP_QUALIFIER__DIRECT_QUERY_DATES_INITIAL_LOAD_FILTER_CONDITION` and
`SP_QUALIFIER__DIRECT_QUERY_DATES_ACTIVEPIVOT_INSTANCE_DESCRIPTION_FILTER_CONDITION` beans excludes all dates that were provided to the
`mr.data-load.initial-business-dates` property. This is so that when running with an in-memory data node, the DirectQuery and in-memory node don’t contain any of
the same `AsOfDates`.

The default beans are defined in `InitialDataDirectQueryConfig` as:

```java theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
@Bean
@Qualifier(SP_QUALIFIER__DIRECT_QUERY_DATES_INITIAL_LOAD_FILTER_CONDITION)
public com.qfs.condition.ICondition filterAsOfDates(){

    if(initialLoadAsOfDates == null){
    return null;
    }

    return not(in(FieldPath.of(StoreFieldConstants.AS_OF_DATE), initialLoadAsOfDates));
}

@Bean
@Qualifier(SP_QUALIFIER__DIRECT_QUERY_DATES_ACTIVEPIVOT_INSTANCE_DESCRIPTION_FILTER_CONDITION)
public com.quartetfs.fwk.filtering.ICondition filterAsOfDatesFwk(){

    if(initialLoadAsOfDates == null){
    return null;
    }

    List<com.quartetfs.fwk.filtering.ICondition> conditions = initialLoadAsOfDates.stream()
    .map(date -> new SubCondition(
        "AsOfDate",
        new NotCondition(new EqualCondition(date))
        )
    ).collect(Collectors.toList());

    return new AndCondition(conditions);
}
```

### Custom implementations

To create a custom implementation, you must create your own `@Primary` `@Bean` with a `@Qualifier` of
`SP_QUALIFIER__DIRECT_QUERY_DATES_INITIAL_LOAD_FILTER_CONDITION` and `SP_QUALIFIER__DIRECT_QUERY_DATES_ACTIVEPIVOT_INSTANCE_DESCRIPTION_FILTER_CONDITION`
that returns an instance of `com.qfs.condition.ICondition` and an instance of `com.quartetfs.fwk.filtering.ICondition` respectively.
This conditions can be thought of as what to include in the DirectQuery data node.

In the instance of `com.quartetfs.fwk.filtering.ICondition`, each condition included must be a `SubCondition`, where the `AsOfDate` is used as the matching field.

### Examples

Here are some simple examples of how to configure this bean.

#### Include all available data in DirectQuery

If we wanted to include all data available on our remote database, we simply return null and there will be no condition on what data can be loaded into the
DirectQuery data node:

```java theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
    @Bean
    @Qualifier(SP_QUALIFIER__DIRECT_QUERY_DATES_INITIAL_LOAD_FILTER_CONDITION)
    public com.qfs.condition.ICondition filterAsOfDates(){
        return null;
    }

    @Bean
    @Qualifier(SP_QUALIFIER__DIRECT_QUERY_DATES_ACTIVEPIVOT_INSTANCE_DESCRIPTION_FILTER_CONDITION)
    public com.quartetfs.fwk.filtering.ICondition filterAsOfDatesFwk(){
        return null;
    }
```

#### Only include specific dates

```java theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
    @Bean
    @Primary
    @Qualifier(SP_QUALIFIER__DIRECT_QUERY_DATES_INITIAL_LOAD_FILTER_CONDITION)
    public com.qfs.condition.ICondition filterAsOfDates(){
        // Restrict DirectQuery node to INCLUDE the specific dates:
        List<LocalDate> asOfDatesToLoad = Arrays.asList(LocalDate.of(2020, 1, 1),
            LocalDate.of(2020, 1, 2),
            LocalDate.of(2020, 1, 3));

        return not(in(FieldPath.of(StoreFieldConstants.AS_OF_DATE), asOfDatesToLoad));
    }

	@Bean
	@Primary
    @Qualifier(SP_QUALIFIER__DIRECT_QUERY_DATES_ACTIVEPIVOT_INSTANCE_DESCRIPTION_FILTER_CONDITION)
	public com.quartetfs.fwk.filtering.ICondition customFilterCondition(){
		// Restrict DirectQuery node to INCLUDE the specific dates:
		List<SubCondition> includeConditions = new ArrayList<>();

        includeConditions.add(new SubCondition(AS_OF_DATE, new EqualCondition(LocalDate.of(2020, 1, 1))));
        includeConditions.add(new SubCondition(AS_OF_DATE, new EqualCondition(LocalDate.of(2020, 1, 2))));
        includeConditions.add(new SubCondition(AS_OF_DATE, new EqualCondition(LocalDate.of(2020, 1, 3))));

		return new OrCondition(includeConditions);
	}
```