Skip to main content

Distributed Date Roll Over

A classic use-case for Distribution is to partition data cubes by time. For example, one can put the most recent dates into an in-memory cube while other dates are in an external database with a DirectQuery cube. With such design comes the need to move the data corresponding to the oldest dates from the in-memory cube to the DirectQuery cube, so we free some memory for the next dates. This process is called Date Roll Over.

Here is an example where we want to move date 101 from the in-memory cube to the DirectQuery cube so we free some memory for date 104:

Before rolling date flow

Atomicity​

One difficulty of this date roll over is the Atomicity of the operation. On one hand, we cannot have the same date in both cubes as it is not supported by Atoti Distribution. On the other hand, if we first delete the data from the in-memory cube and then insert it into the DirectQuery cube, it creates a laps of time when the data is in neither cube.

To solve this issue, a solution is to disable the queries on the query cube during the roll-over. This way, we can first delete the data from the in-memory cube, then insert it into the DirectQuery cube, and finally re-enable the queries on the query cube. The data is temporary in neither cube, but the queries are not executed during this time so the users do not see any inconsistency.

Example​

One way to block the query is to use the experimental feature QueryBlockerUtil to reject all queries and to unregister all the queries from the cube.

In this example, we implement a Spring controller which allows disabling and re-enabling the queries on the cube.

@RestController
@RequestMapping("/access")
public class AccessController {
  private final IActivePivotManager manager;
  @Autowired
  public AccessController(final IActivePivotConfig pivotConfig) {
    this.manager = pivotConfig.activePivotManager();
  }
  @GetMapping("/enable")
  public String enableQueries() {
    QueryBlockerUtil.allowQueries(this.manager);
    return "Access is now enabled.";
  }
  @GetMapping("/disable")
  public String disableQueries() {
    QueryBlockerUtil.blockQueries(this.manager);
    return "Access is now disabled.";
  }
}

QueryBlockerUtil is an experimental feature and is subject to change in future releases, it requires the feature flag activeviam.feature.experimental.query_blocker.enabled to be set to true.

Keep in mind the following limitations:

  • The cube will be un-usable for a short period of time
  • The same date cannot be present in several data cubes at the same time

Workflow​

To be optimal, the roll-over workflow must minimize the time when the queries are not available. Here is an example workflow with an external database supporting time travel:

  • Insert the data of the rolled date in the external database but do not trigger incremental refresh. This data is not used yet as the DirectQuery node use a previous version of the data with Time Travel.
  • Block the queries
  • Remove the rolled date from the in-memory cube
  • Run an incremental refresh on the DirectQuery node to add the rolled date
  • Unblock the queries
  • Add the new date in the in-memory cube

Future developments​

In the future, Atoti will provide a more integrated way to handle the Date Roll Over operation without the need to disable the queries on the cube.