Skip to main content
Data overlap is a feature in a horizontal distribution setup that allows replicating all or part of the data across multiple data nodes. It helps improve availability, performance, and resilience in distributed environments.
In a distributed setup, each member of the distributing level represents a partition of the data.
The query node automatically splits queries across data nodes without duplicating computations on replicated partitions.

​
When to use Data Overlap?

Data overlap is mainly used in three scenarios:
  • Failover support:
    Each data node can be fully replicated so that if one node goes offline, another can take over.
  • DirectQuery hybrid setup:
    A DirectQuery node holds all data, while a Datastore node contains recent partitions for faster queries. If the Datastore node goes offline, the DirectQuery node serves queries. In this setup, data overlap facilitates the data roll over process.
  • Load balancing:
    Frequently accessed partitions can be replicated across multiple nodes to distribute query load. When nodes have the same priority, queries are randomly distributed for automatic load balancing. Note that this is true random selection — there is no guarantee of even distribution across nodes.

​
Does data overlap have constraints or limitations?

  • The application must have at least one distributing level.
  • Replicated partitions must be identical across all data nodes. Improper synchronization will lead to inconsistent results.
  • Data overlap is defined at the query node level. All applications in the distributed setup must comply with these rules.
Atoti does not enforce correct data replication. It is the user’s responsibility to keep replicated partitions synchronized at all times.

​
How query dispatching works with data overlap

The query node maintains a map of which data node contains which partitions of the distributing level. It knows when a partition is replicated. When executing a query:
  • Non-replicated partitions are processed normally.
  • For replicated partitions:
    • The node with the highest priority is selected.
    • Priority is a strictly positive integer; lower values mean higher priority. Zero and negative values are not allowed.
    • Define priority using IDataClusterDefinition#DATA_NODE_PRIORITY. For details, see priority configuration section.
If no priority is defined by the user:
  • The dispatching algorithm selects the node with the fewest distributing level members.
If at least one priority is defined:
  • Nodes without priority default to Integer.MAX_VALUE (lowest priority).
  • If nodes have the same priority, the dispatching algorithm randomly selects one at query time. This is true random selection: there is no explicit sequencing. There is no user session affinity — a user is not tied to a specific data node.

​
How member restrictions are pushed to data nodes

When dispatching a query, the query node restricts each data node to the distributing level members it must serve. Consecutive members served by the same data node are collapsed into a single range condition, a greater-or-equal / less-or-equal bound, instead of one condition per member. This applies to members statically attributed to a single data node, and to replicated partitions dispatched at query time under equal priority: the randomly selected node receives the whole replicated run as one range.
 On a DirectQuery data node, this restriction becomes part of the SQL query sent to the external database. Collapsing members into a range keeps that predicate compact, even when a data node serves a wide window of members. Some external databases reject a predicate that enumerates too many literals: for example, Dremio rejects an IN predicate above roughly 1,500 literals, which is about 5 to 6 years of daily history for a day-level distributing level.
  • Range collapsing applies only when the distributing level’s members are of a temporal or numeric type.
  • Distributing levels with string-typed members keep the previous one-condition-per-member restriction.
A range condition relies on the external database ordering members the same way Java does. String collation in the external database may order strings differently than Java, so a range evaluated under a different ordering could silently select the wrong partitions. This is why string-typed distributing levels keep one condition per member instead of being collapsed into a range.
As with any direct, non-distributed, query on that data node, a range condition can match rows whose member value appeared in the external database after the last refresh.

​
Example of query dispatching

This example illustrates how query dispatching works in a distributed setup with overlapping data across two nodes.

​
Setup

Distributing level: Country Data node priority: Node A has higher priority than Node B Data distribution: Overlapping partitions: Germany, Italy (present in node A and node B)

​
Query examples

​
Query with point coordinate Country = "Germany"

  1. Point coordinate: Country = Germany
  2. List coordinate: Country IN [Italy, Spain]
  3. Wildcard coordinate: Country = *
Results:

​
How to enable data overlap

Activate the feature at startup:
This activation applies to all distributed applications. All applications must respect the constraints defined above.

​
How to configure data node priority

Set the priority at startup:
Priority values must be strictly positive integers. Zero and negative values are not allowed. Nodes without the property default to Integer.MAX_VALUE. If no nodes have priority, the system prioritizes nodes with fewer distributing level members.

​
How to monitor data overlap

Retrieve data partition mapping and node priority from the query node in two ways:

​
Use the Java API

Use the IDistributedActivePivotVersion API:

​
Use JMX

Access the same information through JMX using the getDistributingLevelMemberMapping operation in the MBean com.activeviam:node0=ActivePivotManager,node1=<SchemaName>,node2=<CubeName>.