Skip to main content

Remove data in overlap configuration

The challenge: ensuring query consistency during data removal​

When removing data from a data cube in a distributed setup with data overlap, there is a timing consideration to address:

A query might be planned by the query node before the data removal, but executed on the target data cube after the removal has occurred. Even if the removed members are still available in another data cube, they will be missing from the query results, leading to incorrect or incomplete results.

Remove date setup

Consider this sequence:

  1. Query node receives a query for members that include "Date2"
  2. Query node determines that "Date2" should be retrieved from Data Node 2
  3. Before the query executes, Data Node 2 removes "Date2" from its data
  4. Query executes on Data Node 2, but "Date2" is no longer available
  5. Query results are incomplete, missing data for "Date2"

This timing issue can occur even though the data for "Date2" is duplicated in another data cube, because the query node already decided to retrieve it from Data Node 2.

How to: Safe data removal with masking​

To ensure safe data removal, Atoti provides masking operations directly on the data cube API, which uses a masking mechanism to coordinate data removal with query execution.

The masking concept​

Masking prevents the query node from retrieving specified members from a data cube, even though those members are still physically present in that node. When members are masked:

  1. The query node is notified that certain members should not be retrieved from the data cube
  2. Future queries will retrieve those members from other data cubes (where they are duplicated)
  3. Only after all query nodes acknowledge the masking can the data be safely removed

This ensures that queries planned after the masking will never attempt to retrieve the masked members from the data cube being cleaned up.

API overview​

The masking operations are available directly on data cubes via the IMultiVersionDataActivePivot interface. The main method is maskMembers:

final LevelMembers memberToMask =
    new LevelMembers(LevelIdentifier.simple("Country"), List.of("US", "France"));
final CompletableFuture<IMaskingOperationReport> maskingResult =
    dataNode.maskMembers(memberToMask, branch, retryNumber);

The LevelMembers record encapsulates the distributing level and its associated members to mask.

Important constraints​

Required condition​

Masking is only effective when horizontal data duplication is enabled (IQueryClusterDefinition.HORIZONTAL_DATA_DUPLICATION_PROPERTY is true).

Forbidden configurations​

Masking is not allowed when:

  • The target level provided in the LevelMembers parameter is a distributing level of a query cube that has multiple distributing levels and horizontal data duplication enabled
  • That level is not the distributing level of the application associated with the data cube
warning

The cluster must be stable during the masking operation. In particular, a query cube joining in the middle of a masking may keep trying to retrieve the removed members from the data node because it is not aware of the masking.

warning

The masking operations are forbidden in query cubes where the deprecated IMultiVersionDistributedActivePivot.unloadMembersFromDataNode has already been invoked.

Cross-application considerations​

If a database removal operation removes distributing level members from several data cubes belonging to different applications, you must ensure that appropriate distributing level members are masked for all impacted data cubes before performing the actual removal.

Retry mechanism​

The retryNumber parameter allows automatic retries in case of transient failures such as:

  • Network failures
  • Target branch not yet published when the masking operation is consumed by the query cube

Important: There is no retry if the masking fails because it is forbidden (e.g., multiple distributing levels scenario).

Unmasking​

The IMultiVersionDataActivePivot.unmaskMembers method reverses a masking operation, re-enabling the query cube to retrieve the unmasked members from the data cube.

Unmasking should not be part of a usual data removal workflow. It should only be used to roll back a masking operation that was performed in error or is no longer needed.

Note: Unmasking members that are not currently masked has no effect.

Operation report​

Both maskMembers and unmaskMembers methods return a CompletableFuture<IMaskingOperationReport> that completes once the operation has been processed across all query cubes. The IMaskingOperationReport provides detailed information about the operation outcome:

  • Success status: Whether the operation succeeded across all query cubes
  • Successful query cubes: A list of query cube addresses where the operation was successfully applied
  • Failed query cubes with reasons: A map of query cube addresses to failure reasons for those where the operation failed. In case of repeated failures due to the retry mechanism, only the failure reason from the last attempt is provided.

A failure does not mean the operation failed for all query cubes - the masking or unmasking may be successful for some query cubes but not for others. If masking or unmasking is not successful, the cube continues to operate, but members that are masked cannot be retrieved from the data cube where they are masked.

To observe which members are actually masked from which data cubes, see the Monitoring section.

Example: Cross-application data removal​

This example demonstrates a removal in the datastore impacting two data cubes belonging to distinct applications with different distributing levels.

Setup​

Scenario: Remove facts where Country IN ("US", "France") from the underlying database

Impact on data cubes:

  • Data Cube A (Application A):

    • Distributing level: Country
    • Members to mask: "US", "France"

  • Data Cube B (Application B):

    • Distributing level: Currency
    • Members to mask: "USD", "EUR" (currencies corresponding to the removed facts)

Implementation​

// Create LevelMembers records
final LevelMembers countryMemberToMask =
    new LevelMembers(LevelIdentifier.simple("Country"), List.of("US", "France"));
final LevelMembers currencyMemberToMask =
    new LevelMembers(LevelIdentifier.simple("Currency"), List.of("USD", "EUR"));
// Mask members on both data cubes (with 3 retries)
final CompletableFuture<IMaskingOperationReport> maskingA =
    dataCubeA.maskMembers(countryMemberToMask, IEpoch.MASTER_BRANCH_NAME, 3);
final CompletableFuture<IMaskingOperationReport> maskingB =
    dataCubeB.maskMembers(currencyMemberToMask, IEpoch.MASTER_BRANCH_NAME, 3);
// Wait for both masking operations to complete
CompletableFuture.allOf(maskingA, maskingB).join();
// Check that both masking operations were successful
final IMaskingOperationReport maskingReportA = maskingA.join();
final IMaskingOperationReport maskingReportB = maskingB.join();
if (!maskingReportA.isSuccessful() || !maskingReportB.isSuccessful()) {
  throw new IllegalStateException(
      "Masking failed. Report A: " + maskingReportA + ", Report B: " + maskingReportB);
}
// Now it is safe to remove the data from the database:
try {
  transactionManager.startTransaction("baseStore");
  try {
    transactionManager.removeWhere(
        "baseStore", BaseConditions.in(FieldPath.of("country"), "US", "France"));
  } catch (final Exception exception) {
    transactionManager.rollbackTransaction();
    throw new RuntimeException("Problem while removing data in transaction", exception);
  }
  transactionManager.commitTransaction();
} catch (final Exception exception) {
  // Unmask members on both data cubes (with 3 retries)
  final CompletableFuture<IMaskingOperationReport> unmaskingA =
      dataCubeA.unmaskMembers(countryMemberToMask, IEpoch.MASTER_BRANCH_NAME, 3);
  final CompletableFuture<IMaskingOperationReport> unmaskingB =
      dataCubeB.unmaskMembers(currencyMemberToMask, IEpoch.MASTER_BRANCH_NAME, 3);
  // Wait for both unmasking operations to complete
  CompletableFuture.allOf(unmaskingA, unmaskingB).join();
  // Check that both unmasking operations were successful
  final IMaskingOperationReport unmaskingReportA = unmaskingA.join();
  final IMaskingOperationReport unmaskingReportB = unmaskingB.join();
  if (!unmaskingReportA.isSuccessful() || !unmaskingReportB.isSuccessful()) {
    throw new IllegalStateException(
        "Unmasking failed. Report A: " + unmaskingReportA + ", Report B: " + unmaskingReportB);
  }
}

This approach ensures that:

  1. Both data cubes mask their respective members before any data is removed
  2. All query cubes are aware of the masking before the database removal
  3. Query results remain consistent throughout the operation
  4. The actual data removal only happens after all masking operations complete successfully

Monitoring​

Masked distributing level members can be monitored in two ways:

Using the Java API​

You can programmatically retrieve masked member information using the IDistributedActivePivotVersion API:

final IDistributedActivePivotVersion version = distributedCube.getHead();
final IDistributionInformation distributionInfo = version.getDistributionInformation();
final MemberMapping maskedMembers = distributionInfo.getMaskedMemberMapping();

Using JMX​

The same information can be accessed through JMX using the getDistributingLevelMemberMapping operation in the MBean com.activeviam:node0=ActivePivotManager,node1=<SchemaName>,node2=<CubeName>.