Distribution Properties

This page defines and explains the properties specific to the distributed architecture, and those you can set to modify the default behaviors of both the data and query nodes, and of their associated messengers.

To familiarize yourself with the distributed architecture, we recommend you read the rest of the distributed architecture documentation before continuing.

Query Cube Properties

The query cube properties can be found in the IDistributedActivePivotInstanceDescription interface, and can be set using the fluent builder API IQueryCubeDescriptionBuilder.

Data Replication Property

• setter(s):
  • IQueryCubeDescriptionBuilder#withDataReplication(String value)
  • IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.DATA_REPLICATION, String value)
• default value: REPLICATE_ON_SLICER

The property is used to set the aggregates data replication policy. The query cube can build queries that do not exist locally. As measures from different cubes are independent, Atoti allows the measures to appear on these unknown locations.

The following example explains the different data replication policies:

The distributed cluster is composed of two data cubes and a query cube.

• Data Cube 1 has one hierarchy.
• Data Cube 2 has two hierarchies (distinct from Data Cube 1), Time, that is slicing. The Query Cube represents the merger of the two topologies, and therefore contains three hierarchies.

A query hits the query node with the location AllMember|*|AllMember.

Asking for AllMembers on the year level aggregates the measures over all years, whereas asking for * aggregates the measures for each year, and returns a list of results.

In the data cube 1, the new local location is AllMember. When translating back to the query cube, there are three possibilities:

• REPLICATE: The measures are replicated on all unknown hierarchies and levels. Here, the results are replicated for locations:

  * `AllMember|2019|AllMember`
  * `AllMember|2018|AllMember`
  * `AllMember|2017|AllMember`

• NO_REPLICATION: The measures are not replicated on any unknown location. The result are:

  • AllMember|2019|AllMember in this case, because the hierarchy Time is slicing and 2019, as first member of the hierarchy, it is the default value (unless otherwise specified).
  • AllMember|AllMember|AllMember if the Time hierarchy was not slicing, as the results would be aggregated on the default AllMember.

• REPLICATE_ON_SLICER: The measures are replicated on the first level of unknown hierarchies, and therefore to AllMember for regular hierarchies, or to the first-level member for a slicing hierarchy.

Application Measure Groups

• setter(s):
  • IQueryCubeDescriptionBuilder#withApplicationMeasureGroups(boolean value)
  • IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.APPLICATION_MEASURE_GROUPS, String value)
• default value: "true"

If set to true, distributed hierarchies and measures have the applicationId of the data pivot they come from as measureGroup.

Application Measure Names

• setter(s):
  • IQueryCubeDescriptionBuilder#withApplicationMeasureNames(boolean value)
  • IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.APPLICATION_MEASURE_NAMES, String value)
• default value: "false"

If set to true, the query cubes are allowed to override locally defined measure names. The data pivot's applicationId is added as a suffix.

Application Measure Groups Suffix

• setter(s):
  • IQueryCubeDescriptionBuilder#withApplicationMeasureGroupsSuffix(boolean value)
  • IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.APPLICATION_MEASURE_GROUPS_SUFFIX, String value)
• default value: "false"

If set to true, the query cubes are allowed to append the data pivot's applicationId to locally defined measure groups.

Removal Delay

• setter(s):
  • IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.REMOVAL_DELAY, String value)
• default value: "60"
• unit: second

Timer in seconds after which a complete removal of a given instance's contributions to the query cube topology is triggered. A negative value means no removal, while a null (zero) value means immediate removal.

For more details on this property, see Communication flow and Recommendations.

Pause on Merge Failure

• setter(s):
  • IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.PAUSE_ON_MERGE_FAILURE, String value)
• default value: "false"

If set to true, the query cube pauses in case of a merge failure. A merge failure signifies that the query cube could not merge a data cube's dimensions into its own global topology. This happens when a data cube violates a distributed constraint:

• The uniqueness constraint on the members of the distributed fields within the same application.
• The uniqueness constraint on the measures across different applications. Pausing the data cube stops its messenger, meaning the cube cannot communicate with the rest of the cluster.

Wait for All Applications

• setter(s):
  • IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.WAIT_FOR_ALL_APPLICATIONS, String value)
• default value: "true"

If set to true, the query cube waits until all the data cubes of different applications have joined the cluster before processing their InitialDiscoveryMessages. Since a data cube from a new application entering the cluster can trigger a complete rebuild of the query cube's topology, it can prove more efficient to wait for all JGroups cluster members to have sent their messages before building the query cube's topology.

Transaction Notification

• setter(s):
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IDataClusterDefinition.AWAIT_NOTIFICATIONS_TIME, value) after setting the concealed measures, where value is a string representation of a long value (in seconds) after setting the applicationId
  • IMultiVersionActivePivot.setTransactionProperties(Property property) where property includes the (key,value) entry (AWAIT_NOTIFICATIONS_TIME, value)
• default value: 0

When set to a value greater than 0, the data cube will wait for notifications from other remote instances upon a transaction. Each Query cube will synchronously send a NotificationMessage to the data cube notifying that the transaction has correctly been applied. The data cube will resume after receiving notifications from every remote instance, or after the duration specified by the property has been exceeded.

Pending Initial Discoveries

• setter(s):
  • set activeviam.distribution.maxPendingDiscoveries as an ActiveViamProperty
• default value: 4

The maximum number of pending initial discoveries this instance tolerates before refusing new ones. This property must be set with great care, as the discovery messages are the biggest that transit on the network. The query cube is capable of throttling the discoveries to avoid over-stress. The query node must be able to hold within its heap memory the equivalent of maxPendingDiscoveries * DiscoveryMessageSize in the worst scenario.

Messenger Properties

The following properties can be found in the IMessengerDefinition interface, and can be set using the fluent builder API ICanBuildDataClusterDescriptionBuilder.

Messenger Time Out

• setter(s):
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.MESSENGER_TIMEOUT, String value)
• default value: DEFAULT_MESSAGE_TIMEOUT = 30 000
• unit: millisecond

Timeout used by the messenger when waiting for a reply to any remote message.

Message Size Logging Threshold

• setter(s):
  • set activeviam.distribution.log.size.threshold=<String> as an ActiveViamProperty
• default value: DEFAULT_MESSAGE_TIMEOUT = 4M
• unit: To specify within the argument string. Default unit is empty, meaning a value of 10 is 10 bytes.

System property to configure the threshold (in bytes) above which the size of a IBroadcastMessage sent or received is logged at level INFO.

Auto Start

• setter(s):
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.AUTO_START, String value)
• default value: "true"

If set to true, the distribution infrastructure automatically starts when the query cube starts, sending a message to all data nodes to start.

Hello Message Time Out

• setter(s):
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.HELLO_MESSAGE_TIMEOUT, String value)
• default value: 2 000
• unit: millisecond

Specific timeout used by the messenger when waiting for a reply to a HelloMessage before throwing a timeout exception.

This property only works for data nodes.

Hello Message Number of Trials

• setter(s):
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.HELLO_MESSAGE_NUMBER_TRIALS, String value)
• default value: 10

Number of discovery attempts before giving up. Total wait time is helloMessageNumberTrials * helloMessageTimeout.

This property only works for data nodes.

Retry Unavailable Instances Period

• setter(s):
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.RETRY_UNAVAILABLE_INSTANCES_PERIOD, String value)
• default value: 10
• unit: millisecond

One node of a cluster sends the same IHelloMessage to all members of the cluster. If no answer is received from any of these cluster members within IMessengerDefinition.HELLO_MESSAGE_TIMEOUT, the address of that node is added to the list of ADistributedMessenger#unavailableInstances.

The sender will retry broadcasting the IHelloMessage IMessengerDefinition.HELLO_MESSAGE_NUMBER_TRIALS times, each attempt separated from the previous one by IMessengerDefinition.RETRY_UNAVAILABLE_INSTANCES_PERIOD milliseconds.

Discovery Time Out

• setter(s):
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.DISCOVERY_TIMEOUT, String value)
• default value: twice the value of DEFAULT_MESSAGE_TIMEOUT = 30 000
• unit: millisecond

Specific timeout used by the messenger when waiting for a reply to aIInitialDiscoveryMessage before throwing a timeout exception. If the transmission fails, one more attempt is made to send the data node's topology before the query node is considered unreachable. At this point, it is advised to stop the node and increase this property's value.

This property only works for Data Nodes.

Discovery Serialization Time Out

• setter(s):
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.SERIALIZATION_DISCOVERY_TIMEOUT, String value)
• default value: -1
• unit: millisecond

Specific timeout used by the messenger's serializer, after which the serialization is aborted. Set to a negative value to permit serialization regardless of the time the process takes.

This property only works for Data Nodes.

Transaction Time Out

• setter(s):
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.TRANSACTION_TIMEOUT, String value)
• default value: DEFAULT_MESSAGE_TIMEOUT = 30 000
• unit: millisecond

Specific timeout used by the messenger when waiting for a reply to aITransactionCommittedMessage before throwing a timeout exception. Note that on the next successful transaction, the query node will detect the previous timeout/incorrectly transmitted transaction and will ask the data node to restart.

This property only works for data nodes.

Query Serialization Time Out

• setter(s):
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.SERIALIZATION_QUERY_TIMEOUT, String value)
• default value: -1
• unit: millisecond

Specific timeout used by the messenger's serializer, after which the serialization is aborted. Set to a negative value to permit serialization regardless of the time the process takes.

This property only works for data nodes.

Interruption Check Interval

• setter(s):
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.INTERRUPTION_CHECK_INTERVAL, String value)
• default value: 1 000

The number of write operations during a serialization operation between two interruption checks.

This property only works for data nodes.

Logical Address

• setter(s):
  • set activeviam.logicalAddress=<String> as an ActiveViamProperty
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.LOGICAL_ADDRESS, String value)
• default value: Automatically generated

The logical address of the query or data cube in the cluster. The program property takes precedence over the messenger settings.

Port Range

• setter(s):
  • set activeviam.distribution.netty.portRange=<String> as an ActiveViamProperty
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.PORT_RANGE, String value)
• default value: null

The format must follow this convention: 1000:1010,2000:2010, to allow ports 1000 to 1010 and 2000 to 2010. This is the range of valid ports to which the transporter will bind. It takes the first one available within the range. The program property takes precedence over the messenger settings. A value of null or 0 lets the system pick an ephemeral port in a bind operation.

This property is only relevant to the Netty messenger.

Security Messenger Plugin Key

• setter(s):
  • ICanBuildDataClusterDescriptionBuilder#withProperty( MessengerUtil.SECURITY_MESSENGER_KEY, String value)
• default value: DISTRIBUTED_SECURITY_MANAGER

The plugin key of the security manager for authenticating remote authenticated messages sent to a data node.

This property only works for data nodes.

Message Sizes (Netty Messenger Only)

• setter(s):
  • -Dactiveviam.distribution.nettyMessageMaxSize=<class||shortcut>=<size>m,<class||shortcut>=<size>m... or,
  • Using setMaxSizeForMessage(String classNameOrShortcut, String valueInByte) in the DistributedMessenger MBean operations
• default value:
  • 64MB for IInitialDiscoveryMessage messages
  • 4MB for ITransactionCommittedMessage messages
  • 4MB for ScopedAggregatesRetrievalResultTransporter messages
  • 32MB for DrillthroughMessageWithHeadersAnswers messages
  • 2MB for any other message types

Type	Class to Specify	Shortcut
IInitialDiscoveryMessage	com.activeviam.activepivot.dist.impl.private_.messenger.message.IInitialDiscoveryMessage	initialDiscovery
ITransactionCommittedMessage	com.activeviam.activepivot.dist.impl.private_.messenger.message.ITransactionCommittedMessage	transactionCommitted
ScopedAggregatesRetrievalResultTransporter	com.activeviam.activepivot.dist.impl.internal.cellset.impl.ScopedAggregatesRetrievalResultTransporter	singleGaqResult
DrillthroughMessageWithHeadersAnswer	com.activeviam.activepivot.dist.impl.private_.query.impl.DrillthroughMessageWithHeaders.DrillthroughMessageWithHeadersAnswer	drillthroughQueryResult
DrillthroughHeader	com.activeviam.activepivot.core.impl.internal.context.drillthrough.impl.DrillthroughHeader	drillthroughHeader

Note that the maximum message size should always be higher than the size of the allocated buffer when writing the stream. By default, this buffer's size is set to StreamObjectEncoder#DEFAULT_CHUNK_SIZE + NettyStreamUtils#HEADER_LEN = 1MB + Integer.BYTES = 1.004MB . An exception is thrown if a message size is lower than this buffer's size, and the messages will never be sent.

When exceeding a given message type limit, an IOException is thrown, with an error message specifying the message type limit, for example, "Cannot send a message larger than 1MiB (1048576)". The table below shows that depending on the type of the message exceeding the maximum limit, this error could be critical or less.

Type	Consequence
IInitialDiscoveryMessage	The discovery is never applied and the query cube will never be able to see the data cube hierarchies. The thrown exception can be seen in the logs.
ITransactionCommittedMessage	The updates of the corresponding data cube committed transaction will not be visible by the query cube. The thrown exception can be seen in the logs.
ScopedAggregatesRetrievalResultTransporter	The query will be interrupted and the exception message logged and displayed in Atoti UI
DrillthroughMessageWithHeadersAnswer	The query will be interrupted and the exception message logged and displayed in Atoti UI
DrillthroughHeader	The query will be interrupted and the exception message logged and displayed in Atoti UI