Distribution Properties

Documentation Index

Fetch the complete documentation index at: https://docs.activeviam.com/llms.txt

Use this file to discover all available pages before exploring further.

This page defines and explains the properties specific to the distributed architecture, and those you can set to modify the default behaviors of both the data and query nodes, and of their associated messengers.

To familiarize yourself with the distributed architecture, we recommend you read the rest of the distributed architecture documentation before continuing.

​

Query Cube Properties

The query cube properties can be found in the IDistributedActivePivotInstanceDescription interface, and can be set using the fluent builder API IQueryCubeDescriptionBuilder.

​

Data Replication Property

• setters:
  • IQueryCubeDescriptionBuilder#withDataReplication(String value)
  • IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.DATA_REPLICATION, String value)
• default value: REPLICATE_ON_SLICER

The property is used to set the aggregates data replication policy. The query cube can build queries that do not exist locally. As measures from different cubes are independent, Atoti allows the measures to appear on these unknown locations. The following example explains the different data replication policies: The distributed cluster is composed of two data cubes and a query cube.

• Data Cube 1 has one hierarchy.
• Data Cube 2 has two hierarchies (distinct from Data Cube 1), Time, that is slicing. The Query Cube represents the merger of the two topologies, and therefore contains three hierarchies.

A query hits the query node with the location AllMember|*|AllMember.

Asking for AllMembers on the year level aggregates the measures over all years, whereas asking for * aggregates the measures for each year, and returns a list of results.

In the data cube 1, the new local location is AllMember. When translating back to the query cube, there are three possibilities:

• REPLICATE: The measures are replicated on all unknown hierarchies and levels. Here, the results are replicated for locations:
  • AllMember|2019|AllMember
  • AllMember|2018|AllMember
  • AllMember|2017|AllMember
• NO_REPLICATION: The measures are not replicated on any unknown location. The result are:
  • AllMember|2019|AllMember in this case, because the hierarchy Time is slicing and 2019, as first member of the hierarchy, it is the default value (unless otherwise specified).
  • AllMember|AllMember|AllMember if the Time hierarchy was not slicing, as the results would be aggregated on the default AllMember.
• REPLICATE_ON_SLICER: The measures are replicated on the first level of unknown hierarchies, and therefore to AllMember for regular hierarchies, or to the first-level member for a slicing hierarchy. This mode does not allow replication of data if the cube filter contains restrictions on unknown hierarchies.
• REPLICATE_ON_SLICER_IGNORING_FILTER: The measures are replicated on the first level of unknown hierarchies, and therefore to AllMember for regular hierarchies, or to the first-level member for a slicing hierarchy. This mode allows replication of data even if the cube filter contains restrictions on unknown hierarchies.

​

Application Measure Groups

• setters:
  • IQueryCubeDescriptionBuilder#withApplicationMeasureGroups(boolean value)
  • IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.APPLICATION_MEASURE_GROUPS, String value)
• default value: "true"

If set to true, distributed hierarchies and measures have the applicationId of the data pivot they come from as measureGroup.

​

Application Measure Names

• setters:
  • IQueryCubeDescriptionBuilder#withApplicationMeasureNames(boolean value)
  • IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.APPLICATION_MEASURE_NAMES, String value)
• default value: "false"

If set to true, the query cubes are allowed to override locally defined measure names. The data pivot’s applicationId is added as a suffix.

​

Application Measure Groups Suffix

• setters:
  • IQueryCubeDescriptionBuilder#withApplicationMeasureGroupsSuffix(boolean value)
  • IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.APPLICATION_MEASURE_GROUPS_SUFFIX, String value)
• default value: "false"

If set to true, the query cubes are allowed to append the data pivot’s applicationId to locally defined measure groups.

​

Removal Delay

• setter:
  • IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.REMOVAL_DELAY, String value)
• default value: "60"
• unit: second

Timer in seconds after which a complete removal of a given instance’s contributions to the query cube topology is triggered. A negative value means no removal, while a null (zero) value means immediate removal.

For more details on this property, see Communication flow and Recommendations.

​

Pause on Merge Failure

• setter:
  • IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.PAUSE_ON_MERGE_FAILURE, String value)
• default value: "false"

If set to true, the query cube pauses in case of a merge failure. A merge failure signifies that the query cube could not merge a data cube’s dimensions into its own global topology. This happens when a data cube violates a distributed constraint:

• The uniqueness constraint on the members of the distributed fields within the same application.
• The uniqueness constraint on the measures across different applications. Pausing the data cube stops its messenger, meaning the cube cannot communicate with the rest of the cluster.

​

Wait for All Applications

• setter:
  • IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.WAIT_FOR_ALL_APPLICATIONS, String value)
• default value: "true"

If set to true, the query cube waits until all the data cubes of different applications have joined the cluster before processing their InitialDiscoveryMessages. Since a data cube from a new application entering the cluster can trigger a complete rebuild of the query cube’s topology, it can prove more efficient to wait for all JGroups cluster members to have sent their messages before building the query cube’s topology.

​

Transaction Notification

• setters:
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IDataClusterDefinition.AWAIT_NOTIFICATIONS_TIME, value) after setting the concealed measures, where value is a string representation of a long value (in seconds) after setting the applicationId
  • IMultiVersionActivePivot.setTransactionProperties(Property property) where property includes the (key,value) entry (AWAIT_NOTIFICATIONS_TIME, value)
• default value: 0

When set to a value greater than 0, the data cube will wait for notifications from other remote instances upon a transaction. Each Query cube will synchronously send a NotificationMessage to the data cube notifying that the transaction has correctly been applied. The data cube will resume after receiving notifications from every remote instance, or after the duration specified by the property has been exceeded.

​

Pending Initial Discoveries

• setter:
  • set activeviam.distribution.maxPendingDiscoveries as an ActiveViamProperty
• default value: 4

The maximum number of pending initial discoveries this instance tolerates before refusing new ones. This property must be set with great care, as the discovery messages are the biggest that transit on the network. The query cube is capable of throttling the discoveries to avoid over-stress. The query node must be able to hold within its heap memory the equivalent of maxPendingDiscoveries * DiscoveryMessageSize in the worst scenario.

​

Messenger Properties

The following properties can be found in the IMessengerDefinition interface, and can be set using the fluent builder API ICanBuildDataClusterDescriptionBuilder.

​

Messenger timeout

• setter:
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.MESSENGER_TIMEOUT, String value)
• default value: DEFAULT_MESSAGE_TIMEOUT = 30 000
• unit: millisecond

Timeout used by the messenger when waiting for a reply to any remote message.

​

Message Size Logging Threshold

• setter:
  • set activeviam.distribution.log.size.threshold=<String> as an ActiveViamProperty
• default value: DEFAULT_MESSAGE_TIMEOUT = 4M
• unit: To specify within the argument string. Default unit is empty, meaning a value of 10 is 10 bytes.

System property to configure the threshold (in bytes) above which the size of a IBroadcastMessage sent or received is logged at level INFO.

​

Auto Start

• setter:
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.AUTO_START, String value)
• default value: "true"

If set to true, the distribution infrastructure automatically starts when the query cube starts, sending a message to all data nodes to start.

​

Hello Message timeout

• setter:
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.HELLO_MESSAGE_TIMEOUT, String value)
• default value: 2000
• unit: millisecond

Specific timeout used by the messenger when waiting for a reply to a HelloMessage before throwing a timeout exception.

This property only works for data nodes.

​

Hello Message Number of Trials

• setter:
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.HELLO_MESSAGE_NUMBER_TRIALS, String value)
• default value: 10

Number of discovery attempts before giving up. Maximum wait time is helloMessageNumberTrials * helloMessageTimeout.

This property only works for data nodes.

​

Goodbye Message timeout

• setter:
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.GOODBYE_MESSAGE_TIMEOUT, String value)
• default value: 2000
• unit: millisecond

Specific timeout used by the messenger when waiting for a reply to a GoodbyeMessage before throwing a timeout exception.

This property only works for data nodes.

​

Goodbye Message Number of Trials

• setter:
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.GOODBYE_MESSAGE_NUMBER_TRIALS, String value)
• default value: 3

Number of trials for a data node to leave the cluster. Maximum wait time is goodbyeMessageNumberTrials * goodbyeMessageTimeout.

This property only works for data nodes.

​

Retry Unavailable Instances Period

• setter:
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.RETRY_UNAVAILABLE_INSTANCES_PERIOD, String value)
• default value: 10
• unit: millisecond

One node of a cluster sends the same IHelloMessage to all members of the cluster. If no answer is received from any of these cluster members within IMessengerDefinition.HELLO_MESSAGE_TIMEOUT, the address of that node is added to the list of ADistributedMessenger#unavailableInstances. The sender will retry broadcasting the IHelloMessage IMessengerDefinition.HELLO_MESSAGE_NUMBER_TRIALS times, each attempt separated from the previous one by IMessengerDefinition.RETRY_UNAVAILABLE_INSTANCES_PERIOD milliseconds.

​

Discovery timeout

• setter:
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.DISCOVERY_TIMEOUT, String value)
• default value: twice the value of DEFAULT_MESSAGE_TIMEOUT = 30 000
• unit: millisecond

Specific timeout used by the messenger when waiting for a reply to an initial discovery message (Message sent by a data node to initially share its structure to the rest of the distributed application) before throwing a timeout exception. If the transmission fails, one more attempt is made to send the data node’s topology before the query node is considered unreachable. At this point, it is advised to stop the node and increase this property’s value.

This property only works for Data Nodes.

​

Discovery Serialization timeout

• setter:
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.SERIALIZATION_DISCOVERY_TIMEOUT, String value)
• default value: -1
• unit: millisecond

Specific timeout used by the messenger’s serializer, after which the serialization is aborted. Set to a negative value to permit serialization regardless of the time the process takes.

This property only works for Data Nodes.

​

Transaction timeout

• setter:
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.TRANSACTION_TIMEOUT, String value)
• default value: DEFAULT_MESSAGE_TIMEOUT = 30 000
• unit: millisecond

Specific timeout used by the messenger when waiting for a reply to a transaction committed message (Message exchanged between nodes in a distributed cluster, that represents a committed transaction in the sending node) before throwing a timeout exception. Note that on the next successful transaction, the query node will detect the previous timeout/incorrectly transmitted transaction and will ask the data node to restart.

This property only works for data nodes.

​

Query Serialization timeout

• setter:
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.SERIALIZATION_QUERY_TIMEOUT, String value)
• default value: -1
• unit: millisecond

Specific timeout used by the messenger’s serializer, after which the serialization is aborted. Set to a negative value to permit serialization regardless of the time the process takes.

This property only works for data nodes.

​

Unload members from node timeout

• setter:
  • PluginDescriptionBuilderWithKey<IBuildableMessengerDescriptionBuilder<>>#withProperty( IMessengerDefinition.UNLOAD_MEMBERS_MESSAGE_TIMEOUT, String value)
• default value: 30 minutes
• unit: millisecond

Specific timeout used by the messenger when waiting for a reply to an unload members from node message (Message sent by a query node to a data node in order to atomically unload specified distributing level members from this data node) before throwing a timeout exception.

This property only works for query nodes.

​

Masking operation timeout

• setter(s):
  • PluginDescriptionBuilderWithKey<IBuildableMessengerDescriptionBuilder<>>#withProperty( IMessengerDefinition.MASKING_OPERATION_MESSAGE_TIMEOUT, String value)
• default value: 5 minutes
• unit: millisecond

Specific timeout used by the messenger when waiting for a reply to a masking operation message (Message sent by a data node to the query nodes in order to mask/unmask specified distributing level members, preventing future queries from retrieving masked members from that node) before throwing a timeout exception.

This property only works for data nodes.

​

Interruption Check Interval

• setter:
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.INTERRUPTION_CHECK_INTERVAL, String value)
• default value: 1000

The number of write operations during a serialization operation between two interruption checks.

This property only works for data nodes.

​

Logical Address

• setters:
  • set activeviam.logicalAddress=<String> as an ActiveViamProperty
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.LOGICAL_ADDRESS, String value)
• default value: Automatically generated

The logical address of the query or data cube in the cluster. The program property takes precedence over the messenger settings.

​

Port Range

• setters:
  • set activeviam.distribution.netty.portRange=<String> as an ActiveViamProperty
  • ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.PORT_RANGE, String value)
• default value: null

The format must follow this convention: 1000:1010,2000:2010, to allow ports 1000 to 1010 and 2000 to 2010. This is the range of valid ports to which the transporter will bind. It takes the first one available within the range. The program property takes precedence over the messenger settings. A value of null or 0 lets the system pick an ephemeral port in a bind operation.

This property is only relevant to the Netty messenger.

​

Keep alive

• setter:
  • set activeviam.distribution.netty.keepAlive=<Boolean> as an ActiveViamProperty
• default value: false

The TCP keep-alive configuration for the Netty messenger. If enabled, the underlying TCP connection will periodically send keep-alive packets to prevent the connection from being silently dropped by the operating system or network infrastructure during long periods of inactivity.

This property is only relevant to the Netty messenger.

​

Security Messenger Plugin Key

• setter:
  • ICanBuildDataClusterDescriptionBuilder#withProperty( MessengerUtil.SECURITY_MESSENGER_KEY, String value)
• default value: DISTRIBUTED_SECURITY_MANAGER

The plugin key of the security manager for authenticating remote authenticated messages sent to a data node.

This property only works for data nodes.

​

Message Sizes (Netty Messenger Only)

Message types are listed in NettyMessageType.

• setters:
  • -Dactiveviam.distribution.nettyMessageMaxSize=<key>=<size>m,<key>=<size>m... or,
  • Using setMaxSizeForMessage(String key, String valueInByte) in the DistributedMessenger MBean operations
• default value:
  • 64MB for INITIAL_DISCOVERY_MESSAGE
  • 4MB for TRANSACTION_COMMITTED_MESSAGE
  • 4MB for SCOPEDAGGREGATES_RETRIEVAL_TRANSPORTER
  • 32MB for DRILLTHROUGH_QUERY_RESULT
  • 2MB for any other message types

Note that the maximum message size should always be higher than the size of the allocated buffer when writing the stream. By default, this buffer’s size is set to be slightly bigger than 1MB. An exception is thrown if a message size is lower than this buffer’s size, and the messages will never be sent.

When exceeding a given message type limit, an IOException is thrown, with an error message specifying the message type limit, for example, “Cannot send a message larger than 1MiB (1048576)”. The table below shows that depending on the type of the message exceeding the maximum limit, this error could be critical or less.