Distribution Properties
This page defines and explains the properties specific to the distributed architecture, and those you can set to modify the default behaviors of both the data and query nodes, and of their associated messengers.
To familiarize yourself with the distributed architecture, we recommend you read the rest of the distributed architecture documentation before continuing.
Query Cube Properties
The query cube properties can be found in the IDistributdActivePivotInstanceDescription
interface, and can
be set using the fluent builder API IQueryCubeDescriptionBuilder
.
Data Replication Property
- setter(s):
IQueryCubeDescriptionBuilder#withDataReplication(String value)
IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.DATA_REPLICATION, String value)
- default value:
REPLICATE_ON_SLICER
The property is used to set the aggregates data replication policy. The query cube can build queries that do not exist locally. As measures from different cubes are independent, ActivePivot allows the measures to appear on these unknown locations.
The following example explains the different data replication policies:
The distributed cluster is composed of two data cubes and a query cube.
- Data Cube 1 has one hierarchy.
- Data Cube 2 has two hierarchies (distinct from Data Cube 1), Time, that is
slicing
. The Query Cube represents the merger of the two topologies, and therefore contains three hierarchies.
A query hits the query node with the location AllMember|*|AllMember
.
Asking for AllMembers on the
year
level aggregates the measures over all years, whereas asking for*
aggregates the measures for each year, and returns a list of results.
In the data cube 1, the new local location is AllMember
. When translating back to the query cube,
there are three possibilities:
REPLICATE
: The measures are replicated on all unknown hierarchies and levels. Here, the results are replicated for locations:* `AllMember|2019|AllMember`
* `AllMember|2018|AllMember`
* `AllMember|2017|AllMember`NO_REPLICATION
: The measures are not replicated on any unknown location. The result are:AllMember|2019|AllMember
in this case, because the hierarchy Time isslicing
and 2019, as first member of the hierarchy, it is the default value (unless otherwise specified).AllMember|AllMember|AllMember
if the Time hierarchy was notslicing
, as the results would be aggregated on the defaultAllMember
.
REPLICATE_ON_SLICER
: The measures are replicated on the first level of unknown hierarchies, and therefore toAllMember
for regular hierarchies, or to the first-level member for aslicing
hierarchy.
Application Measure Groups
- setter(s):
IQueryCubeDescriptionBuilder#withApplicationMeasureGroups(boolean value)
IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.APPLICATION_MEASURE_GROUPS, boolean value)
- default value:
true
If set to true, distributed hierarchies and measures have the applicationId
of the data pivot
they come from as measureGroup.
Application Measure Names
- setter(s):
IQueryCubeDescriptionBuilder#withApplicationMeasureNames(boolean value)
IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.APPLICATION_MEASURE_NAMES, boolean value)
- default value:
false
If set to true, the query cubes are allowed to override locally defined measure names. The data
pivot's applicationId
is added as a suffix.
Application Measure Groups Suffix
- setter(s):
IQueryCubeDescriptionBuilder#withApplicationMeasureGroupsSuffix(boolean value)
IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.APPLICATION_MEASURE_GROUPS_SUFFIX, boolean value)
- default value:
false
If set to true, the query cubes are allowed to append the data pivot's applicationId
to
locally defined measure groups.
Removal Delay
- setter(s):
IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.REMOVAL_DELAY, String value)
- default value:
60
- unit: second
Timer in seconds after which a complete removal of a given instance's contributions to the query cube topology is triggered. A negative value means no removal, while a null (zero) value means immediate removal.
For more details on this property, see Communication flow and Recommendations.
Pause on Merge Failure
- setter(s):
IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.PAUSE_ON_MERGE_FAILURE, boolean value)
- default value:
false
If set to true, the query cube pauses in case of a merge failure. A merge failure signifies that the query cube could not merge a data cube's dimensions into its own global topology. This happens when a data cube violates a distributed constraint:
- The uniqueness constraint on the members of the distributed fields within the same application.
- The uniqueness constraint on the measures across different applications. Pausing the data cube stops its messenger, meaning the cube cannot communicate with the rest of the cluster.
Wait for All Applications
- setter(s):
IQueryCubeDescriptionBuilder#withProperty( IDistributedActivePivotInstanceDescription.WAIT_FOR_ALL_APPLICATIONS, boolean value)
- default value:
true
If set to true, the query cube waits until all the data cubes of different applications have
joined the cluster before processing their InitialDiscoveryMessages
. Since a data cube from a new
application entering the cluster can trigger a complete rebuild of the query cube's topology, it can prove more
efficient to wait for all JGroups cluster members to have sent their messages before building the
query cube's topology.
Transaction Notification
- setter(s):
ICanBuildDataClusterDescriptionBuilder#withProperty( IDataClusterDefinition.AWAIT_NOTIFICATIONS_TIME, value)
after setting the concealed measures, wherevalue
is a string representation of along
value (in seconds) after setting theapplicationId
IMultiVersionActivePivot.setTransactionProperties(Property property)
whereproperty
includes the(key,value)
entry(AWAIT_NOTIFICATIONS_TIME, value)
- default value:
0
When set to a value greater than 0
, the data cube will wait for notifications from other remote instances upon a transaction.
Each Query cube will synchronously send a NotificationMessage
to the data cube notifying that the transaction has correctly been applied.
The data cube will resume after receiving notifications from every remote instance, or after the duration specified by the property has been exceeded.
Pending Initial Discoveries
- setter(s):
- set
qfs.distribution.maxPendingDiscoveries
as an ActiveViamProperty
- set
- default value:
4
The maximum number of pending initial discoveries this instance tolerates before refusing new
ones. This property must be set with great care, as the discovery messages are the biggest that
transit on the network. The query cube is capable of throttling the discoveries to avoid
over-stress. The query node must be able to hold within its heap memory the equivalent of
maxPendingDiscoveries * DiscoveryMessageSize
in the worst scenario.
Data Cube Properties
The following properties can be found in the IClusterDefinition
interface, and can be set using
the fluent builder API IDataClusterDescriptionBuilder
.
Execute Post Processors in Data Cubes
- setter(s):
ICanBuildDataClusterDescriptionBuilder#withProperty( IClusterDefinition.EXECUTE_POST_PROCESSORS_IN_DATA_CUBE_PROPERTY, boolean value)
, after setting theapplicationId
- default value:
true
If set to true, a data cube' post-processors are initialized and executed within the data cube.
It can be overridden on a case-by-case basis, using each post-processor's executeInDataCube
property.
You should set this value to false
, apart from exceptional cases. See the
Distributed Post-processors topic for more details about this
property.
Messenger Properties
The following properties can be found in the IMessengerDefinition
interface, and can be set using
the fluent builder API ICanBuildDataClusterDescriptionBuilder
.
Messenger Time Out
- setter(s):
ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.MESSENGER_TIMEOUT, String value)
- default value:
DEFAULT_MESSAGE_TIMEOUT = 30 000
- unit: millisecond
Timeout used by the messenger when waiting for a reply to any remote message.
Message Size Logging Threshold
- setter(s):
- set
qfs.distribution.log.size.threshold=<String>
as an ActiveViamProperty
- set
- default value:
DEFAULT_MESSAGE_TIMEOUT = 4M
- unit: To specify within the argument string. Default unit is
empty
, meaning a value of10
is 10 bytes.
System property to configure the threshold (in bytes) above which the size of a IBroadcastMessage
sent or
received is logged at level INFO.
Auto Start
- setter(s):
ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.AUTO_START, boolean value)
- default value:
true
If set to true, the distribution infrastructure automatically starts when the query cube starts, sending a message to all data nodes to start.
Hello Message Time Out
- setter(s):
ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.HELLO_MESSAGE_TIMEOUT, String value)
- default value:
2 000
- unit: millisecond
Specific timeout used by the messenger when waiting for a reply to a HelloMessage
before throwing
a timeout exception.
This property only works for data nodes.
Hello Message Number of Trials
- setter(s):
ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.HELLO_MESSAGE_NUMBER_TRIALS, String value)
- default value:
10
Number of discovery attempts before giving up. Total wait time is helloMessageNumberTrials *
helloMessageTimeout
.
This property only works for data nodes.
Retry Unavailable Instances Period
- setter(s):
ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.RETRY_UNAVAILABLE_INSTANCES_PERIOD, String value)
- default value:
10
- unit: millisecond
One node of a cluster sends the same IHelloMessage
to all members of the cluster. If no answer is
received from any of these cluster members within IMessengerDefinition.HELLO_MESSAGE_TIMEOUT
, the
address of that node is added to the list of ADistributedMessenger#unavailableInstances
.
The sender will retry broadcasting the IHelloMessage
IMessengerDefinition.HELLO_MESSAGE_NUMBER_TRIALS
times, each attempt separated from the previous
one by IMessengerDefinition.RETRY_UNAVAILABLE_INSTANCES_PERIOD
milliseconds.
Discovery Time Out
- setter(s):
ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.DISCOVERY_TIMEOUT, String value)
- default value: twice the value of
DEFAULT_MESSAGE_TIMEOUT = 30 000
- unit: millisecond
Specific timeout used by the messenger when waiting for a reply to aIInitialDiscoveryMessage
before
throwing a timeout exception. If the transmission fails, one more attempt is made to send the data
node's topology before the query node is considered unreachable. At this point, it is
advised to stop the node and increase this property's value.
This property only works for Data Nodes.
Discovery Serialization Time Out
- setter(s):
ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.SERIALIZATION_DISCOVERY_TIMEOUT, String value)
- default value:
-1
- unit: millisecond
Specific timeout used by the messenger's serializer, after which the serialization is aborted. Set to a negative value to permit serialization regardless of the time the process takes.
This property only works for Data Nodes.
Transaction Time Out
- setter(s):
ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.TRANSACTION_TIMEOUT, String value)
- default value:
DEFAULT_MESSAGE_TIMEOUT = 30 000
- unit: millisecond
Specific timeout used by the messenger when waiting for a reply to aITransactionCommittedMessage
before throwing a timeout exception. Note that on the next successful transaction, the query node
will detect the previous timeout/incorrectly transmitted transaction and will ask the data node to
restart.
This property only works for data nodes.
Query Serialization Time Out
- setter(s):
ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.SERIALIZATION_QUERY_TIMEOUT, String value)
- default value:
-1
- unit: millisecond
Specific timeout used by the messenger's serializer, after which the serialization is aborted. Set to a negative value to permit serialization regardless of the time the process takes.
This property only works for data nodes.
Interruption Check Interval
- setter(s):
ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.INTERRUPTION_CHECK_INTERVAL, String value)
- default value:
1 000
The number of write operations during a serialization operation between two interruption checks.
This property only works for data nodes.
Logical Address
- setter(s):
- set
logicalAddress=<String>
as an ActiveViamProperty ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.LOGICAL_ADDRESS, String value)
- set
- default value: Automatically generated
The logical address of the query or data cube in the cluster. The program property takes precedence over the messenger settings.
Port Range
- setter(s):
- set
qfs.distribution.netty.portRange=<String>
as an ActiveViamProperty ICanBuildDataClusterDescriptionBuilder#withProperty( IMessengerDefinition.PORT_RANGE, String value)
- set
- default value:
null
The format must follow this convention: 1000:1010,2000:2010
, to allow ports 1000 to 1010 and
2000 to 2010.
This is the range of valid ports to which the transporter will bind. It takes the first one
available within the range.
The program property takes precedence over the messenger settings.
A value of null
or 0
lets the system pick an ephemeral port in a bind operation.
This property is only relevant to the Netty messenger.
Security Messenger Plugin Key
- setter(s):
ICanBuildDataClusterDescriptionBuilder#withProperty( MessengerUtil.SECURITY_MESSENGER_KEY, String value)
- default value:
DISTRIBUTED_SECURITY_MANAGER
The plugin key of the security manager for authenticating remote authenticated messages sent to a data node.
This property only works for data nodes.
Message Sizes (Netty Messenger Only)
- setter(s):
-DnettyMessageMaxSize=<class||shortcut>=<size>m,<class||shortcut>=<size>m...
or,- Using
setMaxSizeForMessage(String classNameOrShortcut, String valueInByte)
in theDistributedMessenger
MBean operations
- default value:
- 64MB for
IInitialDiscoveryMessage
messages - 4MB for
ITransactionCommittedMessage
messages - 4MB for
ScopedAggregatesRetrievalResultTransporter
messages - 32MB for
DrillthroughMessageWithHeadersAnswers
messages - 2MB for any other message types
- 64MB for
Type | Class to Specify | Shortcut |
---|---|---|
IInitialDiscoveryMessage | com.qfs.messenger.message.IInitialDiscoveryMessage | initialDiscovery |
ITransactionCommittedMessage | com.qfs.messenger.message.ITransactionCommittedMessage | transactionCommitted |
ScopedAggregatesRetrievalResultTransporter | com.quartetfs.biz.pivot.cellset.impl.ScopedAggregatesRetrievalResultTransporter | singleGaqResult |
DrillthroughMessageWithHeadersAnswer | com.qfs.query.impl.DrillthroughMessageWithHeaders.DrillthroughMessageWithHeadersAnswer | drillthroughQueryResult |
DrillthroughHeader | com.quartetfs.biz.pivot.context.drillthrough.impl.DrillthroughHeader | drillthroughHeader |
Note that the maximum message size should always be higher than the size of the allocated buffer when writing the stream. By default, this buffer's size is set to
StreamObjectEncoder#DEFAULT_CHUNK_SIZE + NettyStreamUtils#HEADER_LEN = 1MB + Integer.BYTES = 1.004MB
. An exception is thrown if a message size is lower than this buffer's size, and the messages will never be sent.
When exceeding a given message type limit, an IOException
is thrown, with an error message
specifying the message type limit, for example, "Cannot send a message larger than 1MiB (1048576)".
The table below shows that depending on the type of the message exceeding the maximum limit, this error could be critical or less.
Type | Consequence |
---|---|
IInitialDiscoveryMessage | The discovery is never applied and the query cube will never be able to see the data cube hierarchies. The thrown exception can be seen in the logs. |
ITransactionCommittedMessage | The updates of the corresponding data cube committed transaction will not be visible by the query cube. The thrown exception can be seen in the logs. |
ScopedAggregatesRetrievalResultTransporter | The query will be interrupted and the exception message logged and displayed in ActiveUI |
DrillthroughMessageWithHeadersAnswer | The query will be interrupted and the exception message logged and displayed in ActiveUI |
DrillthroughHeader | The query will be interrupted and the exception message logged and displayed in ActiveUI |