Distribution Monitoring

Tracing

The distribution layer leverages tracing to provides insights into the queries executed in the cluster. Check tracing to configure it.

Logging

Without tracing activated, the distribution layer provides some logging.

Logger naming

To retrieve all the distribution logs, two loggers should be activated:

atoti.server.distribution (new namespace)
com.activeviam.activepivot.dist (legacy namespace)

The issue with the legacy logger is that its name hierarchy (e.g. com.activeviam.activepivot.dist.impl.private_) contains internal packages and classes, thus making it hard to filter using a more granular logger, as the full logger name hierarchy is not public, and can be changed between two bugfix versions. The new logger (atoti.server.distribution) will provide some more granular loggers, to focus on one particular aspect of the distribution such as “atoti.server.distribution.messaging”, … The exhaustive list will be made available in the future, but will be fully contained under atoti.server.distribution. As loggers will be moved gradually from com.activeviam.activepivot.dist to atoti.server.distribution, it is recommended to activate both loggers in your project, to avoid missing any logs when upgrading to a newer bugfix version.

What is the distribution state of a query cube?

The distribution state is the view a query cube holds of its cluster. It has three levels: the query cube, its remote data cubes, and the branches of those data cubes. The state is built only from what the query cube has received, starting with the messages published by its data cubes and the JGroups cluster membership notifications. Reading it tells you how far each data cube and each branch has got: which ones the query cube has fully accepted, which are still waiting for a message, and which have failed. This section assumes familiarity with the distributed architecture and with the communication flow between data cubes and query cubes.

The distribution state reports exactly one thing: what the query cube has received.

It is not what the data cubes have sent. A message a data cube has already published, but that has not reached the query cube yet, leaves no trace in the state.
It is not what the query cube has applied onto its own cube. A transaction message that has arrived already advances the state of the branch it targets, while the transaction itself may not have been committed onto the query cube yet, so a query does not return its data.

A data cube reported as LIVE therefore means that the query cube received its discovery, not that every message it sent has arrived, and not that the work has been applied. The endpoints listed in Where does an event stand in the processing pipeline report where that input stands in the processing pipeline.

How to request the distribution state

Every distribution observability endpoint is served under the same prefix:

/activeviam/distribution-observability/rest/v1

They are all GET endpoints. Each one takes the name of a query cube as a path variable and returns JSON. Each one answers 404 when no query cube goes by that name, and when distribution is not started on that cube. They are documented request by request in the REST service. The state of a query cube is read from the state endpoint:

GET /activeviam/distribution-observability/rest/v1/{queryCubeName}/state

The response nests the three levels:

The top object carries queryCube, the state of the query cube itself, and dataCubes, a map keyed by data cube address.
Each entry of dataCubes carries dataCube, the state of that data cube, and branches, a map keyed by branch name.
Each entry of branches carries the state of that branch.

The response below describes a converged cluster. It holds one data cube, known under the address data-cube-A, with a single master branch:

{
  "queryCube": {
    "state": "READY",
    "discoveredApplications": null,
    "undiscoveredApplications": null,
    "retainedActions": null
  },
  "dataCubes": {
    "data-cube-A": {
      "dataCube": {
        "state": "LIVE",
        "lastHelloViewId": 7,
        "lastHelloSequenceNumber": 0,
        "membershipRegistered": true,
        "pendingDiscoveryAgeSeconds": null,
        "reservationExpired": null,
        "failure": null
      },
      "branches": {
        "master": {
          "state": "LIVE",
          "lastProcessableSequenceNumber": 1,
          "lastProcessableEpoch": 2,
          "failure": null
        }
      }
    }
  }
}

What are the query cube states?

The query cube reports one of two states.

State	Meaning
`AWAITING_DISCOVERY`	The schema of at least one application has not been received, so the actions that contribute to the cube are held back. This state is only reachable when the `IDistributedActivePivotInstanceDescription.WAIT_FOR_ALL_APPLICATIONS` property is `true`.
`READY`	The cube applies the actions it receives as they are produced, holding nothing back.

The queryCube object carries these fields.

Field	Meaning
`state`	The state of the query cube.
`discoveredApplications`	The applications that have published their schema.
`undiscoveredApplications`	The applications whose schema are still awaited.
`retainedActions`	The actions held back until the last schema arrives.

Every field but state is null except in the AWAITING_DISCOVERY state. See Wait for All Applications to configure the property that makes that state reachable.

What are the remote data cube states?

Each data cube the query cube knows reports one of six states.

State	Meaning
`AWAITING_HELLO`	The data cube is in the cluster view, but its Hello has not been received yet.
`AWAITING_DISCOVERY`	The Hello has been received, the discovery has not. No transaction can be processed.
`LIVE`	The discovery has been submitted. Transactions can be processed.
`PENDING_REMOVAL`	The cluster membership has been removed from the query cube, and the contributions are kept. A data cube rejoining shortly is therefore not forced into a rebuild.
`REMOVED`	The cluster membership and the contributions have both been removed from the query cube.
`BROKEN`	An action execution for this data node failed. Messages for it are no longer applied. A removal is still honored, so a partially broken cluster can be cleaned.

Each dataCube object carries these fields.

Field	Meaning
`state`	The state of the data cube.
`lastHelloViewId`	The cluster view of the last Hello received.
`lastHelloSequenceNumber`	The sequence number of the last Hello received.
`membershipRegistered`	Whether the query cube still holds the cluster membership of the data cube.
`pendingDiscoveryAgeSeconds`	How long ago the pending discovery slot was reserved. It is `null` except in the `AWAITING_DISCOVERY` state.
`reservationExpired`	Whether that reservation stopped counting towards the limit. It is `null` except in the `AWAITING_DISCOVERY` state.
`failure`	The failure that broke the data cube, rendered as a single line of text. It is `null` except in the `BROKEN` state.

A reservation expires after the delay set by Pending discovery timeout, and the number of reservations held at once is capped by Pending initial discoveries.

What are the remote branch states?

Each branch of a data cube reports one of four states.

State	Meaning
`PENDING_CREATION`	The branch-creation commit has not been applied yet. Either it has not been received, or a message preceding it is still missing.
`LIVE`	The creation commit has been submitted. Transactions can be processed.
`DELETED`	The branch has been deleted.
`BROKEN`	An action execution for this branch failed. Messages for it are no longer applied. A removal is still honored, so a partially broken cluster can be cleaned.

Each branch object carries these fields.

Field	Meaning
`state`	The state of the branch.
`lastProcessableSequenceNumber`	The last sequence number declared processable.
`lastProcessableEpoch`	The epoch of that message. It is `null` in the `PENDING_CREATION` and `DELETED` states, where the branch holds nothing.
`failure`	The failure that broke the branch, rendered as a single line of text. It is `null` except in the `BROKEN` state.

Where does an event stand in the processing pipeline?

The state endpoint stops at what the query cube received. Four more endpoints report where that input stands in the processing pipeline.

Endpoint	Content
`/{queryCubeName}/queuedEvents`	The events received but not yet processed.
`/{queryCubeName}/queuedActions`	The actions produced and waiting to be applied.
`/{queryCubeName}/inFlightAction`	The action currently being applied. The response is empty while the executor is idle.
`/{queryCubeName}/scheduledEvents`	The delayed events waiting to fire, such as restart orders and scheduled removals.

Which messages is a query cube holding back?

A data cube publishes its messages in a sequence. The query cube buffers a message whose predecessor in that sequence has not arrived. It keeps that message until the gap is filled. A growing buffer means the cluster is waiting on a message that never came.

GET /activeviam/distribution-observability/rest/v1/{queryCubeName}/queuedMessages

The response groups the buffered messages by data cube address, then by branch, in ascending sequence order. A data cube buffering nothing is left out. An empty response therefore means every data cube is up to date.

How to request the same information over JMX

Every endpoint above has a JMX counterpart, each carried by its own monitored child of the query cube MBean.

Monitored child	Operation	REST equivalent
`DistributionState`	`printDistributionState`	`state`
`EventInbox`	`listQueuedEvents`	`queuedEvents`
`DistributedActionQueue`	`listQueuedActions`	`queuedActions`
`DistributedActionExecutor`	`getInFlightAction`	`inFlightAction`
`DistributedEventScheduler`	`listScheduledEvents`	`scheduledEvents`
`DistributionState` child, one per data cube address	`listQueuedMessages`	`queuedMessages`

See JMX monitoring to connect an MBean client to the server.

listQueuedMessages is reported per data cube, as one child of DistributionState per data cube address, while the queuedMessages endpoint returns every data cube in a single response.

Releases and upgrades

Getting started

Concepts

Step 1: Bring your data into Atoti

Step 2: Design your Cube

Step 3: Build an Atoti application

Step 4: Query an Atoti application

Configure Atoti

Secure Atoti

Operate Atoti

Reference

Distribution Monitoring

Tracing

Logging

Logger naming

What is the distribution state of a query cube?

How to request the distribution state

What are the query cube states?

What are the remote data cube states?

What are the remote branch states?

Where does an event stand in the processing pipeline?

Which messages is a query cube holding back?

How to request the same information over JMX

What to read next

​Tracing

​Logging

​Logger naming

​What is the distribution state of a query cube?

​How to request the distribution state

​What are the query cube states?

​What are the remote data cube states?

​What are the remote branch states?

​Where does an event stand in the processing pipeline?

​Which messages is a query cube holding back?

​How to request the same information over JMX

​What to read next

Tracing

Logging

Logger naming

What is the distribution state of a query cube?

How to request the distribution state

What are the query cube states?

What are the remote data cube states?

What are the remote branch states?

Where does an event stand in the processing pipeline?

Which messages is a query cube holding back?

How to request the same information over JMX

What to read next