Skip to main content

​
What is the Continuous GAQ REST API?

The Continuous GAQ (GetAggregatesQuery) REST API is an experimental API that allows applications to run queries against the cube and receive continuous streaming updates as data changes. Results are delivered in Apache Arrow format.

​
Experimental API

This API is experimental and may change in future versions without notice.

​
Why use the Continuous GAQ REST API?

The Continuous GAQ API provides:
  • Efficient data format: Results are delivered in Apache Arrow format for optimal performance
  • Direct query execution: Execute GetAggregatesQuery operations programmatically
  • Real-time updates: Receive data changes as they occur without polling
  • Incremental updates: Only affected cells are transmitted, not the entire result set
  • Persistent connections: Maintain a single connection for multiple queries

​
Prerequisites

Before using this API:
  • Authentication: All endpoints require authenticated users
  • GAQ familiarity: Understanding of GetAggregatesQuery concepts (measures, levels, coordinates)
  • Apache Arrow: Client-side Arrow deserialization capability
  • Server JVM option: The Atoti Server JVM must be started with --add-opens java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED

​
Key features

  • Apache Arrow streaming: Results are streamed in application/vnd.apache.arrow.stream format
  • Publisher-subscriber pattern: Publishers group queries with shared lifecycle management
  • User-based authentication: All operations require authenticated users
  • Query lifecycle management: Create publisher, subscribe, unsubscribe, and stop publishers

​
Usage workflow

A typical workflow for using the Continuous GAQ REST API:
  1. Create a publisher using /publisher/create to obtain a publisher ID
  2. Subscribe to queries using /subscribe with the publisher ID and query request
  3. Receive streaming updates in Apache Arrow format as data changes
  4. Unsubscribe from specific queries when no longer needed using /unsubscribe
  5. Stop the publisher when all queries are complete using /publisher/stop

​
API endpoints

All endpoints are available under the base path: 
/activeviam/pivot/rest/v10/cube/{cubeName}/queries/continuous-gaq
For detailed endpoint specifications, request/response schemas, and examples, see the Continuous GAQ Query REST API in the OpenAPI documentation.

​
Create publisher

Creates a new publisher for continuous GAQ streams. Clients must use the returned publisher ID to subscribe to GAQ streams. Endpoint: POST /publisher/create Response: JSON containing the publisher ID

​
Subscribe to GAQ

Subscribes to a GAQ query and returns streaming results in Apache Arrow format. The publisher ID identifies the publisher object where the streamed query results are written. Multiple queries can share one publisher. This groups related queries under the same lifecycle. Endpoint: POST /subscribe Response: Streaming HTTP response body in Apache Arrow format (application/vnd.apache.arrow.stream) Batch Size Configuration: The number of rows per batch when streaming GAQ results can be configured using the JVM property activeviam.gaq.arrow.batchsize.

​
Unsubscribe from GAQ

Unsubscribes from a specific query on a publisher. Endpoint: DELETE /unsubscribe

​
Stop publisher

Stops a publisher and unsubscribes all queries associated with it. Endpoint: DELETE /publisher/stop

​
Authentication

All endpoints require authentication. The API uses the current authenticated user’s credentials to execute queries and manage subscriptions. If the current thread is not authenticated, a ForbiddenAccessException is thrown.

​
Streaming result format

Results are streamed using the Apache Arrow IPC (Inter-Process Communication) format. The stream consists of multiple Arrow record batches, each representing either query results or failure notifications.

​
Apache Arrow streaming format

The API uses Apache Arrow’s streaming format (application/vnd.apache.arrow.stream). For more information about Apache Arrow, see the official documentation.

​
Message types

The stream carries two kinds of result messages, distinguished by the fullRefresh key in the schema metadata:
  • Full-refresh message (fullRefresh=true): an authoritative snapshot for the measures it carries. Replace the local view of those measures with its contents. A full-refresh message that carries zero rows is an explicit signal to clear every cell for the measures it declares.
  • Delta message (fullRefresh=false): incremental changes since the previous message. Apply them per cell using the cell state rule. A delta message that carries zero rows is a no-op (nothing changed).
Delta messages are available since version 6.1.22. Before that, every message is a full-refresh message.
A single epoch update may be delivered as one stream or split across two (a delta stream and a full-refresh stream). The tablesInUpdate metadata key indicates how many streams compose the epoch: when it is 2, wait for both streams to arrive and apply them together atomically (never apply one without the other); when it is 1, the stream is the whole update.

​
Arrow record batch structure

Each message is an Arrow IPC message. A message may be split into multiple Arrow RecordBatches when the result set is large; Arrow-compatible libraries reassemble them transparently.

​
Schema metadata (custom metadata)

Metadata embedded in the Arrow schema provides context about the batch:
FieldDescription
publisherIdPublisher identifier
queryIdUnique query identifier
branchIdCube branch name
epochIdQuery execution epoch
versionFormat version of this stream. Currently 1.0.
fullRefreshtrue = full-refresh message (replace the view for the measures it carries). false = delta message (apply per-cell changes).
tablesInUpdateNumber of Arrow IPC streams composing the update of the epoch this stream belongs to. A value of 2 means the epoch is split into a delta stream and a full-refresh stream that must be applied together, atomically. A value of 1 means the stream is the whole update. For forward compatibility, treat an absent value as 1.
error(Optional) Whether the update triggered an error, in which case the content of the message should be discarded
On a failure message fullRefresh is always omitted, error is always present and epochId is included when known (for example, a failure raised during query execution carries epochId, while a failure raised at init does not).

​
Data columns

Every message contains:
  • Level columns: level members at the row’s location.
  • Measure columns: aggregated values. Their shape depends on the message type.

​
Full-refresh message

Measure columns are flat: one column per measure, with the aggregated value.
Currency (UTF-8)Amount (Float64)
“USD”44.0
”EUR”52.0
”GBP”18.0

​
Delta message

Each measure is a struct column with two children:
  • value: same type as the measure, nullable.
  • is_removed: boolean, non-nullable.
Example with one measure Amount:
Currency
(UTF-8)		Amount (struct)Amount struct set?
value
(Float64)		is_removed
(Bool)
“USD”47.0falseyes
”EUR”nulltrueyes
”GBP”n/an/ano

​
Cell state

Decode each (row, measure) cell of a delta message with the following rule:
Struct set? (validity bit)is_removedMeaning for the client view
non/aUNCHANGED: keep the current value
yesfalseUPDATED: replace with value
yestrueREMOVED: drop the cell
“Struct set?” is the Arrow struct validity bit; Arrow-compatible libraries expose it as a null / non-null check on the struct cell.

​
Reading results

To read the continuous Arrow stream:
  1. Connect to the streaming endpoint - Establish HTTP connection to /subscribe.
  2. Read Arrow IPC messages sequentially - Each message carries one update.
  3. Extract schema metadata - Read custom metadata from the schema to get:
    • publisherId: Publisher identifier.
    • queryId: Query identifier for this specific query.
    • branchId: Cube branch name.
    • epochId: Epoch number (increments with each update).
    • version: Format version of this stream (currently 1.0).
    • fullRefresh: true for a full-refresh message, false for a delta message.
    • tablesInUpdate: Number of streams composing this epoch update (absent means 1).
    • error: Failure event information (if present).
  4. Check for failure events - If the error key is present, handle the failure and skip the remaining steps. fullRefresh is absent and the record batch is empty. epochId may or may not be set depending on when the failure occurred.
  5. Apply the update:
    • If fullRefresh is true, replace the local view of the measures the message carries with its rows.
    • If fullRefresh is false, iterate the rows and apply the cell state rule per measure.
    • If tablesInUpdate is 2, the epoch is split into a delta stream and a full-refresh stream. Do not apply either stream on its own: wait for both to arrive, then apply them together atomically. Applying the first and discovering the second is still pending would leave the view in an inconsistent state.
  6. Continue reading - Process subsequent updates as they arrive.
  7. Handle completion - Connection closes when the publisher is stopped.

​
Chunking for large results

When a single update contains many cells, it is split into multiple chunks based on the configured batch size (activeviam.gaq.arrow.batchsize JVM property). This uses Arrow’s Record Batch and should be seamless to the end user.

​
Failure event format

When a query execution fails, a failure event is sent instead of query results. The associated record batch will be empty. The failure event metadata contains an error key whose value is a string with the following fields:
  • streamId: publisher identifier on which the failure occurred
  • queryId: query identifier that failed
  • id: event identifier
  • type: simple name of the underlying error class (for example, QueryExecutionException)
  • message: error message
  • cause: nested cause event in the same format, or null if absent
Example failure detection: 
Schema Custom Metadata:
  error: "FailureEvent[streamId=pub-123, queryId=query-456, id=1, type=QueryExecutionException, message=Query timeout exceeded, cause=null]"

​
Connection limitations

When using HTTP/1.1, browsers and HTTP clients typically limit the number of simultaneous open connections to the same host. The standard limit is 6 concurrent connections per domain. Since each active publisher maintains a persistent streaming connection, this means:
  • Maximum 6 publishers can run simultaneously in a browser environment
  • Once the limit is reached, additional subscription requests will be queued or blocked
  • Stopping a publisher releases its connection, allowing new publishers to be created
Recommendations:
  • Reuse publishers: Share publishers across related queries instead of creating multiple publishers
  • Stop unused publishers: Call /publisher/stop when queries are no longer needed
  • Use HTTP/2: HTTP/2 supports multiplexing, allowing many concurrent streams over a single connection