> ## Documentation Index
> Fetch the complete documentation index at: https://docs.activeviam.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Continuous GAQ REST API
## What is the Continuous GAQ REST API?
The Continuous GAQ (GetAggregatesQuery) REST API is an experimental API that allows applications to run queries against the cube and receive continuous streaming updates as data changes. Results are delivered in Apache Arrow format.
### Experimental API
This API is experimental and may change in future versions without notice.
## Why use the Continuous GAQ REST API?
The Continuous GAQ API provides:
* **Efficient data format**: Results are delivered in Apache Arrow format for optimal performance
* **Direct query execution**: Execute GetAggregatesQuery operations programmatically
* **Real-time updates**: Receive data changes as they occur without polling
* **Incremental updates**: Only affected cells are transmitted, not the entire result set
* **Persistent connections**: Maintain a single connection for multiple queries
## Prerequisites
Before using this API:
* **Authentication**: All endpoints require authenticated users
* **GAQ familiarity**: Understanding of GetAggregatesQuery concepts (measures, levels, coordinates)
* **Apache Arrow**: Client-side Arrow deserialization capability
* **Server JVM option**: The Atoti Server JVM must be started with `--add-opens java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED`
## Key features
* **Apache Arrow streaming**: Results are streamed in `application/vnd.apache.arrow.stream` format
* **Publisher-subscriber pattern**: Publishers group queries with shared lifecycle management
* **User-based authentication**: All operations require authenticated users
* **Query lifecycle management**: Create publisher, subscribe, unsubscribe, and stop publishers
## Usage workflow
A typical workflow for using the Continuous GAQ REST API:
1. **Create a publisher** using `/publisher/create` to obtain a publisher ID
2. **Subscribe to queries** using `/subscribe` with the publisher ID and query request
3. **Receive streaming updates** in Apache Arrow format as data changes
4. **Unsubscribe** from specific queries when no longer needed using `/unsubscribe`
5. **Stop the publisher** when all queries are complete using `/publisher/stop`
```mermaid theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
sequenceDiagram
participant Client
participant Server
Client->>Server: POST /publisher/create
Server-->>Client: publisherId
Client->>Server: POST /subscribe (publisherId + query)
Server-->>Client: Full-refresh message (fullRefresh=true)
loop Data changes
Server-->>Client: Delta message (fullRefresh=false) and/or full-refresh message
end
Client->>Server: DELETE /unsubscribe (queryId)
Server-->>Client: OK
Client->>Server: DELETE /publisher/stop (publisherId)
Server-->>Client: OK
```
## API endpoints
All endpoints are available under the base path:
```
/activeviam/pivot/rest/v10/cube/{cubeName}/queries/continuous-gaq
```
For detailed endpoint specifications, request/response schemas, and examples, see the [Continuous GAQ Query REST API](https://docs.activeviam.com/engine/rest/6.2.0-beta/continuous-gaq-query-rest-api-experimental/) in the OpenAPI documentation.
### Create publisher
Creates a new publisher for continuous GAQ streams. Clients must use the returned publisher ID to subscribe to GAQ streams.
**Endpoint**: `POST /publisher/create`
**Response**: JSON containing the publisher ID
### Subscribe to GAQ
Subscribes to a GAQ query and returns streaming results in Apache Arrow format.
The publisher ID identifies the publisher object where the streamed query results are written. Multiple queries can share one publisher. This groups related queries under the same lifecycle.
**Endpoint**: `POST /subscribe`
**Response**: Streaming HTTP response body in Apache Arrow format (`application/vnd.apache.arrow.stream`)
**Batch Size Configuration**: The number of rows per batch when streaming GAQ results can be configured using the JVM property `activeviam.gaq.arrow.batchsize`.
### Unsubscribe from GAQ
Unsubscribes from a specific query on a publisher.
**Endpoint**: `DELETE /unsubscribe`
### Stop publisher
Stops a publisher and unsubscribes all queries associated with it.
**Endpoint**: `DELETE /publisher/stop`
## Authentication
All endpoints require authentication. The API uses the current authenticated user's credentials to execute queries and manage subscriptions. If the current thread is not authenticated, a `ForbiddenAccessException` is thrown.
## Streaming result format
Results are streamed using the Apache Arrow IPC (Inter-Process Communication) format. The stream consists of multiple Arrow record batches, each representing either query results or failure notifications.
### Apache Arrow streaming format
The API uses Apache Arrow's streaming format (`application/vnd.apache.arrow.stream`).
For more information about Apache Arrow, see the [official documentation](https://arrow.apache.org/).
### Message types
The stream carries two kinds of result messages, distinguished by the `fullRefresh` key in the schema metadata:
* **Full-refresh message** (`fullRefresh=true`): an authoritative snapshot for the measures it carries. Replace the local view of those measures with its contents. A full-refresh message that carries zero rows is an explicit signal to clear every cell for the measures it declares.
* **Delta message** (`fullRefresh=false`): incremental changes since the previous message. Apply them per cell using the [cell state](#cell-state) rule. A delta message that carries zero rows is a no-op (nothing changed).
Delta messages are available since version 6.1.22. Before that, every message is a full-refresh message.
A single epoch update may be delivered as one stream or split across two (a delta stream and a full-refresh stream). The `tablesInUpdate` metadata key indicates how many streams compose the epoch: when it is `2`, wait for both streams to arrive and apply them together atomically (never apply one without the other); when it is `1`, the stream is the whole update.
### Arrow record batch structure
Each message is an Arrow IPC message. A message may be split into multiple Arrow RecordBatches when the result set is large; Arrow-compatible libraries reassemble them transparently.
#### Schema metadata (custom metadata)
Metadata embedded in the Arrow schema provides context about the batch:
| Field | Description |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `publisherId` | Publisher identifier |
| `queryId` | Unique query identifier |
| `branchId` | Cube branch name |
| `epochId` | Query execution epoch |
| `version` | Format version of this stream. Currently `1.0`. |
| `fullRefresh` | `true` = full-refresh message (replace the view for the measures it carries). `false` = delta message (apply per-cell changes). |
| `tablesInUpdate` | Number of Arrow IPC streams composing the update of the epoch this stream belongs to. A value of `2` means the epoch is split into a delta stream and a full-refresh stream that must be applied together, atomically. A value of `1` means the stream is the whole update. For forward compatibility, treat an absent value as `1`. |
| `error` | (Optional) Whether the update triggered an error, in which case the content of the message should be discarded |
On a failure message `fullRefresh` is always omitted, `error` is always present and `epochId` is included when known (for example, a failure raised during query execution carries `epochId`, while a failure raised at init does not).
#### Data columns
Every message contains:
* **Level columns**: level members at the row's location.
* **Measure columns**: aggregated values. Their shape depends on the message type.
#### Full-refresh message
Measure columns are flat: one column per measure, with the aggregated value.
| Currency (UTF-8) | Amount (Float64) |
| ---------------- | ---------------- |
| "USD" | 44.0 |
| "EUR" | 52.0 |
| "GBP" | 18.0 |
#### Delta message
Each measure is a **struct** column with two children:
* `value`: same type as the measure, nullable.
* `is_removed`: boolean, non-nullable.
Example with one measure `Amount`:
Currency
(UTF-8) |
Amount (struct) |
Amount struct set? |
value
(Float64) |
is\_removed
(Bool) |
| "USD" | 47.0 | false | yes |
| "EUR" | null | true | yes |
| "GBP" | n/a | n/a | no |
#### Cell state
Decode each (row, measure) cell of a delta message with the following rule:
| Struct set? (validity bit) | `is_removed` | Meaning for the client view |
| -------------------------- | ------------ | --------------------------------- |
| no | n/a | UNCHANGED: keep the current value |
| yes | false | UPDATED: replace with `value` |
| yes | true | REMOVED: drop the cell |
"Struct set?" is the Arrow struct validity bit; Arrow-compatible libraries expose it as a null / non-null check on the struct cell.
### Reading results
To read the continuous Arrow stream:
1. **Connect to the streaming endpoint** - Establish HTTP connection to `/subscribe`.
2. **Read Arrow IPC messages sequentially** - Each message carries one update.
3. **Extract schema metadata** - Read custom metadata from the schema to get:
* `publisherId`: Publisher identifier.
* `queryId`: Query identifier for this specific query.
* `branchId`: Cube branch name.
* `epochId`: Epoch number (increments with each update).
* `version`: Format version of this stream (currently `1.0`).
* `fullRefresh`: `true` for a full-refresh message, `false` for a delta message.
* `tablesInUpdate`: Number of streams composing this epoch update (absent means `1`).
* `error`: Failure event information (if present).
4. **Check for failure events** - If the `error` key is present, handle the failure and skip the remaining steps. `fullRefresh` is absent and the record batch is empty. `epochId` may or may not be set depending on when the failure occurred.
5. **Apply the update**:
* If `fullRefresh` is `true`, replace the local view of the measures the message carries with its rows.
* If `fullRefresh` is `false`, iterate the rows and apply the [cell state](#cell-state) rule per measure.
* If `tablesInUpdate` is `2`, the epoch is split into a delta stream and a full-refresh stream. Do not apply either stream on its own: wait for both to arrive, then apply them together atomically. Applying the first and discovering the second is still pending would leave the view in an inconsistent state.
6. **Continue reading** - Process subsequent updates as they arrive.
7. **Handle completion** - Connection closes when the publisher is stopped.
### Chunking for large results
When a single update contains many cells, it is split into multiple chunks based on the configured batch size (`activeviam.gaq.arrow.batchsize` JVM property).
This uses Arrow's Record Batch and should be seamless to the end user.
### Failure event format
When a query execution fails, a failure event is sent instead of query results. The associated record batch will be empty.
The failure event metadata contains an `error` key whose value is a string with the following fields:
* `streamId`: publisher identifier on which the failure occurred
* `queryId`: query identifier that failed
* `id`: event identifier
* `type`: simple name of the underlying error class (for example, `QueryExecutionException`)
* `message`: error message
* `cause`: nested cause event in the same format, or `null` if absent
**Example failure detection**:
```
Schema Custom Metadata:
error: "FailureEvent[streamId=pub-123, queryId=query-456, id=1, type=QueryExecutionException, message=Query timeout exceeded, cause=null]"
```
## Connection limitations
When using HTTP/1.1, browsers and HTTP clients typically limit the number of simultaneous open connections to the same host. The standard limit is **6 concurrent connections** per domain.
Since each active publisher maintains a persistent streaming connection, this means:
* **Maximum 6 publishers** can run simultaneously in a browser environment
* Once the limit is reached, additional subscription requests will be queued or blocked
* Stopping a publisher releases its connection, allowing new publishers to be created
**Recommendations**:
* **Reuse publishers**: Share publishers across related queries instead of creating multiple publishers
* **Stop unused publishers**: Call `/publisher/stop` when queries are no longer needed
* **Use HTTP/2**: HTTP/2 supports multiplexing, allowing many concurrent streams over a single connection
## Related documentation
* [Cube REST API](atoti_rest_api)
* [REST APIs Overview](rest-apis-overview)
* [Query Execution Monitoring](../monitoring/monitoring-query-execution)
* [GAQ REST API](gaq_rest_api)