Versioning
Versioning on a DirectQuery connector is less powerful than on a Datastore.
Old versions access
All external databases do not support native versioning (i.e. time travel).
For these databases you can only query the last snapshot of the data.
Non-time travel databases are subject to de-synchronization issue (see heading below).
| Database | Time travel Support |
|---|---|
| Amazon Redshift |  |
| ClickHouse | |
| Databricks |  (not on views) |
| Dremio | |
| Google BigQuery | |
| Microsoft Azure Synapse | |
| Microsoft SQL Server | |
| Snowflake | (not on views except dynamic table) |
Data refresh
Unlike the datastore, external databases are not able to push data change event.
The user must warn the DirectQuery connector that a new version of the data is available.
There is two ways to do this :
- If you know the scope of your changes you can use an Incremental Refresh.
- If you do not know the scope of the change, you can trigger a full refresh.
app.fullRefresh();
A full refresh is a costly operation because it implies to rebuild all hierarchies and aggregate providers.
During these operations, the cube is still queryable.
De-synchronization issue with non-time travel databases
Atoti cube leverages database streams to maintain some versioned data structures:
- Hierarchy member tree
- Aggregate providers
Aggregate providers are used to store already computed primitive measures.
On a non-time travel database, on data change and before the refresh of the application, aggregate providers are
not up-to-date.
So the measures computed from the primitive measures of aggregate provider are outdated too.
However, measures which are not computed from aggregate provider measures (i.e. computed by a database query) are
up-to-date.
This could lead to inconsistent results if you are mixing up outdated and up-to-date measures.
For instance, let's take an initial version V1 and an update in the database changing the data to a new version
V2.
Before triggering the refresh in Atoti application, the aggregate provider is still in state V1 but the database
connector has no more access to the V1 data.
The connector will return the data from the current state of the database (which is V2).
Branching
DirectQuery databases does not support branching.