This section describes some incremental refresh key mechanisms.Documentation Index
Fetch the complete documentation index at: https://docs.activeviam.com/llms.txt
Use this file to discover all available pages before exploring further.
Incremental refresh in details
The Atoti application is made of two main components:- Cube which answers business queries
- A database which stores the fact data. This database could be a Datastore or a DirectQuery connector.
- Hierarchies which contains the members existing in the cube hierarchy levels at a specific version
- Aggregate provider which stores precomputed primitive aggregates at a specific version.
These components are materialized views of the result of the query and are considered to be always up-to-date.
On a new version of the data, database is in charge to warn the cube and provide a description of the data change. To fulfill this need the cube will register streams on the database. A stream is a registered query along with a listener, the registration usually sends the initial data.
After the registration the listener is warned about new version and data changes. In a DirectQuery application the cube registers a stream for each hierarchy and a stream for each aggregate provider. When describing a change DirectQuery connector will search into the registered streams the impacted ones.
For each impacted streams, it will compute a list of operations to refresh these streams. Schema Fact hierarchy on
sales.dateFactless hierarchy on
products.colorAggregate provider on
sales.date, product.color -> sum(sales.quantity * products.price)
Partial or Wrong change description
Partial or wrong change description are not supported.In this example, we describe what could happen if an incorrect hint is provided. Changes One row with a sale on product
P1 has been added.There is already a sale on this product on the same date.
| date | product | quantity |
|---|---|---|
| 2024-01-01 | P1 | 10.0 |
| 2024-01-01 | P2 | 20.0 |
| 2024-01-01 | P1 | 30.0 |
It will capture the added row but also the first row.
| Table Update Detail | sales |
|---|---|
| Change type | ADD_ROWS |
| Change Scope | sales.product = ‘P1’ |
sales.date.
2024-01-01 member count will be off by one (4 instead of 3).
| Old version (member / count) | New version (member / count) |
|---|---|
| 2024-01-01, blue / 2 | 2024-01-01 / 4 |
sales.date, product.color -> sum(sales.quantity * products.price) will be wrong.
The first row will be counted twice in the aggregate provider.
| Old version (points / count) | New version (points / count) |
|---|---|
| 2024-01-01, blue / 1 | 2024-01-01, blue / 3 |
| 2024-01-01, red / 1 | 2024-01-01, red / 1 |
Orthogonal condition on remove or update
Sometimes, despite a clear description of the changed data, it is not possible to exploit the condition to compute a remove where operation.Take an aggregate provider indexed by (K1,K2).
If the condition of the remove or the update is not on some of the two keys, the only solution is to fully refresh the provider.
Impact along Optional Relationship
In this example we will show why Optional relationship could trigger a full
refresh.Here the relationship between
Sales and Products is Optional. So it means, there could have some sales on a product not
present in products.
Data before changes
Sales
| date | product | quantity |
|---|---|---|
| 2024-01-01 | P1 | 10.0 |
| 2024-01-01 | P2 | 20.0 |
| 2024-01-01 | P3 | 30.0 |
Products
| date | product | color | unit_price |
|---|---|---|---|
| 2024-01-01 | P1 | blue | 5.0 |
| 2024-01-01 | P2 | red | 10.0 |
| date | color | sum(quantity*unit_price) |
|---|---|---|
| 2024-01-01 | blue | 50.0 |
| 2024-01-01 | red | 200.0 |
| 2024-01-01 | N/A | 0.0 |
| date | product | color | unit_price |
|---|---|---|---|
| 2024-01-01 | P1 | blue | 5.0 |
| 2024-01-01 | P2 | red | 10.0 |
| 2024-01-01 | P3 | green | 15.0 |
| Table Update Detail | products |
|---|---|
| Change type | ADD_ROWS |
| Change Scope | products.color = ‘green’ |
Optional relationship we will end up with a full refresh of the aggregate provider.