Incremental Refresh
Incremental refresh is called through the Application API.
final TableUpdateDetail tableUpdateDetail =
TableUpdateDetail.create(
"SALES",
ChangeType.ADD_ROWS,
ConditionFactory.equal("DATE", LocalDate.of(2022, 4, 16)));
app.refresh(ChangeDescription.create(List.of(tableUpdateDetail)));
Here some change examples with their change description.
Multi table add rows
In this case, we have added some new sales for the seller google.
We have also added a new product.
sales
has a many-to-one relationship with product
.
Table sales
changes
date | product | seller | quantity |
---|---|---|---|
2024-01-01 | P1 | microsoft | 20 |
2024-01-01 | P2 | amazon | 30 |
2024-01-01 | P2 | amazon | 20 |
2024-01-01 | P1 | 20 | |
2024-01-01 | P2 | 30 | |
2024-01-01 | P3 | 20 |
Table products
changes
date | product | quantity |
---|---|---|
2024-01-01 | P1 | 10 |
2024-01-01 | P2 | 20 |
2024-01-01 | P3 | 30.0 |
Change description
Table Update Detail | sales | products |
---|---|---|
Change type | ADD_ROWS | ADD_ROWS |
Change Scope | sales.seller = 'google' | product.product = 'P3' |
final List<TableUpdateDetail> updateDetails =
List.of(
TableUpdateDetail.create(
"SALES", ChangeType.ADD_ROWS, ConditionFactory.equal("SELLER", "google")),
TableUpdateDetail.create(
"PRODUCTS", ChangeType.ADD_ROWS, ConditionFactory.equal("PRODUCT", "P3")));
app.refresh(ChangeDescription.create(updateDetails));
Multi table add rows using a relative condition
In this case, we have added new rows in sales table.
All these new sales has a new seller which has been added in users table.
sales
has a many-to-one relationship with users
.
Table sales
changes
date | product | seller | quantity |
---|---|---|---|
2024-01-01 | P1 | microsoft | 20 |
2024-01-01 | P2 | amazon | 30 |
2024-01-01 | P2 | amazon | 20 |
2024-01-02 | P1 | 20 | |
2024-01-02 | P2 | alibaba | 30 |
2024-01-02 | P3 | ibm | 20 |
Table users
changes
id | warehouse |
---|---|
microsoft | nyo-1 |
amazon | chi-1 |
tor-1 | |
alibaba | bei-1 |
ibm | par-1 |
This change can be expressed with this description which could cumbersome to build because of the IN
condition.
Change description
Table Update Detail | sales | users |
---|---|---|
Change type | ADD_ROWS | ADD_ROWS |
Change Scope | sales.date = '2024-01-02' | users.id in ('google', 'alibaba','ibm') |
Instead of using an IN
condition we tell that all the rows added are relative to the row added in others table
Relative change description
Table Update Detail | sales | users |
---|---|---|
Change type | ADD_ROWS | ADD_ROWS |
Change Scope | sales.date = '2024-01-02' | Relative condition |
This solution could have adverse consequences.
If you have defined a factless hierarchy on user, the missing link with the fact table forbid us to use the sales
table scope.
Connector will full refresh on this hierarchy stream.
final List<TableUpdateDetail> updateDetails =
List.of(
TableUpdateDetail.create(
"SALES",
ChangeType.ADD_ROWS,
ConditionFactory.equal("DATE", LocalDate.of(2024, 1, 2))),
TableUpdateDetail.create("PRODUCTS", ChangeType.ADD_ROWS, ConditionFactory.relative()));
app.refresh(ChangeDescription.create(updateDetails));