Skip to main content

Incremental Refresh

Incremental refresh is called through the Application API.

final TableUpdateDetail tableUpdateDetail =
TableUpdateDetail.create(
"SALES",
ChangeType.ADD_ROWS,
ConditionFactory.equal("DATE", LocalDate.of(2022, 4, 16)));
app.refresh(ChangeDescription.create(List.of(tableUpdateDetail)));

Here some change examples with their change description.

Multi table add rows

In this case, we have added some new sales for the seller google.
We have also added a new product. sales has a many-to-one relationship with product.

Table sales changes

dateproductsellerquantity
2024-01-01P1microsoft20
2024-01-01P2amazon30
2024-01-01P2amazon20
2024-01-01P1google20
2024-01-01P2google30
2024-01-01P3google20

Table products changes

dateproductquantity
2024-01-01P110
2024-01-01P220
2024-01-01P330.0

Change description

Table Update Detailsalesproducts
Change typeADD_ROWSADD_ROWS
Change Scopesales.seller = 'google'product.product = 'P3'

final List<TableUpdateDetail> updateDetails =
List.of(
TableUpdateDetail.create(
"SALES", ChangeType.ADD_ROWS, ConditionFactory.equal("SELLER", "google")),
TableUpdateDetail.create(
"PRODUCTS", ChangeType.ADD_ROWS, ConditionFactory.equal("PRODUCT", "P3")));
app.refresh(ChangeDescription.create(updateDetails));

Multi table add rows using an unknown condition

In this case, we have added new rows in sales table.
All these new sales has a new seller which has been added in users table. sales has a many-to-one relationship with users.

Table sales changes

dateproductsellerquantity
2024-01-01P1microsoft20
2024-01-01P2amazon30
2024-01-01P2amazon20
2024-01-02P1google20
2024-01-02P2alibaba30
2024-01-02P3ibm20

Table users changes

idwarehouse
microsoftnyo-1
amazonchi-1
googletor-1
alibababei-1
ibmpar-1

This change can be expressed with this description which could cumbersome to build because of the IN condition.

Change description

Table Update Detailsalesusers
Change typeADD_ROWSADD_ROWS
Change Scopesales.date = '2024-01-02'users.id in ('google', 'alibaba','ibm')

Here we managed to resolve a cumbersome condition looking at the data, but sometimes it is not possible to come up with a convenient condition.
In this case you can use the unknown condition.

Change description with an unknown condition

Table Update Detailsalesusers
Change typeADD_ROWSADD_ROWS
Change Scopesales.date = '2024-01-02'Unknown condition

This solution could have adverse consequences.
If you have defined a factless hierarchy on user table, the missing link with the fact table forbid us to use the sales table scope.
Connector will full refresh on this hierarchy stream.

final List<TableUpdateDetail> updateDetails =
List.of(
TableUpdateDetail.create(
"SALES",
ChangeType.ADD_ROWS,
ConditionFactory.equal("DATE", LocalDate.of(2024, 1, 2))),
TableUpdateDetail.create("PRODUCTS", ChangeType.ADD_ROWS, ConditionFactory.unknown()));
app.refresh(ChangeDescription.create(updateDetails));