Skip to main content

Incremental Refresh

Incremental refresh is called through the Application API. It is important to note that all table names and field names are those from the DirectQuery local schema.

final TableUpdateDetail tableUpdateDetail =
TableUpdateDetail.create(
"SALES",
ChangeType.ADD_ROWS,
ConditionFactory.equal("DATE", LocalDate.of(2022, 4, 16)));
app.refresh(ChangeDescription.create(List.of(tableUpdateDetail)));

Here some change examples with their change description.

Multi table add rows

In this case, we have added some new sales for the seller google.
We have also added a new product. sales has a many-to-one relationship with product.

Table sales changes

dateproductsellerquantity
2024-01-01P1microsoft20
2024-01-01P2amazon30
2024-01-01P2amazon20
2024-01-01P1google20
2024-01-01P2google30
2024-01-01P3google20

Table products changes

dateproductquantity
2024-01-01P110
2024-01-01P220
2024-01-01P330.0

Change description

Table Update Detailsalesproducts
Change typeADD_ROWSADD_ROWS
Change Scopesales.seller = 'google'product.product = 'P3'

final List<TableUpdateDetail> updateDetails =
List.of(
TableUpdateDetail.create(
"SALES", ChangeType.ADD_ROWS, ConditionFactory.equal("SELLER", "google")),
TableUpdateDetail.create(
"PRODUCTS", ChangeType.ADD_ROWS, ConditionFactory.equal("PRODUCT", "P3")));
app.refresh(ChangeDescription.create(updateDetails));

Multi table add rows using an unknown condition

In this case, we have added new rows in sales table.
All these new sales has a new seller which has been added in users table. sales has a many-to-one relationship with users.

Table sales changes

dateproductsellerquantity
2024-01-01P1microsoft20
2024-01-01P2amazon30
2024-01-01P2amazon20
2024-01-02P1google20
2024-01-02P2alibaba30
2024-01-02P3ibm20

Table users changes

idwarehouse
microsoftnyo-1
amazonchi-1
googletor-1
alibababei-1
ibmpar-1

This change can be expressed with this description which could be cumbersome to build because of the IN condition.

Change description

Table Update Detailsalesusers
Change typeADD_ROWSADD_ROWS
Change Scopesales.date = '2024-01-02'users.id in ('google', 'alibaba','ibm')

Here we managed to resolve a cumbersome condition looking at the data, but sometimes it is not possible to come up with a convenient condition.
In this case you can use the unknown condition.

Change description with an unknown condition

Table Update Detailsalesusers
Change typeADD_ROWSADD_ROWS
Change Scopesales.date = '2024-01-02'Unknown condition

This solution could have adverse consequences.
If you have defined a factless hierarchy on user table, the missing link with the fact table forbid us to use the sales table scope.
Connector will full refresh on this hierarchy stream.

final List<TableUpdateDetail> updateDetails =
List.of(
TableUpdateDetail.create(
"SALES",
ChangeType.ADD_ROWS,
ConditionFactory.equal("DATE", LocalDate.of(2024, 1, 2))),
TableUpdateDetail.create("PRODUCTS", ChangeType.ADD_ROWS, ConditionFactory.unknown()));
app.refresh(ChangeDescription.create(updateDetails));