Incremental Refresh

Incremental refresh is called through the Application API.

final TableUpdateDetail tableUpdateDetail =
    TableUpdateDetail.create(
        "SALES",
        ChangeType.ADD_ROWS,
        ConditionFactory.equal("DATE", LocalDate.of(2022, 4, 16)));
app.refresh(ChangeDescription.create(List.of(tableUpdateDetail)));

Here some change examples with their change description.

Multi table add rows

In this case, we have added some new sales for the seller google.
We have also added a new product. sales has a many-to-one relationship with product.

Table sales changes

date	product	seller	quantity
2024-01-01	P1	microsoft	20
2024-01-01	P2	amazon	30
2024-01-01	P2	amazon	20
2024-01-01	P1	google	20
2024-01-01	P2	google	30
2024-01-01	P3	google	20

Table products changes

date	product	quantity
2024-01-01	P1	10
2024-01-01	P2	20
2024-01-01	P3	30.0

Change description

Table Update Detail	sales	products
Change type	ADD_ROWS	ADD_ROWS
Change Scope	sales.seller = 'google'	product.product = 'P3'

final List<TableUpdateDetail> updateDetails =
    List.of(
        TableUpdateDetail.create(
            "SALES", ChangeType.ADD_ROWS, ConditionFactory.equal("SELLER", "google")),
        TableUpdateDetail.create(
            "PRODUCTS", ChangeType.ADD_ROWS, ConditionFactory.equal("PRODUCT", "P3")));
app.refresh(ChangeDescription.create(updateDetails));

Multi table add rows using an unknown condition

In this case, we have added new rows in sales table.
All these new sales has a new seller which has been added in users table. sales has a many-to-one relationship with users.

Table sales changes

date	product	seller	quantity
2024-01-01	P1	microsoft	20
2024-01-01	P2	amazon	30
2024-01-01	P2	amazon	20
2024-01-02	P1	google	20
2024-01-02	P2	alibaba	30
2024-01-02	P3	ibm	20

Table users changes

id	warehouse
microsoft	nyo-1
amazon	chi-1
google	tor-1
alibaba	bei-1
ibm	par-1

This change can be expressed with this description which could cumbersome to build because of the IN condition.

Change description

Table Update Detail	sales	users
Change type	ADD_ROWS	ADD_ROWS
Change Scope	sales.date = '2024-01-02'	users.id in ('google', 'alibaba','ibm')

Here we managed to resolve a cumbersome condition looking at the data, but sometimes it is not possible to come up with a convenient condition.
In this case you can use the unknown condition.

Change description with an unknown condition

Table Update Detail	sales	users
Change type	ADD_ROWS	ADD_ROWS
Change Scope	sales.date = '2024-01-02'	Unknown condition

This solution could have adverse consequences.
If you have defined a factless hierarchy on user table, the missing link with the fact table forbid us to use the sales table scope.
Connector will full refresh on this hierarchy stream.

final List<TableUpdateDetail> updateDetails =
    List.of(
        TableUpdateDetail.create(
            "SALES",
            ChangeType.ADD_ROWS,
            ConditionFactory.equal("DATE", LocalDate.of(2024, 1, 2))),
        TableUpdateDetail.create("PRODUCTS", ChangeType.ADD_ROWS, ConditionFactory.unknown()));
app.refresh(ChangeDescription.create(updateDetails));