Scope Parameters

Scope Parameters are a key-value map. They can be provided in a DlcRequest and can be used to perform the following:

• Filter what gets loaded
• Provide default values

Filter Files to Load

Base Directory of CSV Source

The scope parameters is used with the base directory of a CSV source.

Scope parameters will be inserted using text replacement for strings in the base directory which start with “#{” and end in “}”.

Example

This source accepts an AS_OF_DATE scope parameter

dlc:
  csv:
    local:
      sources:
        root-base-dir: /base-dir/#{AS_OF_DATE}

This request provides the AS_OF_DATE scope parameter to the base directory and we then retrieve the data from /base-dir/2021-01-01

{
  "operation": "LOAD",
  "topics": [ "trades" ],
  "scope": {
    "AS_OF_DATE": "2021-01-01"
  }
}

Column Calculator

The scope parameter is used with a ColumnCalculatorDescription

Example

@Bean
ColumnCalculatorDescription calculatorDescription() {
    return ColumnCalculatorDescription.of(
            "columnCalculator",
            (DlcOperationScope s)-> new ColumnCalculatorWhichUsesScope(s)
    );
}

Sink

The scope parameter is used with a SinkDescription

Example

@Bean
public SinkDescription sinkDescription() {
    return SinkDescription.of("sink", "store_name",
            (DlcOperationScope s)-> new TuplePublisherWhichUsesScope(s)
    );
}

Garbage Collection

Garbage Collection may need to be performed before or after executing a DLC operation. This can be done by adding one of the following key-value entries into your DLC request scope:

{
  "scope": {
    "performGcOnStart": true,
    "performGcOnCompletion": true
  }
}

These properties run a GC request as follows:

• "performGcOnStart": before the Operation is executed.
• "performGcOnCompletion": after the Operation is completed