Scope Parameters

Scope Parameters are a key-value map. They can be provided in a DlcRequest and can be used to perform the following:

• Filter what gets loaded
• Provide default values

Filter Files to Load

Base Directory of CSV Source

The scope parameters is used with the base directory of a CSV source.

Scope parameters will be inserted using text replacement for strings in the base directory which start with “#{” and end in “}”.

Example

This source accepts an AS_OF_DATE scope parameter

dlc:
  csv-topics:
    - topic-name: topic1
      base-dir: /base-dir/#{AS_OF_DATE}

This request provides the AS_OF_DATE scope parameter to the base directory and we then retrieve the data from /base-dir/2021-01-01

{
  "operation": "LOAD",
  "topics": [ "trades" ],
  "scope": {
    "AS_OF_DATE": "2021-01-01"
  }
}

Column Calculator

The scope parameter is used with a ColumnCalculatorDescription

Example

@Bean
ColumnCalculatorDescription calculatorDescription() {
    return ColumnCalculatorDescription.of(
            "columnCalculator",
            (DlcOperationScope s)-> new ColumnCalculatorWhichUsesScope(s)
    );
}

Sink

The scope parameter is used with a SinkDescription

Example

@Bean
public SinkDescription sinkDescription() {
    return SinkDescription.of("sink", "store_name",
            (DlcOperationScope s)-> new TuplePublisherWhichUsesScope(s)
    );
}