Scope Parameters

DLC Scope Parameters are a key-value map. They can be provided in a DlcRequest and can be used to perform the following:

• Filter what gets loaded
• Provide default values

Filter Files to Load

Files

The files scope key allows you to explicitly specify a collection of files to load.

By default, all files are loaded for every topic in the request, regardless of the topic’s file pattern. To modify this behavior, set the apply-file-pattern key to true so that each topic in the request only loads files matching its file pattern.

Base Directory Resolution

Custom scope parameters can be used to dynamically define the base directory of a source. These parameters allow for flexible configuration by substituting placeholders in the directory path with actual values provided in the request.

Scope parameters are inserted into the base directory path using a text replacement mechanism:

• Placeholders in the path must start with #{ and end with }.
• At request time, these placeholders are substituted with the corresponding values from the scope, and are blank if not provided.

Example

This source accepts an AS_OF_DATE scope parameter

dlc:
  csv:
    local:
      sources:
        root-base-dir: /base-dir/#{AS_OF_DATE}

This request provides the AS_OF_DATE scope parameter to the base directory and we then retrieve the data from /base-dir/2021-01-01

{
  "operation": "LOAD",
  "topics": [
    "trades"
  ],
  "scope": {
    "AS_OF_DATE": "2021-01-01"
  }
}

Provide default values

Column Calculator

The scope parameters can be used in a CustomFieldDescription

Example

@Bean
CustomFieldDescription calculatorDescription() {
    return CustomFieldDescription.of(
            "columnCalculator",
            (DlcScope s) -> new ColumnCalculatorWhichUsesScope(s)
    );
}

Target

The scope parameters can be used in a TargetDescription

Example

@Bean
public TargetDescription targetDescription() {
    return TargetDescription.of("target", "store_name",
            (DlcScope s) -> new TuplePublisherWhichUsesScope(s)
    );
}