Navigation :

Configuration Flow

The following sections describe the different ways to define configurations and their order of precedence.

Configuration Flow Diagram

graph TD subgraph STARTUP CONFIGURATION B(Java Configuration) B --> C(Configuration Properties) end subgraph REQUEST TIME CONFIGURATION C --> D(Request Time Configuration) end style B fill:#bbf,stroke:#333 style C fill:#bfb,stroke:#333 style D fill:#ffb,stroke:#333

Startup Configuration

On startup, the DLC reads the configuration from Configuration Properties as well as Java Spring Beans.

Configuration Properties

Configuration Properties get converted to Description objects on startup. These Configurations take precedence over Java Configuration.

note

Configuration Properties have some limitations — for example, column calculators and sinks cannot be defined in configuration properties, only referenced by name.

Example

dlc:
  local-csv-sources:
    - source-name: source1
      root-base-dir: /base-dir
  csv-topics:
    - topic-name: topic1
      file-pattern: trades*.csv

Java Configuration

Description objects in Java are picked up as configuration as long as they are Spring Beans.

Example

@Bean
public LocalCsvSourceDescription source1() {
    return LocalCsvSourceDescription.builder("source1", "/base-dir").build();
}

@Bean
public CsvTopicDescription topic1(NamedEntityResolverService namedEntityResolver) {
    return CsvTopicDescription.builder("topic1")
            .filePattern("trades*.csv")
            .build();
}

Request Time Configuration

At request time, you can also override the configurations defined at startup for a topic by providing configuration properties in your request.

Topics

Anonymous Topics

These do not override a topic which exists in the startup configuration.

Example

{
  "operation": "LOAD",
  "csvTopicOverrides": {
    "newTopic": {
      "filePattern": "alternative_trades*.csv",
      "channels": [
        {
          "targetName": "trades"
        }
      ]
    }
  }
}

Override Topics

These override some or all of the configurations which belong to a topic which was configured at startup.

Example

{
  "operation": "LOAD",
  "csvTopicOverrides": {
    "trades": {
      "filePattern": "alternative_trades*.csv"
    }
  }
}