Configuration flow

Configurations defined in Java have the lowest precedence and can be overridden by configuration properties, which in turn can be overridden at request time.

Configuration flow

Configurations defined in Java (Description Spring Beans) have the lowest precedence and can be overridden by configurations defined in configuration properties. In addition, loading Topic Configurations can be overridden at request time.

Startup configuration

Configuration properties

Configuration Properties get converted to Description objects on startup. These Configurations take precedence over Java Configuration.

Configuration Properties have some limitations — for example, column calculators and targets cannot be defined in configuration properties, only referenced by name.

YAML example

dlc:
  csv:
    local:
      sources:
        source1:
          root-base-dir: /base-dir
    topics:
      topic1:
        file-pattern: glob:trades*.csv

Java configuration

Description Spring Beans are picked up as configuration.

Java example

@Bean
public LocalCsvSourceDescription source1() {
    return LocalCsvSourceDescription.builder("source1", "/base-dir").build();
}

@Bean
public CsvTopicDescription topic1(NamedEntityResolverService namedEntityResolver) {
    return CsvTopicDescription.builder("topic1", "glob:trades*.csv")
            .build();
}

Request time

Loading topic overrides

At request time, the configurations defined at startup for a loading topic can also be overridden by providing configuration properties in the request. These override some or all of the configurations which belong to a loading topic which was configured at startup.

Example

{
  "operation": "LOAD",
  "csvTopicOverrides": {
    "trades": {
      "filePattern": "glob:alternative_trades*.csv"
    }
  }
}

Functional behavior

Garbage collection

⌘I

Operations

Request API

Configuration

Components

Functional

Extensions

Configuration flow