☰
Navigation :

Local Avro Source

This source is for loading data from Avro files stored locally.

The DLC ensures that only files within that source can be loaded.

Configuration Properties

Properties for a Local Avro Source are defined by name in the dlc.avro.local.sources namespace, and are picked up as configurations for the DLC.

Key Required Type Description
root-base-dir Y String Root base directory. Can include Scope Parameters.
Example #{SCOPE_PARAM}
topics-to-include Set<String> Topics to include. See source to topic matching.
topics-to-exclude Set<String> Topics to exclude. See source to topic matching.
accepts-topic-overrides Boolean Whether this source is allowed to accept Topic Overrides in the Load Request.

YAML Example 

dlc:
  avro:
    local:
      sources:
        localAvroSource:
          root-base-dir: data
        datedSource:
          root-base-dir: data/#{AS_OF_DATE}

Java Configuration

LocalAvroSourceDescription Spring Beans are picked up as configurations for the DLC.

Parameter Required Type Description
name Y String Name of the source.
rootBaseDir Y String Root base directory.
topicsToInclude Set<String> Topics to include. See source to topic matching.
topicsToExclude Set<String> Topics to exclude. See source to topic matching.
acceptsTopicOverrides Boolean Whether this source is allowed to accept Topic Overrides in the Load Request.

Java Example 

@Bean
LocalAvroSourceDescription source() {
    return LocalAvroSourceDescription.builder("localAvroSource", "data")
            .build();
}

@Bean
LocalAvroSourceDescription datedSource() {
    return LocalAvroSourceDescription.builder("datedSource", "data/#{AS_OF_DATE}")
            .build();
}