Navigation :

Local CSV Source

This source is for loading CSV files from the local file system.

You need to specify a directory. The DLC ensures that only files within that directory can be loaded.

Configuration Properties

The configuration properties for a Local CSV Source are picked up as configurations for the DLC.

Properties for a Local CSV Source are defined in the dlc.local-csv-sources namespace. Multiple sources can be defined.

Key	Required	Type		Description
source-name	Y	String		Name of the source .
root-base-dir	Y	String		Root base directory. Can include scope parameters for example `#{SCOPE_PARAM}`.
source-properties		CsvSourceProperties	Default CSV Source Properties	CSV Source Properties
topics-to-include		Set<String>		Topics to include.
topics-to-exclude		Set<String>		Topics to exclude.
accepts-topic-overrides		Boolean		Whether this source can accept Topic Overrides.

Example

dlc:
  local-csv-sources:
    - source-name: source
      root-base-dir: 'data'
    - source-name: datedSource
      root-base-dir: 'data/#{AS_OF_DATE}'

Java Configuration

LocalCsvSourceDescription Spring Beans are picked up as configurations for the DLC.

LocalCsvSourceDescription

Parameter	Required	Type		Description
name	Y	String		Name of the source.
rootBaseDir	Y	String		Root base directory.
sourceProperties		CsvSourceConfiguration	Default CSV Source Config	CSV Source Configuration.
topicsToInclude		Set<String>		Topics to include.
topicsToExclude		Set<String>		Topics to exclude.
acceptsTopicOverrides		Boolean		Whether this source can accept Topic Overrides.

Example

@Bean
LocalCsvSourceDescription source() {
    return LocalCsvSourceDescription.builder("source", "data")
            .build();
}

@Bean
LocalCsvSourceDescription datedSource() {
    return LocalCsvSourceDescription.builder("source", "data/#{AS_OF_DATE}")
            .build();
}