Navigation :

AWS CSV Source

This source is for loading data from CSV files stored on Amazon S3.

The DLC ensures that only files within that source can be loaded.

note

To use this source, an S3Client bean must be defined. A default one can be used by enabling dlc.create-default-cloud-client.aws.

Configuration Properties

Properties for an AWS CSV Source are defined by name in the dlc.csv.aws.sources namespace, and are picked up as configurations for the DLC.

Key	Required	Type	Description
bucket	Y	`String`	The bucket of the directory.
prefix	Y	`String`	The directory. Can include scope parameters. Example `#{SCOPE_PARAM}`
source-properties		`CsvSourceProperties`	CSV Source Properties
topics-to-include		`Set<String>`	Topics to include.
topics-to-exclude		`Set<String>`	Topics to exclude.
accepts-topic-overrides		`Boolean`	Whether this source is allowed to accept Topic Overrides in the DLC Load Request.

YAML Example

dlc:
  csv:
    aws:
      sources:
        awsCsvSource:
          bucket: bucket
          prefix: dataDir

Java Configuration

AwsCsvSourceDescription Spring Beans are picked up as configurations for the DLC.

Parameter	Required	Type	Description
name	Y	`String`	Name of the source.
bucket	Y	`String`	The bucket of the directory
prefix	Y	`String`	The directory. Can include scope parameters. Example `#{SCOPE_PARAM}`
sourceConfiguration		`CsvSourceConfiguration`	CSV Source Configuration
topicsToInclude		`Set<String>`	Topics to include.
topicsToExclude		`Set<String>`	Topics to exclude.
acceptsTopicOverrides		`Boolean`	Whether this source is allowed to accept Topic Overrides in the DLC Load Request.

Java Example

@Bean
AwsCsvSourceDescription source() {
    return AwsCsvSourceDescription.builder(
                    "awsCsvSource",
                    "bucket",
                    "dataDir"
            )
            .build();
}