☰
Navigation :

AWS CSV Source

This source is for loading data from CSV files stored on Amazon S3.

The DLC ensures that only files within that source can be loaded.

note

To use this source, an S3Client bean must be defined. A default one can be used by enabling dlc.create-default-cloud-client.aws.

Configuration Properties

Properties for an AWS CSV Source are defined by name in the dlc.csv.aws.sources namespace, and are picked up as configurations for the DLC.

Key Required Type Description
bucket Y String The bucket of the directory.
prefix Y String The directory. Can include Scope Parameters.
Example #{SCOPE_PARAM}
source-properties CsvSourceConfigurationProperties CSV Source Configuration
cloud-fetching-config CloudFetchingConfigurationProperties Cloud Fetching Config
topics-to-include Set<String> Topics to include. See source to topic matching.
topics-to-exclude Set<String> Topics to exclude. See source to topic matching.
accepts-topic-overrides Boolean Whether this source is allowed to accept Topic Overrides in the Load Request.

YAML Example 

dlc:
  csv:
    aws:
      sources:
        awsCsvSource:
          bucket: bucket
          prefix: dataDir

Java Configuration

AwsCsvSourceDescription Spring Beans are picked up as configurations for the DLC.

Parameter Required Type Description
name Y String Name of the source.
bucket Y String The bucket of the directory
prefix Y String The directory. Can include Scope Parameters.
Example #{SCOPE_PARAM}
sourceConfiguration CsvSourceConfiguration CSV Source Configuration
cloudFetchingConfig CloudFetchingConfig Cloud Fetching Config
topicsToInclude Set<String> Topics to include. See source to topic matching.
topicsToExclude Set<String> Topics to exclude. See source to topic matching.
acceptsTopicOverrides Boolean Whether this source is allowed to accept Topic Overrides in the Load Request.

Java Example 

@Bean
AwsCsvSourceDescription source() {
    return AwsCsvSourceDescription.builder(
                    "awsCsvSource",
                    "bucket",
                    "dataDir"
            )
            .build();
}