Azure CSV source

The container and prefix to read from are set through the container and prefix properties. The DLC ensures that only files within that source can be loaded.

To use this source, a BlobServiceClient bean must be defined. A default one can be used by enabling dlc.create-default-cloud-client.azure.

Configuration properties

Properties for an Azure CSV Source are defined by name in the dlc.csv.azure.sources namespace, and are picked up as configurations for the DLC.

Key	Required	Type	Description
container	Y	`String`	The container of the directory.
prefix	Y	`String`	The directory. Can include Scope Parameters. Example `#{SCOPE_PARAM}`
source-properties		`CsvSourceConfigurationProperties`	CSV Source Configuration
cloud-fetching-config		`CloudFetchingConfigurationProperties`	Cloud Fetching Config
topics-to-include		`Set<String>`	Topics to include. See source to topic matching.
topics-to-exclude		`Set<String>`	Topics to exclude. See source to topic matching.
accepts-topic-overrides		`Boolean`	Whether this source is allowed to accept Topic Overrides in the Load Request.

YAML example

dlc:
  csv:
    azure:
      sources:
        azureCsvSource:
          container: container
          prefix: dataDir

Java configuration

AzureCsvSourceDescription Spring Beans are picked up as configurations for the DLC.

Parameter	Required	Type	Description
name	Y	`String`	Name of the source.
container	Y	`String`	The container of the directory.
prefix	Y	`String`	The directory. Can include Scope Parameters. Example `#{SCOPE_PARAM}`
sourceConfiguration		`CsvSourceConfiguration`	CSV Source Configuration
cloudFetchingConfig		`CloudFetchingConfig`	Cloud Fetching Config
topicsToInclude		`Set<String>`	Topics to include. See source to topic matching.
topicsToExclude		`Set<String>`	Topics to exclude. See source to topic matching.
acceptsTopicOverrides		`Boolean`	Whether this source is allowed to accept Topic Overrides in the Load Request.

Java example

@Bean
AzureCsvSourceDescription source() {
    return AzureCsvSourceDescription.builder(
                    "azureCsvSource",
                    "container",
                    "dataDir"
            )
            .build();
}

AWS CSV source

GCP CSV source

⌘I

Operations

Request API

Configuration

Components

Functional

Extensions

Configuration properties

YAML example

Java configuration

Java example

​Configuration properties

​YAML example

​Java configuration

​Java example

Configuration properties

YAML example

Java configuration

Java example