☰
Navigation :

Azure CSV Source

This source is for loading data from CSV files stored on Azure Blob Storage.

The DLC ensures that only files within that source can be loaded.

note

To use this source, a BlobServiceClient bean must be defined. A default one can be used by enabling dlc.create-default-cloud-client.azure.

Configuration Properties

Properties for an Azure CSV Source are defined by name in the dlc.csv.azure.sources namespace, and are picked up as configurations for the DLC.

Key Required Type Description
container Y String The container of the directory.
prefix Y String The directory. Can include Scope Parameters.
Example #{SCOPE_PARAM}
source-properties CsvSourceConfigurationProperties CSV Source Configuration
topics-to-include Set<String> Topics to include. See source to topic matching.
topics-to-exclude Set<String> Topics to exclude. See source to topic matching.
accepts-topic-overrides Boolean Whether this source is allowed to accept Topic Overrides in the Load Request.

YAML Example 

dlc:
  csv:
    azure:
      sources:
        azureCsvSource:
          container: container
          prefix: dataDir

Java Configuration

AzureCsvSourceDescription Spring Beans are picked up as configurations for the DLC.

Parameter Required Type Description
name Y String Name of the source.
container Y String The container of the directory.
prefix Y String The directory. Can include Scope Parameters.
Example #{SCOPE_PARAM}
sourceConfiguration CsvSourceConfiguration CSV Source Configuration
topicsToInclude Set<String> Topics to include. See source to topic matching.
topicsToExclude Set<String> Topics to exclude. See source to topic matching.
acceptsTopicOverrides Boolean Whether this source is allowed to accept Topic Overrides in the Load Request.

Java Example 

@Bean
AzureCsvSourceDescription source() {
    return AzureCsvSourceDescription.builder(
                    "azureCsvSource",
                    "container",
                    "dataDir"
            )
            .build();
}