Cloud fetching config

Configuration properties

Properties for the Cloud Fetching Config are provided inside the configuration of a cloud source. For example: AWS CSV Source.

Key	Type	Default	Description
download-thread-count	Integer	1.5 times the number of the server’s logical cores, rounded.	The maximum number of downloader threads for one single file.
part-length	Integer	8 * 1024 * 1024	The amount of chunks that can be downloaded in advance. This is used to compensate network speed instability.
prefetched-parts-limit	Integer	`max(1000 - threadCount, 0)`	The size in bytes of an individual downloaded part.

YAML example

dlc:
  csv:
    aws:
      sources:
        awsCsvSource:
          bucket: bucket
          prefix: dataDir
          cloud-fetching-config:
            prefetched-parts-limit: 10
            part-length: 20
            download-thread-count: 30

Java configuration

The CloudFetchingConfig Java object can be used in the configuration of a cloud source. For more information, see the Atoti Server documentation on CloudFetchingConfig.

Java example

@Bean
AwsCsvSourceDescription source() {
    return AwsCsvSourceDescription.builder("awsCsvSource", "bucket", "dataDir")
            .cloudFetchingConfig(CloudFetchingConfig.builder()
                    .prefetchedPartsLimit(10)
                    .partLength(20)
                    .downloadThreadCount(30)
                    .build())
            .build();
}

Channel

Custom field

⌘I

Operations

Request API

Configuration

Components

Functional

Extensions

Cloud fetching config

Configuration properties

YAML example

Java configuration

Java example

​Configuration properties

​YAML example

​Java configuration

​Java example

Configuration properties

YAML example

Java configuration

Java example