Navigation :

Cloud Fetching Config

Configuration Properties

Properties for the Cloud Fetching Config are provided inside the configuration of a cloud source. For example: AWS CSV Source.

Key	Type	Default	Description
download-thread-count	Integer	1.5 times the number of the server’s logical cores, rounded.	The maximum number of downloader threads for one single file.
part-length	Integer	8 * 1024 * 1024	The amount of chunks that can be downloaded in advance. This is used to compensate network speed instability.
prefetched-parts-limit	Integer	$$\max\left(1000 - threadCount, 0\right)$$	The size in bytes of an individual downloaded part.

YAML Example

dlc:
  csv:
    aws:
      sources:
        awsCsvSource:
          bucket: bucket
          prefix: dataDir
          cloud-fetching-config:
            prefetched-parts-limit: 10
            part-length: 20
            download-thread-count: 30

Java Configuration

The CloudFetchingConfig Java object can be used in the configuration of a cloud source.

For more information, see the Atoti Server documentation on CloudFetchingConfig.

Java Example

@Bean
AwsCsvSourceDescription source() {
    return AwsCsvSourceDescription.builder("awsCsvSource", "bucket", "dataDir")
            .cloudFetchingConfig(CloudFetchingConfig.builder()
                    .prefetchedPartsLimit(10)
                    .partLength(20)
                    .downloadThreadCount(30)
                    .build())
            .build();
}