Skip to main content

​
Configuration properties

Properties for the Cloud Fetching Config are provided inside the configuration of a cloud source. For example: AWS CSV Source.
KeyTypeDefaultDescription
download-thread-countInteger1.5 times the number of the server’s logical cores, rounded.The maximum number of downloader threads for one single file.
part-lengthInteger8 * 1024 * 1024The amount of chunks that can be downloaded in advance. This is used to compensate network speed instability.
prefetched-parts-limitIntegermax(1000 - threadCount, 0)The size in bytes of an individual downloaded part.

​
YAML example

dlc:
  csv:
    aws:
      sources:
        awsCsvSource:
          bucket: bucket
          prefix: dataDir
          cloud-fetching-config:
            prefetched-parts-limit: 10
            part-length: 20
            download-thread-count: 30

​
Java configuration

The CloudFetchingConfig Java object can be used in the configuration of a cloud source. For more information, see the Atoti Server documentation on CloudFetchingConfig.

​
Java example

@Bean
AwsCsvSourceDescription source() {
    return AwsCsvSourceDescription.builder("awsCsvSource", "bucket", "dataDir")
            .cloudFetchingConfig(CloudFetchingConfig.builder()
                    .prefetchedPartsLimit(10)
                    .partLength(20)
                    .downloadThreadCount(30)
                    .build())
            .build();
}