Cloud Fetching Config
Configuration Properties
Properties for the Cloud Fetching Config are provided inside the configuration of a cloud source. For example: AWS CSV Source.
Key | Type | Default | Description |
---|---|---|---|
download-thread-count | Integer | 1.5 times the number of the server’s logical cores, rounded. | The maximum number of downloader threads for one single file. |
part-length | Integer | 8 * 1024 * 1024 | The amount of chunks that can be downloaded in advance. This is used to compensate network speed instability. |
prefetched-parts-limit | Integer | $$\max\left(1000 - threadCount, 0\right)$$ | The size in bytes of an individual downloaded part. |
YAML Example
dlc:
csv:
aws:
sources:
awsCsvSource:
bucket: bucket
prefix: dataDir
cloud-fetching-config:
prefetched-parts-limit: 10
part-length: 20
download-thread-count: 30
Java Configuration
The CloudFetchingConfig
Java object can be used in the configuration of a
cloud source.
For more information, see the Atoti Server documentation on CloudFetchingConfig.
Java Example
@Bean
AwsCsvSourceDescription source() {
return AwsCsvSourceDescription.builder("awsCsvSource", "bucket", "dataDir")
.cloudFetchingConfig(CloudFetchingConfig.builder()
.prefetchedPartsLimit(10)
.partLength(20)
.downloadThreadCount(30)
.build())
.build();
}