Skip to main content

Amazon S3 Cloud Source

This documentation page assumes you are already familiar with Amazon Simple Storage Service (Amazon S3).

The Amazon S3 Cloud Source relies on AWS SDK for Java. Make sure you are familiar with this SDK when using the Amazon S3 Cloud Source.

In order to use the AWS source, add the following lines to your pom.xml:

<dependency>
    <groupId>com.activeviam.source</groupId>
    <artifactId>cloud-source-aws</artifactId>
    <version>${atoti-server.version}</version>
</dependency>

Cloud Source to Amazon S3 concepts​

Amazon Simple Storage Service uses two main concepts: the S3 objects that are the fundamental entities stored in Amazon S3 and the buckets that are containers to organize the S3 objects. The objects can be represented by a S3Entity or a S3EntityPath in our cloud source.

Entities​

The AWS implementation of ICloudEntity is S3Entity. It is essentially a wrapper around a S3 object from the AWS Simple Storage Service SDK.

Locating an entity​

Entity paths​

IS3EntityPath implements ICloudEntityPath. It is a reference to an S3 object.

Directories​

The AWS implementation for ICloudDirectory is represented by the S3CloudDirectory implementation.

A directory is tied to a bucket. It contains all S3 objects whose names start with a certain prefix. For example, a directory on a certain container with the prefix directory1/subdirectory2 would contain the first three of the following blobs:

inside:
    directory1/subdirectory2/blob1.txt
    directory1/subdirectory2/blob2.txt
    directory1/subdirectory2/subdirectory3/blob3.txt

not inside:
    blob4.txt
    other_directory/blob5.txt
    directory1/blob6.txt

A directory with an empty prefix corresponds to the root of the container.

An AWS directory object can be constructed by specifying the AmazonS3 client, a bucket name and a prefix.

The AmazonS3 client is the configuration of the connection to Amazon. It can be configured as follows using the Amazon SDK.

final S3Client client =
    S3Client.builder()
        .region(Region.EU_WEST_1)
        .credentialsProvider(DefaultCredentialsProvider.create())
        .httpClientBuilder(
            ApacheHttpClient.builder().maxConnections(128) // Default is 50
            )
        .build();

CsvDataProviderFactory​

To configure the CSV source to read Amazon object, you can use the AwsCsvDataProviderFactory class to configure how the files are downloaded.

Configuration example: how to configure it with a CSV source​

First let's define a generic CSV source configuration. This abstract configuration contains all the part that is common to all the sources and can be used to switch from local to cloud sources easily. In this example we will create a CSV source which loads 2 topics from the cloud:

  • one single CSV file "products.csv" for the products
  • a folder "desks" for the desks table

public abstract class GenericCsvSourceConfig<I> {
  protected abstract ICsvTopic<I> createTopic(
      String topic, String fileName, CsvParserConfiguration parserConfig);
  protected abstract ICsvTopic<I> createDirectoryTopic(
      String topic, String directory, CsvParserConfiguration parserConfig);
  /** Returns CSV source configured with two topics. */
  public ICsvSource<I> csvSource() {
    final ICsvSource<I> csvSource = ICsvSource.<I>builder().build();
    final CsvParserConfiguration parserConfig = getParserConfiguration();
    final ICsvTopic<I> productsTopic = createTopic("PRODUCTS_TOPIC", "products.csv", parserConfig);
    csvSource.addTopic(productsTopic);
    final ICsvTopic<I> desksTopic = createDirectoryTopic("DESKS_TOPIC", "desks", parserConfig);
    csvSource.addTopic(desksTopic);
    return csvSource;
  }
}

Then let's define the AWS specific configuration:

public class AwsSourceConfiguration extends GenericCsvSourceConfig<ICloudEntityPath<S3Entity>> {
  @Override
  protected ICsvTopic<ICloudEntityPath<S3Entity>> createTopic(
      final String topic, final String fileName, final CsvParserConfiguration parserConfig) {
    return new CloudEntityCsvTopic<>(
        topic, parserConfig, dataProviderFactory(), rootDirectory().getEntity(fileName));
  }
  @Override
  protected ICsvTopic<ICloudEntityPath<S3Entity>> createDirectoryTopic(
      final String topic, final String directory, final CsvParserConfiguration parserConfig) {
    return new CloudDirectoryCsvTopic<>(
        topic,
        parserConfig,
        dataProviderFactory(),
        rootDirectory().getSubDirectory(directory),
        null);
  }
  public ICloudDirectory<S3Entity> rootDirectory() {
    return new S3CloudDirectory(createClient(), "myBucket", "root");
  }
  public ICloudCsvDataProviderFactory<S3Entity> dataProviderFactory() {
    return AwsCsvDataProviderFactory.create(
        CloudFetchingConfig.builder().downloadThreadCount(10).build());
  }
}