☰
Navigation :

Migration Notes - 5.0

This page explains the changes required to migrate from 4.2.0-AS6.1 to 5.0.x version of Atoti Data Connectors.

5.0.0-M2

Migrate to 5.0.0-M2

Overview

The DLC has been completely re-written. For more information, see the Release Notes and our step guide to migrating to the new DLC API

This tool is compatible with Atoti Server 6.1.1.

For new features and fixes included in this Atoti Server release, please see the Atoti Server6.1.1 Changelog.

Architectural Changes

Description objects

  • Previously, you had to register all Sources, Channels, Topics, Converters, etc. within the Data Load Controller itself. These registered objects were wrappers around core Atoti objects on which actions (such as load) would be directly performed.
    In the new DLC, the DLC environment is built of Description objects. These description objects can be defined in Java as Spring beans or via configuration properties. The properties can be used to override / customize the Java beans to allow for different execution behavior.

  • The DLC’s description objects will be converted to Atoti core objects at request time. This means that DLC Topic & Source definitions will be converted into Atoti Source & Topic instances during a DLC request. The Atoti instances will not be persisted (except when a listening pipe is opened).

Services

  • What used to be the DataLoadController is now the DataLoadControllerService. The DataLoadControllerService is responsible for handling the requests which come to the DLC.

  • The logic for handling a specific operation used to be handled by registered implementations of ADatastoreTransactionTask. This is now an operation bean.

  • The DLC now makes inferences where possible, so the minimal configuration is drastically reduced.

Dependency Changes

This version includes modules from the previous version of the DLC which are now prefixed deprecated-. These modules were included to aid the migration process and will not be present in the final release.

A starting point for your migration is to replace existing DLC modules with these deprecated modules, and then import the new ones outlined in the Getting Started guide.

Configuration Classes

This version of the DLC uses Spring Auto Configuration to load required configuration classes.

Current limitations

Operations

This milestone release supports LOAD, UNLOAD, START_LISTEN, LISTEN_STATUS and STOP_LISTEN operations.

Sources

This milestone release supports Local CSV, JDBC, Tuple, Avro, Parquet, AWS CSV, Azure CSV, and GCP CSV sources.

The final release will add support for Kafka, RabbitMQ and fetching/listening for any files from cloud storage (AWS, Azure, GCP).

Configuration files

We have added the ability to configure the DLC through Configuration Properties.

Configuration in Java

The Java Configuration is now done via Spring Beans.

DEE 5.0.0-M2 Migration

The DEE has not been altered in this release of Atoti Data Connectors, therefore no migration is needed.

DLC 5.0.0-M2 Migration

The rewritten DLC in Atoti Data Connectors 5.0 requires significant steps to migrate to the new DLC API.

We outline these steps in the following sections:

  1. Dependency Management
  2. Remove IDataLoadController
  3. Source migration
  4. Topic migration
  5. Requests
  6. Operations
  7. REST API

Dependency Management

First we will refactor our project to use the “deprecated” DLC API.

Each of the deprecated modules provided in Atoti Data Connectors 4.x have been prefixed with deprecated-.

<dependency>
    <groupId>com.activeviam.io</groupId>
    <artifactId>data-connectors-csv</artifactId>
    <version>4.2.0-AP56.1</version>
</dependency>

Migrates to:

<dependency>
    <groupId>com.activeviam.io</groupId>
    <artifactId>deprecated-data-connectors-csv</artifactId>
    <version>5.0.0</version>
</dependency>

This allows our application to compile while we work on our migration. For the time being, we can stop using the old DLC code we have, but can keep it for reference / testing purposes.

See the Imports section of the getting started guide for information on which Maven modules you need to include to import the new DLC.

Removal of IDataLoadController

The IDataLoadController interface was where we used to register Source, Channels, and Removal-Condition-Converters.

In the new DLC we now define configurations as Java Beans or in YAML.

The IDataLoadController.registerChannel method used to take a channel, targetStores and scopeToRemoveWhereConditionConverter. These components and how they are related and defined have changed:

  • channel: See the Channels section.
  • targetStores: This configuration is now part of the SinkDescription.
  • scopeToRemoveWhereConditionConverter: These conditions are replaced with new unload topics.

Source Migration

In the old DLC we had to configure Sources in Java. Now we can define sources as Java Beans or in YAML.

Sources used to be registered to the IDataLoadController directly,

Sources can now be individually defined through configuration in YAML or in Java via ISourceDescription Spring Beans.

CSV Source

Definition

CSV Sources used to be configured with ICsvSourceConfiguration java objects.

Now CSV Sources are defined via LocalCsvSourceDescription Beans in Java, or via Configuration Properties in YAML.

Channels

Channels used to be registered within the IDataLoadController along with target stores and an IScopeToRemoveWhereConditionConverter.

Now channels are defined via DLC SinkDescriptions and DLC ChannelDescriptions. These objects can be defined partially via YAML or fully in Java.

JDBC Source Definition

JDBC Sources used to be directly defined in Java via a JdbcScopedFetchSource instance.

Now JDBC Sources are defined via JdbcSourceDescription Beans in Java, or via Configuration Properties in YAML.

Topic Migration

Topics can now be defined through configuration in YAML or in Java via ITopicDescription Spring Beans.

Topics are no longer registered within a source, but instead defined on their own. This means a single CSV topic can be used across several CSV sources, e.g. Local, Azure, AWS, etc.

CSV Topics

CSV topics used to be defined with the following properties that have been moved into other configurations.

note

Note these particular changes:

  1. CsvFilesScopedFetchTopic.rootDirectoryPath: Value has been moved from being defined in the topic, to being defined in the source.
  2. CsvFilesScopedFetchTopic.fetchScopeToFileScanParametersConverter: The source definition now contains a rootBaseDir that contains the scope keys. See Local CSV Source Configuration Properties for details.

CSV Topic Property New Java Property Location New Configuration Property Location
CsvFilesScopedFetchTopic.name CsvTopicDescription.name CsvTopicProperties.name
1CsvFilesScopedFetchTopic.rootDirectoryPath Removed Removed
CsvFilesScopedFetchTopic.pathMatcher CsvTopicDescription.filePattern CsvTopicProperties.filePattern
2CsvFilesScopedFetchTopic.fetchScopeToFileScanParametersConverter LocalCsvSourceDescription.rootBaseDir LocalCsvSourceProperties.rootBaseDir
CsvFilesScopedFetchTopic.csvParserConfiguration CsvTopicDescription.parser CsvTopicProperties.parser

Requests

The Java API now has typed requests and responses, for example, executing a DlcLoadRequest will produce a DlcLoadResponse. These typed requests / responses have properties unique to that request type.

Operations

The operations which the DLC executes are no longer managed by the Atoti Server Registry, they are now IDlcOperation beans.

REST API

The REST API has changed slightly, and the new REST path can be found in the REST Endpoints section.

Aliases

Aliases can be defined in Java or via YAML configuration. Their functionality has not changed.