Migration Notes - 5.0

This page explains the changes required to migrate to the stated version of Atoti Data Connectors.

Migrate to 5.0.3

No migration needed.

This tool is using Atoti Server 6.1.6. For new features and fixes included in this Atoti Server release, please see the Atoti Server 6.1.6 Changelog.

Headline announcement

• Compiled against Atoti Server 6.1.6: Compiled against updated Atoti Server version to resolve breaking changes impacting the data-extraction-engine module.

Changes

• Support for SQL override for JDBC Topics: JDBC topics now allow for overriding of their SQL queries.

JDBC Query Overrides

JDBC topics now allow overriding of their SQL queries at request time. This works in the same way as overriding the filepath for CSV topics.

To aid in creating JDBC override topic descriptions, the SQL parameter of JdbcTopicDescriptionis no longer required to be non-null.

Migrate to 5.0.1

No migration needed.

Upgrading from version 5.0.0, see Atoti Data Connectors 5.0 Release Notes

This tool is using Atoti Server 6.1.4. For new features and fixes included in this Atoti Server release, please see the Atoti Server6.1.4 Changelog.

Headline announcement

• Compiled against Atoti Server 6.1.4: Compiled against updated Atoti Server version to resolve breaking change issues impacting Tuple, Kafka & Avro sources.
• DLC no longer throws ActiveViamRuntimeException The DLC will now throw IllegalArgumentException and DataLoadControllerException rather than ActiveViamRuntimeExceptions. This allows clients to implement better error handling on their side.

Migrate to 5.0.0

Upgrading from version 4.2.0-AS6.1, see Atoti Data Connectors 5.0 Release Notes

Overview

The DLC has been completely re-written. For more information, see the Release Notes and our step guide to migrating to the new DLC API

This tool is compatible with Atoti Server 6.1.1.

For new features and fixes included in this Atoti Server release, please see the Atoti Server6.1.1 Changelog.

Architectural Changes

Description objects

• Previously, you had to register all Sources, Channels, Topics, Converters, etc. within the Data Load Controller itself. These registered objects were wrappers around core Atoti objects on which actions (such as load) would be directly performed.
  In the new DLC, the DLC environment is built of Description objects. These description objects can be defined in Java as Spring beans or via configuration properties. The properties can be used to override / customize the Java beans to allow for different execution behavior.
  
  

• The DLC’s description objects will be converted to Atoti core objects at request time. This means that DLC Topic & Source definitions will be converted into Atoti Source & Topic instances during a DLC request. The Atoti instances will not be persisted (except when a listening pipe is opened).

Services

• What used to be the DataLoadController is now the DataLoadControllerService. The DataLoadControllerService is responsible for handling the requests which come to the DLC.

• The logic for handling a specific operation used to be handled by registered implementations of ADatastoreTransactionTask. This is now an operation bean.

• The DLC now makes inferences where possible, so the minimal configuration is drastically reduced.

Dependency Changes

Import the new modules outlined in the Getting Started guide.

Configuration Classes

This version of the DLC uses Spring Auto Configuration to load required configuration classes.

Current limitations

Operations

The DLC supports LOAD, UNLOAD, START_LISTEN, LISTEN_STATUS and STOP_LISTEN operations.

Sources

This release supports the following sources:

• CSV variants: Local CSV, AWS CSV, Azure CSV, and GCP CSV
• JDBC, Tuple, Avro, Parquet, and Kafka.

Configuration files

We have added the ability to configure the DLC through Configuration Properties.

Configuration in Java

The Java Configuration is now done via Spring Beans.

DEE 5.0.0 Migration

The DEE has not been altered in this release of Atoti Data Connectors, therefore no migration is needed.

DLC 5.0.0 Migration

The rewritten DLC in Atoti Data Connectors 5.0 requires significant steps to migrate to the new DLC API.

We outline these steps in the following sections:

1. Dependency Management
2. Remove IDataLoadController
3. Source migration
4. Topic migration
5. Requests
6. Operations
7. REST API

Dependency Management

See the Imports section of the getting started guide for information on which Maven modules you need to include to import the new DLC.

Removal of IDataLoadController

The IDataLoadController interface was where we used to register Source, Channels, and Removal-Condition-Converters.

In the new DLC we now define configurations as Java Beans or in YAML.

The IDataLoadController.registerChannel method used to take a channel, targetStores and scopeToRemoveWhereConditionConverter. These components and how they are related and defined have changed:

• channel: See the Channels section.
• targetStores: This configuration is now part of the TargetDescription.
• scopeToRemoveWhereConditionConverter: These conditions are replaced with new unload topics.

Source Migration

In the old DLC we had to configure Sources in Java. Now we can define sources as Java Beans or in YAML.

Sources used to be registered to the IDataLoadController directly,

Sources can now be individually defined through configuration in YAML or in Java via ISourceDescription Spring Beans.

CSV Source

Definition

CSV Sources used to be configured with ICsvSourceConfiguration java objects.

Now CSV Sources are defined via LocalCsvSourceDescription Beans in Java, or via Configuration Properties in YAML.

Channels

Channels used to be registered within the IDataLoadController along with target stores and an IScopeToRemoveWhereConditionConverter.

Now channels are defined via DLC TargetDescriptions and DLC ChannelDescriptions. These objects can be defined partially via YAML or fully in Java.

JDBC Source Definition

JDBC Sources used to be directly defined in Java via a JdbcScopedFetchSource instance.

Now JDBC Sources are defined via JdbcSourceDescription Beans in Java, or via Configuration Properties in YAML.

Messaging Source

Messaging Sources used to be configured with IListeningDataSource java objects.

Now the Messaging Sources are defined via IMessagingSourceDescription java objects

Topic Migration

Topics can now be defined through configuration in YAML or in Java via ITopicDescription Spring Beans.

Topics are no longer registered within a source, but instead defined on their own. This means a single CSV topic can be used across several CSV sources, e.g. Local, Azure, AWS, etc.

CSV Topics

CSV topics used to be defined with the following properties that have been moved into other configurations.

CSV Topic Property	New Java Property Location	New Configuration Property Location
CsvFilesScopedFetchTopic.name	CsvTopicDescription.name	CsvTopicProperties.name
1CsvFilesScopedFetchTopic.rootDirectoryPath	Removed	Removed
CsvFilesScopedFetchTopic.pathMatcher	CsvTopicDescription.filePattern	CsvTopicProperties.filePattern
2CsvFilesScopedFetchTopic.fetchScopeToFileScanParametersConverter	LocalCsvSourceDescription.rootBaseDir	LocalCsvSourceProperties.rootBaseDir
CsvFilesScopedFetchTopic.csvParserConfiguration	CsvTopicDescription.parser	CsvTopicProperties.parser

Requests

The Java API now has typed requests and responses, for example, executing a DlcLoadRequest will produce a DlcLoadResponse. These typed requests / responses have properties unique to that request type.

Operations

The operations which the DLC executes are no longer managed by the Atoti Server Registry, they are now IDlcOperation beans.

REST API

The REST API has changed slightly, and the new REST path can be found in the REST Endpoints section.

Aliases

Aliases can be defined in Java or via YAML configuration. Their functionality has not changed.