Data Load Controller (Deprecated)

How to Get Started

The DLC is provided as a library that can be added as a maven dependency. It is configured via Spring Configuration as described in Configuring the DLC.

If you are looking for how to interact with DLC to load / unload data, start / stop messaging consumers or check the status of a request, please refer to the User Guide.

What Is The Data Load Controller?

The Data Load Controller (DLC) is a component within Atoti Server that sits on top of whatever target sources we may want to use (e.g. CSV source, JDBC source, custom sources), and allows loading and unloading of data for particular ‘source topics’ in a consolidated way.

From DLC’s perspective one just tells, for instance, “I want to fetch data for topic A & B” (and specifying a particular scope of fetching, e.g. for a particular COB date) without caring about the underlying implementation details, such as which source is actually responsible for the fetching of topic A or topic B.

Moreover, DLC is not only a local component but also exposed as a web-service by Atoti Server. This is quite fundamental, in fact, because it allows to keep “data orchestration” (loading/unloading) outside of Atoti Server. Atoti Server is a computational framework, and not a data orchestration framework.

So, an external data orchestrator component, such as another application or just a collection of Control-M jobs, etc., will be developed and maintained by customers outside Atoti Server, and Atoti Server simply gives ways for that orchestrator to tell it what to load/unload.

Diagram showing where the DLC lives in respect to Atoti Data Connectors and Atoti Server Application:

Source and Topics

The DLC is made up of Sources and Topics. Sources define where the data is coming from and how to interact with that data source. Topics define what data to load from that source. A Source contains multiple Topics. Topics are globally unique as two sources cannot contain the same Topic.

Source Types

The Data Load Controller allows you to load multiple types of files from the local file system or from the cloud. The DLC Supports two types of data sources - Fetching and Messaging. Fetching will load the entire file in one go while Messaging is used for data sources that stream their data.

Fetching Data Source

Fetching Data Sources are sources that can be called upon to load in a single operation. When initiating a LOAD Operation on a Fetching Source, the DLC will load a particular file in a single request. Once the file is loaded it is safe to delete / move the file as it is no longer being used by the DLC. All Fetch Topic’s LOAD Operations execute completely and do not finish until the entire file has been processed.

Our fetching data sources consist of:

• CSV Files:
  • Local Filesystem
  • AWS S3 Bucket
  • Azure Blob Container
  • Google Cloud Platform Container
• Avro
• JDBC
• Parquet

Listening on Fetch Sources

It is possible to Listen on Fetch Source. For information on how this works, see the Listen-on-Fetch section.

Messaging Data Sources

Messaging Data Sources are sources where a connection needs to be open between a messenger and consumer in order to get data from the source. A message channel is opened when a START_LISTEN Operation is executed. The messaging channel will remain open and continue to read / poll new data until a STOP_LISTEN Operation is executed, closing the messaging channel.

Our messaging data sources consist of:

• Kafka
• RabbitMq

Event Monitoring

As part of the data loading/unloading phase, clients can monitor the status of the load, create monitoring dashboards, which will enable them to track events like the start/end times and exceptions/failures if any. This section is mainly aimed at developers looking to tap into the event handling mechanism that has been built into the DLC.

Basic events, like task summary, task completion, task failure, and event handlers for the supported data sources have been provided as part of the DLC. These event handlers write the corresponding events to the DLC cache.