DLC Data Source Configuration

Configuring Data Sources

This section will help guide you through configuring Data Sources for your project.

Once you configure your source, you can complete configuring your controller by registering your source.

DLC Data Sources

The Data Load Controller allows you to load multiple types of files from the local file system or from the cloud. The DLC Supports two types of loading - Fetching and Messaging. Fetching will load the entire file in one go while Messaging is used for data sources that stream their data. CSV files can be loaded from the local filesystem (local CSV) or from a cloud container (Azure Blob, Google Cloud Platform, AWS S3 Bucket). Other supported sources are Avro, [RabbitMQ][rabbitmq] and Kafka.

Fetching Data Sources

Fetching Data Sources are sources that can be called upon to load in a single operation. When initiating a LOAD Operation on a Fetching Source, the DLC will load a particular file in a single request. Once the file is loaded it is safe to delete / move the file as it is no longer being used by the DLC. All Fetch Topic’s LOAD Operations execute completely and do not finish until the entire file has been processed.

Listening Data Sources

Listening allows for Listening on a Fetching Data Source. When initiating a START_LISTEN Operation on a Listening Source, the DLC will first load the file in its entirety (similarly to a Fetch Source) then keep a Listening channel open on that file. This allows for any modifications to the file to be picked up and the file to be re-loaded as changes are detected. This Listening channel will stay open until the STOP_LISTENING Operation is called for the particular Topic.

Messaging Data Sources

Messaging Data Sources are sources where a connection needs to be open between a messenger and consumer in order to get data from the source.