# Atoti Hub: Atoti Solutions Data Load Controller ## Data Load Controller ### Data Load Controller 6.2 #### Data Load Controller (DLC) - [Changelog and release notes](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/changelog.md): Where to find release notes and changelog information for the Data Load Controller (DLC). - [Configure and start](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/getting-started.md): How to get started with the DLC, an Atoti Java SDK extension, by importing a source module (local CSV, JDBC, tuple, cloud CSV, Avro, Parquet, or Kafka), configuring sources and topics, and submitting load, unload, and listen requests. ##### Operations - [Operations](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/operations/overview.md): Reference for the `IDlcOperation` interface and the default DLC operations `LOAD`, `UNLOAD`, `START_LISTEN`, `LISTEN_STATUS`, and `STOP_LISTEN` triggered through the `DataLoadControllerService`. - [LOAD](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/operations/operations-load.md): Reference for the `LOAD` DLC operation, covering the `DlcLoadRequest` fields (topics, overrides, scope, branch, garbage collection) and the `DlcLoadResponse` and `LoadReport` returned. - [UNLOAD](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/operations/operations-unload.md): Reference for the `UNLOAD` DLC operation, covering the `DlcUnloadRequest` fields (topics, scope, branch, garbage collection) and the `DlcUnloadResponse` and `UnloadReport` returned. - [START_LISTEN](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/operations/operations-start-listen.md): Reference for the `START_LISTEN` DLC operation, covering the `DlcStartListenRequest` and `DlcStartListenResponse`, and how file source listeners trigger a `LOAD` on each detected file change. - [STOP_LISTEN](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/operations/operations-stop-listen.md): Reference for the `STOP_LISTEN` DLC operation, covering the `DlcStopListenRequest`, identified by `listenId`, and the `DlcStopListenResponse` returned when a listener is closed. - [LISTEN_STATUS](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/operations/operations-listen-status.md): Reference for the `LISTEN_STATUS` DLC operation, covering the `DlcListenStatusRequest`, the `DlcListenStatusResponse`, and the `ListenReport` returned for a listener identified by `listenId`. ##### Request API - [Request API](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/request-api/overview.md): Overview linking to the DLC request API reference: the DLC Request format, the DLC Response format, and the REST endpoints used to submit requests. - [DLC request](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/request-api/dlc-request.md): How to submit a DLC request via REST (`/connectors/rest/dlc/v2/execute`) or Java (`DataLoadControllerService.execute`), specifying an operation and its parameters. - [DLC response](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/request-api/dlc-response.md): What a DLC response returns after the `DataLoadControllerService` executes an operation, including status and per-operation results such as `LoadReport` and `UnloadReport`. - [REST endpoints](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/request-api/rest.md): Reference for the DLC REST API base path (`/connectors/rest/dlc/v2`) and the `execute` endpoint used to submit a DLC Request and receive a DLC Response. ##### Configuration - [Configuration](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/overview.md): Configuration reference for global DLC properties such as `enabled`, `create-default-cloud-client`, and `rollback-transaction-on-error`, defined under the `dlc` namespace. - [Aliases](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/aliases.md): How to configure `AliasesDescription` to group multiple topics under a single alias name, so a load or unload request can reference one alias instead of each individual topic. - [Channel](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/channel.md): Configuration reference for `ChannelDescription`, which links a topic to a `Target` and optional `Custom Fields`, with an `accumulate` option for chunked messages. - [Cloud fetching config](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/cloud-fetching-config.md): Configuration reference for `CloudFetchingConfig`, which tunes downloader thread count, part length, and prefetched parts limit for cloud sources such as the AWS CSV Source. - [Custom field](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/custom-field.md): Configuration reference for `CustomFieldDescription` and `AnonymousCustomFieldDescription`, Java objects that define a named or anonymous column calculator for use in a Channel. - [Parser overrides](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/parser-overrides.md): How to use `ParserOverridesDescription` to override the DLC's default CSV column parser type per store field, instead of relying on implicit column-to-field name matching. - [Target](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/target.md): Configuration reference for `TargetDescription` and `AnonymousTargetDescription`, Java objects that define the target store and tuple publisher factory for use in a Channel. - [Unload topic](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/unload-topic.md): How to configure `UnloadTopicDescription` to remove facts from a datastore table using a removal condition factory, including custom and implicit unload topics. ###### CSV - [CSV](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/csv/overview.md): Overview of CSV configuration for the DLC, covering the CSV Topic Description for file format and the Local, AWS, Azure, and GCP CSV Sources for file location. - [Local CSV source](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/csv/local-csv-source.md): Configuration reference for `LocalCsvSourceDescription`, which loads CSV files from a local root directory and restricts loading to files within that source. - [AWS CSV source](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/csv/aws-csv-source.md): Configuration reference for `AwsCsvSourceDescription`, which loads CSV files from an Amazon S3 bucket and prefix using an `S3Client` bean, with optional cloud fetching tuning. - [Azure CSV source](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/csv/azure-csv-source.md): Configuration reference for `AzureCsvSourceDescription`, which loads CSV files from an Azure Blob Storage container and prefix using a `BlobServiceClient` bean. - [GCP CSV source](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/csv/gcp-csv-source.md): Configuration reference for `GcpCsvSourceDescription`, which loads CSV files from a Google Cloud Storage bucket and prefix using a `Storage` bean. - [CSV loading topic](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/csv/csv-topic.md): Configuration reference for `CsvTopicDescription` properties such as `file-pattern`, `parser`, `channels`, and source restrictions, plus implicit CSV loading topics enabled by default. - [CSV parser](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/csv/csv-parser.md): Configuration reference for `CsvParserDescription` properties such as `charset`, `separator`, `number-skipped-lines`, `accept-overflowing-lines`, and `columns`, used inside a CSV Topic. - [CSV source configuration](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/csv/csv-source-config.md): Configuration reference for `CsvSourceConfiguration` properties such as `parser-threads`, `buffer-size`, `synchronous-mode`, and `accepts-topic-overrides`, shared across CSV sources. ###### JDBC - [JDBC](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/jdbc/overview.md): Overview of JDBC configuration for the DLC, covering the JDBC Topic Description's SQL statement and the JDBC Source's connection parameters. - [JDBC source](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/jdbc/jdbc-source.md): Configuration reference for `JdbcSourceDescription` properties such as `url`, `username`, `password`, and `driver-class`, defined under the `dlc.jdbc.sources` namespace. - [JDBC loading topic](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/jdbc/jdbc-topic.md): Configuration reference for `JdbcTopicDescription` properties such as `sql`, `parameterOrder`, `channels`, and source restrictions, defined under the `dlc.jdbc.topics` namespace. ###### Avro - [Avro](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/avro/overview.md): Overview of Avro configuration for the DLC, covering the Avro Topic Description for file format and the Local Avro Source for file location. - [Local Avro source](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/avro/local-avro-source.md): Configuration reference for `LocalAvroSourceDescription`, which loads Avro files from a local root directory and restricts loading to files within that source. - [Avro loading topic](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/avro/avro-topic.md): Configuration reference for `AvroTopicDescription` properties such as `file-pattern`, `schema-path`, `columns`, and `channels`, defined under the `dlc.avro.topics` namespace. ###### Parquet - [Parquet](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/parquet/overview.md): Overview of Parquet configuration for the DLC, covering the Parquet Topic Description for file format and the Local Parquet Source for file location. - [Local Parquet source](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/parquet/local-parquet-source.md): Configuration reference for `LocalParquetSourceDescription`, which loads Parquet files from a local root directory and restricts loading to files within that source. - [Parquet loading topic](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/parquet/parquet-topic.md): Configuration reference for `ParquetTopicDescription` properties such as `file-pattern`, `columns`, `source-store-mapping`, `channels`, and source restrictions, defined under the `dlc.parquet.topics` namespace. ###### Tuple - [Tuple](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/tuple/overview.md): Overview of Tuple configuration for the DLC, covering the Tuple Topic Description that generates data in Java and the Tuple Source that loads it. - [Tuple source](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/tuple/tuple-source.md): Configuration reference for `TupleSourceDescription` properties such as `topics-to-include` and `topics-to-exclude`, defined under the `dlc.tuple.sources` namespace. - [Tuple loading topic](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/tuple/tuple-topic.md): Configuration reference for `TupleTopicDescription` properties such as `tupleSupplier`, `columns`, `channels`, and source restrictions, configurable only through Java Spring beans. ###### Messaging - [Messaging](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/messaging/overview.md): Overview of messaging configuration for the DLC, covering Kafka message consumers, transaction controller tasks, messaging topics, and messaging sources. - [Kafka source](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/messaging/kafka-source.md): Configuration reference for `KafkaSourceDescription` properties such as `topics-to-include`, `topics-to-exclude`, and `accepts-topic-overrides`, defined under the `dlc.messaging.kafka` namespace. - [Kafka topic](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/configuration/messaging/kafka-topic.md): Configuration reference for `KafkaTopicDescription`, covering the broker topic, consumer properties, target stores, and performance tuning via `MessagingTopicConfigProperties` and `TxControllerTaskConfigProperties`. ##### Components - [Components](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/components/overview.md): Reference list linking to the DLC's components: its internal components, the `DataLoadControllerService` that executes requests, and the `NamedEntityResolverService` that resolves named entities. - [DataLoadControllerService](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/components/data-load-controller-service.md): How the `IDataLoadControllerService` executes an incoming `IDlcRequest` by resolving the target operation, opening a datastore transaction, processing the operation, and committing the result. - [NamedEntityResolverService](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/components/named-entity-resolver-service.md): How the `NamedEntityResolverService` retrieves named `CustomFieldDescription` and `TargetDescription` objects, including implicit targets created for each Atoti table. ###### Internal components - [Internal components](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/components/internal/overview.md): Reference list linking to the DLC's internal components: `DlcDescription`, `DlcLoadOperationsService`, the Source Fetcher, and the Source Handler, used to resolve sources and process load operations. - [DlcDescription](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/components/internal/dlc-description.md): Reference for `DlcDescription`, the configuration object holding aliases, source handlers, and topic descriptions that the `DataLoadControllerService` uses to process operations. - [DlcLoadOperationsService](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/components/internal/dlc-load-operations-service.md): How the `DlcLoadOperationsService` retrieves the Source Handler for a topic type and uses its Source Fetcher to perform the fetch operation used by the `LOAD` operation. - [Source fetcher](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/components/internal/source-fetcher.md): Reference for `ISourceFetcher`, which belongs to a Source Handler and converts a Source Description and Topic Descriptions into Atoti `ISource`, `ITopic`, and channel objects to run a fetch. - [Source handler](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/components/internal/source-handler.md): Reference for `ISourceHandler`, which holds the source descriptions, default source, topic-building logic, and Source Fetcher for one Atoti source type. ##### Functional - [Functional behavior](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/functional/overview.md): Overview linking to functional DLC topics, including configuration precedence, garbage collection triggers, scope parameters, and source to topic matching rules. - [Configuration flow](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/functional/configurations-flow.md): How DLC configuration precedence works across Java Spring bean descriptions, YAML configuration properties, and request-time loading topic overrides. - [Garbage collection](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/functional/garbage-collection.md): How to trigger a JVM garbage collection before or after a DLC operation using the `performGcOnStart` and `performGcOnCompletion` request properties. - [Scope parameters](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/functional/scope-parameters.md): How to use `DlcScope` key-value parameters in a DLC request to filter which files load, substitute placeholders in a source's base directory, and supply default values to column calculators and targets. - [Source to topic matching](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/functional/source-to-topic-matching.md): How `restrict-to-sources`, `restrict-from-sources`, `topics-to-include`, and `topics-to-exclude` control which topics can operate on which DLC sources. ##### Extensions - [Extensions](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/extensions/overview.md): Overview of DLC extension points for developers, linking to guides for adding custom `IDlcOperation` beans and enforcing custom topic processing order in the `LOAD` operation. - [Custom operations](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/extensions/custom-operations.md): How to override default DLC operations in `DefaultDlcOperationsConfig` or add a new `IDlcOperation` bean to extend the DLC with custom load, unload, or other request handling. - [Custom topic ordering](https://docs.activeviam.com/solutions/libraries/data-load-controller/6.2/extensions/custom-topic-ordering.md): How to override the `LOAD` operation to enforce a custom processing order for a list of topic sets extracted from the incoming DLC request. ### Data Load Controller (DLC) - [Changelog and release notes](https://docs.activeviam.com/solutions/libraries/data-load-controller/changelog.md): Where to find release notes and changelog information for the Data Load Controller (DLC). - [Configure and start](https://docs.activeviam.com/solutions/libraries/data-load-controller/getting-started.md): How to get started with the DLC, an Atoti Java SDK extension, by importing a source module (local CSV, JDBC, tuple, cloud CSV, Avro, Parquet, or Kafka), configuring sources and topics, and submitting load, unload, and listen requests. #### Operations - [Operations](https://docs.activeviam.com/solutions/libraries/data-load-controller/operations/overview.md): Reference for the `IDlcOperation` interface and the default DLC operations `LOAD`, `UNLOAD`, `START_LISTEN`, `LISTEN_STATUS`, and `STOP_LISTEN` triggered through the `DataLoadControllerService`. - [LOAD](https://docs.activeviam.com/solutions/libraries/data-load-controller/operations/operations-load.md): Reference for the `LOAD` DLC operation, covering the `DlcLoadRequest` fields (topics, overrides, scope, branch, garbage collection) and the `DlcLoadResponse` and `LoadReport` returned. - [UNLOAD](https://docs.activeviam.com/solutions/libraries/data-load-controller/operations/operations-unload.md): Reference for the `UNLOAD` DLC operation, covering the `DlcUnloadRequest` fields (topics, scope, branch, garbage collection) and the `DlcUnloadResponse` and `UnloadReport` returned. - [START_LISTEN](https://docs.activeviam.com/solutions/libraries/data-load-controller/operations/operations-start-listen.md): Reference for the `START_LISTEN` DLC operation, covering the `DlcStartListenRequest` and `DlcStartListenResponse`, and how file source listeners trigger a `LOAD` on each detected file change. - [STOP_LISTEN](https://docs.activeviam.com/solutions/libraries/data-load-controller/operations/operations-stop-listen.md): Reference for the `STOP_LISTEN` DLC operation, covering the `DlcStopListenRequest`, identified by `listenId`, and the `DlcStopListenResponse` returned when a listener is closed. - [LISTEN_STATUS](https://docs.activeviam.com/solutions/libraries/data-load-controller/operations/operations-listen-status.md): Reference for the `LISTEN_STATUS` DLC operation, covering the `DlcListenStatusRequest`, the `DlcListenStatusResponse`, and the `ListenReport` returned for a listener identified by `listenId`. #### Request API - [Request API](https://docs.activeviam.com/solutions/libraries/data-load-controller/request-api/overview.md): Overview linking to the DLC request API reference: the DLC Request format, the DLC Response format, and the REST endpoints used to submit requests. - [DLC request](https://docs.activeviam.com/solutions/libraries/data-load-controller/request-api/dlc-request.md): How to submit a DLC request via REST (`/connectors/rest/dlc/v2/execute`) or Java (`DataLoadControllerService.execute`), specifying an operation and its parameters. - [DLC response](https://docs.activeviam.com/solutions/libraries/data-load-controller/request-api/dlc-response.md): What a DLC response returns after the `DataLoadControllerService` executes an operation, including status and per-operation results such as `LoadReport` and `UnloadReport`. - [REST endpoints](https://docs.activeviam.com/solutions/libraries/data-load-controller/request-api/rest.md): Reference for the DLC REST API base path (`/connectors/rest/dlc/v2`) and the `execute` endpoint used to submit a DLC Request and receive a DLC Response. #### Configuration - [Configuration](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/overview.md): Configuration reference for global DLC properties such as `enabled`, `create-default-cloud-client`, and `rollback-transaction-on-error`, defined under the `dlc` namespace. - [Aliases](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/aliases.md): How to configure `AliasesDescription` to group multiple topics under a single alias name, so a load or unload request can reference one alias instead of each individual topic. - [Channel](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/channel.md): Configuration reference for `ChannelDescription`, which links a topic to a `Target` and optional `Custom Fields`, with an `accumulate` option for chunked messages. - [Cloud fetching config](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/cloud-fetching-config.md): Configuration reference for `CloudFetchingConfig`, which tunes downloader thread count, part length, and prefetched parts limit for cloud sources such as the AWS CSV Source. - [Custom field](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/custom-field.md): Configuration reference for `CustomFieldDescription` and `AnonymousCustomFieldDescription`, Java objects that define a named or anonymous column calculator for use in a Channel. - [Parser overrides](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/parser-overrides.md): How to use `ParserOverridesDescription` to override the DLC's default CSV column parser type per store field, instead of relying on implicit column-to-field name matching. - [Target](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/target.md): Configuration reference for `TargetDescription` and `AnonymousTargetDescription`, Java objects that define the target store and tuple publisher factory for use in a Channel. - [Unload topic](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/unload-topic.md): How to configure `UnloadTopicDescription` to remove facts from a datastore table using a removal condition factory, including custom and implicit unload topics. ##### CSV - [CSV](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/csv/overview.md): Overview of CSV configuration for the DLC, covering the CSV Topic Description for file format and the Local, AWS, Azure, and GCP CSV Sources for file location. - [Local CSV source](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/csv/local-csv-source.md): Configuration reference for `LocalCsvSourceDescription`, which loads CSV files from a local root directory and restricts loading to files within that source. - [AWS CSV source](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/csv/aws-csv-source.md): Configuration reference for `AwsCsvSourceDescription`, which loads CSV files from an Amazon S3 bucket and prefix using an `S3Client` bean, with optional cloud fetching tuning. - [Azure CSV source](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/csv/azure-csv-source.md): Configuration reference for `AzureCsvSourceDescription`, which loads CSV files from an Azure Blob Storage container and prefix using a `BlobServiceClient` bean. - [GCP CSV source](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/csv/gcp-csv-source.md): Configuration reference for `GcpCsvSourceDescription`, which loads CSV files from a Google Cloud Storage bucket and prefix using a `Storage` bean. - [CSV loading topic](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/csv/csv-topic.md): Configuration reference for `CsvTopicDescription` properties such as `file-pattern`, `parser`, `channels`, and source restrictions, plus implicit CSV loading topics enabled by default. - [CSV parser](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/csv/csv-parser.md): Configuration reference for `CsvParserDescription` properties such as `charset`, `separator`, `number-skipped-lines`, `accept-overflowing-lines`, and `columns`, used inside a CSV Topic. - [CSV source configuration](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/csv/csv-source-config.md): Configuration reference for `CsvSourceConfiguration` properties such as `parser-threads`, `buffer-size`, `synchronous-mode`, and `accepts-topic-overrides`, shared across CSV sources. ##### JDBC - [JDBC](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/jdbc/overview.md): Overview of JDBC configuration for the DLC, covering the JDBC Topic Description's SQL statement and the JDBC Source's connection parameters. - [JDBC source](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/jdbc/jdbc-source.md): Configuration reference for `JdbcSourceDescription` properties such as `url`, `username`, `password`, and `driver-class`, defined under the `dlc.jdbc.sources` namespace. - [JDBC loading topic](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/jdbc/jdbc-topic.md): Configuration reference for `JdbcTopicDescription` properties such as `sql`, `parameterOrder`, `channels`, and source restrictions, defined under the `dlc.jdbc.topics` namespace. ##### Avro - [Avro](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/avro/overview.md): Overview of Avro configuration for the DLC, covering the Avro Topic Description for file format and the Local Avro Source for file location. - [Local Avro source](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/avro/local-avro-source.md): Configuration reference for `LocalAvroSourceDescription`, which loads Avro files from a local root directory and restricts loading to files within that source. - [Avro loading topic](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/avro/avro-topic.md): Configuration reference for `AvroTopicDescription` properties such as `file-pattern`, `schema-path`, `columns`, and `channels`, defined under the `dlc.avro.topics` namespace. ##### Parquet - [Parquet](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/parquet/overview.md): Overview of Parquet configuration for the DLC, covering the Parquet Topic Description for file format and the Local Parquet Source for file location. - [Local Parquet source](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/parquet/local-parquet-source.md): Configuration reference for `LocalParquetSourceDescription`, which loads Parquet files from a local root directory and restricts loading to files within that source. - [Parquet loading topic](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/parquet/parquet-topic.md): Configuration reference for `ParquetTopicDescription` properties such as `file-pattern`, `columns`, `source-store-mapping`, `channels`, and source restrictions, defined under the `dlc.parquet.topics` namespace. ##### Tuple - [Tuple](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/tuple/overview.md): Overview of Tuple configuration for the DLC, covering the Tuple Topic Description that generates data in Java and the Tuple Source that loads it. - [Tuple source](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/tuple/tuple-source.md): Configuration reference for `TupleSourceDescription` properties such as `topics-to-include` and `topics-to-exclude`, defined under the `dlc.tuple.sources` namespace. - [Tuple loading topic](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/tuple/tuple-topic.md): Configuration reference for `TupleTopicDescription` properties such as `tupleSupplier`, `columns`, `channels`, and source restrictions, configurable only through Java Spring beans. ##### Messaging - [Messaging](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/messaging/overview.md): Overview of messaging configuration for the DLC, covering Kafka message consumers, transaction controller tasks, messaging topics, and messaging sources. - [Kafka source](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/messaging/kafka-source.md): Configuration reference for `KafkaSourceDescription` properties such as `topics-to-include`, `topics-to-exclude`, and `accepts-topic-overrides`, defined under the `dlc.messaging.kafka` namespace. - [Kafka topic](https://docs.activeviam.com/solutions/libraries/data-load-controller/configuration/messaging/kafka-topic.md): Configuration reference for `KafkaTopicDescription`, covering the broker topic, consumer properties, target stores, and performance tuning via `MessagingTopicConfigProperties` and `TxControllerTaskConfigProperties`. #### Components - [Components](https://docs.activeviam.com/solutions/libraries/data-load-controller/components/overview.md): Reference list linking to the DLC's components: its internal components, the `DataLoadControllerService` that executes requests, and the `NamedEntityResolverService` that resolves named entities. - [DataLoadControllerService](https://docs.activeviam.com/solutions/libraries/data-load-controller/components/data-load-controller-service.md): How the `IDataLoadControllerService` executes an incoming `IDlcRequest` by resolving the target operation, opening a datastore transaction, processing the operation, and committing the result. - [NamedEntityResolverService](https://docs.activeviam.com/solutions/libraries/data-load-controller/components/named-entity-resolver-service.md): How the `NamedEntityResolverService` retrieves named `CustomFieldDescription` and `TargetDescription` objects, including implicit targets created for each Atoti table. ##### Internal components - [Internal components](https://docs.activeviam.com/solutions/libraries/data-load-controller/components/internal/overview.md): Reference list linking to the DLC's internal components: `DlcDescription`, `DlcLoadOperationsService`, the Source Fetcher, and the Source Handler, used to resolve sources and process load operations. - [DlcDescription](https://docs.activeviam.com/solutions/libraries/data-load-controller/components/internal/dlc-description.md): Reference for `DlcDescription`, the configuration object holding aliases, source handlers, and topic descriptions that the `DataLoadControllerService` uses to process operations. - [DlcLoadOperationsService](https://docs.activeviam.com/solutions/libraries/data-load-controller/components/internal/dlc-load-operations-service.md): How the `DlcLoadOperationsService` retrieves the Source Handler for a topic type and uses its Source Fetcher to perform the fetch operation used by the `LOAD` operation. - [Source fetcher](https://docs.activeviam.com/solutions/libraries/data-load-controller/components/internal/source-fetcher.md): Reference for `ISourceFetcher`, which belongs to a Source Handler and converts a Source Description and Topic Descriptions into Atoti `ISource`, `ITopic`, and channel objects to run a fetch. - [Source handler](https://docs.activeviam.com/solutions/libraries/data-load-controller/components/internal/source-handler.md): Reference for `ISourceHandler`, which holds the source descriptions, default source, topic-building logic, and Source Fetcher for one Atoti source type. #### Functional - [Functional behavior](https://docs.activeviam.com/solutions/libraries/data-load-controller/functional/overview.md): Overview linking to functional DLC topics, including configuration precedence, garbage collection triggers, scope parameters, and source to topic matching rules. - [Configuration flow](https://docs.activeviam.com/solutions/libraries/data-load-controller/functional/configurations-flow.md): How DLC configuration precedence works across Java Spring bean descriptions, YAML configuration properties, and request-time loading topic overrides. - [Garbage collection](https://docs.activeviam.com/solutions/libraries/data-load-controller/functional/garbage-collection.md): How to trigger a JVM garbage collection before or after a DLC operation using the `performGcOnStart` and `performGcOnCompletion` request properties. - [Scope parameters](https://docs.activeviam.com/solutions/libraries/data-load-controller/functional/scope-parameters.md): How to use `DlcScope` key-value parameters in a DLC request to filter which files load, substitute placeholders in a source's base directory, and supply default values to column calculators and targets. - [Source to topic matching](https://docs.activeviam.com/solutions/libraries/data-load-controller/functional/source-to-topic-matching.md): How `restrict-to-sources`, `restrict-from-sources`, `topics-to-include`, and `topics-to-exclude` control which topics can operate on which DLC sources. #### Extensions - [Extensions](https://docs.activeviam.com/solutions/libraries/data-load-controller/extensions/overview.md): Overview of DLC extension points for developers, linking to guides for adding custom `IDlcOperation` beans and enforcing custom topic processing order in the `LOAD` operation. - [Custom operations](https://docs.activeviam.com/solutions/libraries/data-load-controller/extensions/custom-operations.md): How to override default DLC operations in `DefaultDlcOperationsConfig` or add a new `IDlcOperation` bean to extend the DLC with custom load, unload, or other request handling. - [Custom topic ordering](https://docs.activeviam.com/solutions/libraries/data-load-controller/extensions/custom-topic-ordering.md): How to override the `LOAD` operation to enforce a custom processing order for a list of topic sets extracted from the incoming DLC request.