☰
Navigation :

LOAD

Overview

The LOAD operation performs a single, blocking, load request in a single datastore transaction.

Request

DlcLoadRequest

Key Required Type Description
topics Y Set<String> Set of topics or aliases.
topicOverrides Set<ILoadingTopicDescription> Topic overrides.
scope IDlcScope Scope Parameters
branch String The branch you want to operate on.
performGcOnStart Boolean Perform a Garbage collection before the datastore transaction.
performGcOnCompletion Boolean Perform a Garbage collection after the datastore transaction.
sourceName String The source name you want to operate on. See source to topic matching.
sourceType DlcSourceType The source type you want to operate on. See source to topic matching.

Example:

DlcLoadRequest.builder()
    .topics("Topic1", "Topic2") // Required
    .topicOverrides(Set.of(CsvTopicDescription.builder("TopicToOverride", "glob:file*.csv")...build()))
    .scope(DlcScope.of("ScopeKey1", "ScopeValue1"))
    .branch("BranchToLoadInto")
    .sourceName("NameOfSourceToUse")
    .sourceType(DlcSourceType.LOCAL_SOURCE)
    .build();

DlcLoadRequestDTO

Key Required Type Description
topics Y Set<String> Set of topics or aliases.
csvTopicOverrides Map<String, CsvTopicProperties> CSV Topic overrides.
jdbcTopicOverrides Map<String, JdbcTopicProperties> JDBC Topic overrides.
scope Map<String, Object> Scope Parameters
branch String The branch you want to operate on.
performGcOnStart Boolean Perform a Garbage collection before the datastore transaction.
performGcOnCompletion Boolean Perform a Garbage collection after the datastore transaction.
sourceName String The source name you want to operate on. See source to topic matching.
sourceType DlcSourceType The source type you want to operate on. See source to topic matching.

Example:

{
  "operation": "LOAD",
  "topics": ["Topic1", "Topic2"],
  "csvTopicOverrides": {
    "TopicToOverride": {
      "path": "glob:file*.csv"
    }
  },
  "scope": {
    "ScopeKey1": "ScopeValue1"
  },
  "branch": "BranchToLoadInto",
  "performGcOnStart": true,
  "performGcOnCompletion": true,
  "sourceName": "NameOfSourceToUse",
  "sourceType": "LOCAL_SOURCE"
}

Response

DlcLoadResponse

Key Type Description
report LoadReport Report of what occurred during the loading operation.
status DlcStatus Status of the DLC Operation.

LoadReport

Key Type Description
parsingReport Map<String, Map<String, ITopicParsingReport>> Per source and per topic parsing report provided by the Atoti API. Will contain fine-grain details. Needs to be enabled either in the Source Configuration or through global property activeviam.msg.reporting.enabled
errors Map<String, List<DlcMessageCollector.OnError>> Errors that occurred while loading.
messagesPublished Map<String, List<DlcMessageCollector.OnMessagePublished>> The Atoti message when the records are published to the datastore.
messageStarted Map<String, List<DlcMessageCollector.OnMessageStarted>> The Atoti message when the records are sent to the Tuple Publisher.

DlcStatus

A simple enum to capture if the DLC request was executed.

Key Description
OK Request was handled successfully.
ERROR Request was not able to be performed correctly.

DlcLoadResponseDTO

Key Type Description
parsingReport Map<String, Map<String, ITopicParsingReport>> Per source and per topic parsing report provided by the Atoti API. Will contain fine-grain details. Needs to be enabled either in the Source Configuration or through global property activeviam.msg.reporting.enabled
errors Map<String, List<DlcMessageCollector.OnError>> Errors that occurred while loading.
messagesPublished Map<String, List<DlcMessageCollector.OnMessagePublished>> The Atoti message when the records are published to the datastore.
messageStarted Map<String, List<DlcMessageCollector.OnMessageStarted>> The Atoti message when the records are sent to the Tuple Publisher.
status DlcStatus Status of the DLC Operation.

Example:

{
  "parsingReport": {},
  "errors": {},
  "messagesPublished": {
    "sourceName": [
      {
        "key": "FileSystemFileInfo [fullName=root/file.csv, lastModifiedTime=Tue Feb 25 16:14:22 EST 2025]",
        "info": {
          "publishedObjectCount": 30,
          "characterCount": 6342,
          "lineCount": 31,
          "byteCount": 6342,
          "elapsedTime": 0.021108000,
          "aggregatedTime": 15301750,
          "aggregatedReadingTime": 3976791,
          "aggregatedDecodingTime": 78834,
          "aggregatedParsingTime": 702750,
          "aggregatedPublicationTime": 10543375,
          "exceptions": [],
          "creationTime": "2025-03-18T00:59:18.449823Z",
          "numberSkippedLines": 1
        }
      }
    ]
  },
  "messageStarted": {
    "sourceName": [
      {
        "key": "FileSystemFileInfo [fullName=root/file.csv, lastModifiedTime=Tue Feb 25 16:14:22 EST 2025]"
      }
    ]
  },
  "status": "OK"
}