Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.activeviam.com/llms.txt

Use this file to discover all available pages before exploring further.

The Data Load Controller lets you load and unload data without worrying about the underlying implementation details.
For the Data Load Controller documentation, click here.

​
Overview

The Data Load Controller supports managing data within . It provides an interface between the ETL (Extract, Transform Load) layer and the management of what data is loaded into . There are three main parameters in this interface (all of which are configurable):
  • Operation: the operation the DLC performs. Typically, this is load or unload.
  • Topics: the list of data sources defined in the ETL for which to perform the operation. Topic aliases are used to easily specify groups of topics.
  • Scope: a filter on the data to be loaded. For example, to load/unload data for a particular as-of date.
 

​
Initial load

Starting up the application triggers the initial loading of data. This is managed programmatically within the Solution, with Spring beans initiating the loading of topics and scopes at startup. For more information, see Initial Load.

​
Data Orchestrator

Beyond the initial data load, the data orchestrator is a custom external component that is responsible for managing (loading and unloading) data in . For example, the same process that copies the data ready for the Solution to load can also use the REST service to tell the Data Load Controller to trigger the loading of the data into the Solution.

​
REST Service

The DLC provides a RESTservice, which the data orchestrator or other external components can use to manage the loading and unloading of data in the Solution. The DLC REST API has two endpoints:
  • execute: for loading and unloading data.
  • status: to check on a previous command.
REST URL for execute: 
http://<hostname>:<port>/frtb-starter/connectors/rest/dlc/execute
The FRTBDataLoadControllerRestController and its rest service /frtb-starter/services/rest/dlc/execute are deprecated and will be removed in future versions. Use the new REST service /frtb-starter/connectors/rest/dlc/v1/execute instead.

​
Execute

For execute, requests use POST and send JSON objects with the following fields: Operation: ‘load’ or ‘unload’.
  • load: load the data according to topic and scope.
  • unload: delete data from the datastore according to topic and scope. This doesn’t depend on how or when the data was loaded.
This doesn’t delete reference data that was added as part of the data normalization. For example, if you delete sensitivities, the corresponding risk-factors are not removed.
Topics: a list of topics to load or unload. In addition to the topic for each file type (see Input file formats), aliases are defined to load sets of files. For a list of FRTB-specific topic aliases, see Topic Aliases. Scope: three scopes are defined in : as-of date, configuration, and historical. For more information on these, see Scopes. This example snippet loads FXData for the 27th of September, 2018. 
{  
"operation": "LOAD",  
"topics": ["FXData"],  
"scope": {  
"AsOfDate": "2018-09-27",  
}  
}
This example snippet removes the 27th of September, 2018 from the SA and IMA cubes: 
{  
"operation":"unload",  
"topics":[  
"ALLSA", "ALLIMADaily"  
],  
"scope":{  
"AsOfDate":"2018-09-27"  
}
}

​
Loading data into a branch

You can add the ‘Branch’ keyword to any scope section to specify a branch in the cube for data update. On the ‘load’ feature, it is used to find the resource to load (mainly the path for the file). On the ‘unload’ feature, it is used to build the remove criteria (Condition). For example, here’s how to load new FXData on the “test” branch: 
{  
"operation": "LOAD",  
"topics":[  
"FXData"  
],  
"scope": {  
"AsOfDate": "2018-09-28",  
"Branch": "test"  
}  
}

​
Status

The status endpoint can be used to query the status of a request. The Processing ID (also called Transaction ID) for a particular load is returned as part of the response. The status is already part of the load/unload response, but if you want to check it again, you can make the following status call through GET: 
REST URL:
http://<hostname>:<port>/frtb-starter/connectors/rest/dlc/v1/status?processingId=\<Processing ID/>