Getting started with the Data Load Controller

The Data Load Controller lets you load and unload data without worrying about the underlying implementation details.

Overview

The data load controller supports managing data within the accelerator. It provides an interface between the ETL (Extract, Transform Load) layer and the management of what data is loaded into the accelerator.

There are three main parameters in this interface (all of which are configurable):

• Operation: the operation the DLC performs. Typically, this is load or unload.
• Topics: the list of data sources defined in the ETL for which to perform the operation. Topic aliases are used to easily specify groups of topics.
• Scope: a filter on the data to be loaded. For example, to load/unload data for a particular as-of date.

Initial load

Starting up the application triggers the initial loading of data. This is managed programmatically within the accelerator, with Spring beans initiating the loading of topics and scopes at startup.

For more information, see Initial Load.

Data Orchestrator

Beyond the initial data load, the data orchestrator is a custom external component that is responsible for managing (loading and unloading) data in the accelerator.

For example, the same process that copies the data ready for the accelerator to load can also use the REST service to tell the Data Load Controller to trigger the loading of the data into the accelerator.

REST Service

The DLC provides a RESTservice, which the data orchestrator or other external components can use to manage the loading and unloading of data in the accelerator.

The DLC REST API has two endpoints:

• execute: for loading and unloading data.
• status: to check on a previous command.

REST URL for execute:

http://<hostname>:<port>/frtb-starter/services/rest/dlc/execute

Execute

For execute, requests use POST and send JSON objects with the following fields:

Operation: ‘load’ or ‘unload’.

• load: load the data according to topic and scope.
• unload: delete data from the datastore according to topic and scope. This doesn’t depend on how or when the data was loaded.

Topics: a list of topics to load or unload. In addition to the topic for each file type (see Input file formats), aliases are defined to load sets of files. For a list of FRTB-specific topic aliases, see Topic Aliases.

Scope: three scopes are defined in the accelerator: as-of date, configuration, and historical. For more information on these, see Scopes.

This example snippet loads FXData for the 27th of September, 2018.

{  
"operation": "LOAD",  
"topics": ["FXData"],  
"scope": {  
"AsOfDate": "2018-09-27",  
}  
}

This example snippet removes the 27th of September, 2018 from the SA and IMA cubes:

{  
"operation":"unload",  
"topics":[  
"ALLSA", "ALLIMADaily"  
],  
"scope":{  
"AsOfDate":"2018-09-27"  
}
}

Loading data into a branch

You can add the ‘Branch’ keyword to any scope section to specify a branch in the cube for data update. On the ‘load’ feature, it is used to find the resource to load (mainly the path for the file). On the ‘unload’ feature, it is used to build the remove criteria (Condition).

For example, here’s how to load new FXData on the “test” branch:

{  
"operation": "LOAD",  
"topics":[  
"FXData"  
],  
"scope": {  
"AsOfDate": "2018-09-28",  
"Branch": "test"  
}  
}

Status

The status endpoint can be used to query the status of a request.

The Processing ID (also called Transaction ID) for a particular load is returned as part of the response. The status is already part of the load/unload response, but if you want to check it again, you can make the following status call through GET:

REST URL:
http://<hostname>:<port>/frtb-starter/services/rest/dlc/status?processingId=\<Processing ID/>