Definitions and concepts

Data Load Controller

The data load controller is designed to manage the loading (and unloading) of data in all accelerators.

Loading of data is expected to be triggered on start-up (“initial load”) and via a REST service by the external orchestration process used to provide the data to the accelerator–for example, the same process that copies files into a directory for the accelerator to load will also use the REST service to tell the Data Load Controller to load the data into the accelerator.

Scope

The scope describes which data is loaded (and unloaded). For example, the has an as-of date scope for all daily data; it would also be possible to add a “desk” parameter to the scope if you wanted to load data by desk.

Each request to the Data Load Controller contains a scope. In the reference implementation, the as-of date scope is turned into a directory and all the files in that directory are loaded. When unloading data, the scope is turned into a RemoveWhere clause to be run on the datastore.