☰
Navigation :

User guide

Overview

The py_dlc library can expose REST services to remotely initiate loading and unloading data in an Atoti Cube. We configure the DataLoadController by defining Topics and Scopes.

A Topic defines a collection of data to load while a scope limits where to gather that set of data from.

A Topic could be trade*.csv files, and the scope can be a specific AsOfDate.

Walk-through Examples

  • A Simple tutorial with no scope can be found here.
  • A Tutorial using scope can be found here.

Importing py_dlc

We can import the library and all required libraries into our python project as follows:

import atoti
from atoti import Session

import py_dlc
from py_dlc import Scope
from typing import Optional
Import Description
import atoti The actual Atoti import is needed.
import py_dlc The DLC library its self.
from atoti import Session The Session is used as an input type in the DLC’s callback operations. This allows our Operation method signature to be clean and consistent.
from py_dlc import Scope The Scope is used to narrow down a DLC’s operation. This class is needed as our Operation method is typed.
from typing import Optional The Optional is used within our Scope as Scopes can be “optionally” empty. This class is needed as our Operation method is typed.

REST Services

The REST services will be created with the following endpoints:

POST /execute

Executes an operation against the DLC. The PORT can be resolved by reading the session.port variable.

http://localhost:<PORT>/atoti/pyapi/load-controller/execute

The /execute endpoint takes a POST request with a JSON object with the following structure:

{
    "operation": "OPERATION_TYPE",
    "topics": ["TOPIC_1", "TOPIC_2", ...],
    "scope" :{
      "SCOPE_KEY" : "SCOPE_VALUE"
    }
}

This will return JSON with the following format:

{
    "Task Name": "TaskNameHexString",
    "Time Taken (MS)": 2569,
    "Status": "SUCCESS",
    "Events": [
        {
            "Data Root": "file:/C:/Users/YourName/AppData/Local/Temp/atoti-zj1dg1gy/tmpwhbybivj.parquet",
            "Source": "PARQUET",
            "Lines Loaded": 2,
            "Duration (MS)": 452,
            "Errors": 0
        }
    ]
}

GET /status

Retrieve the past operation status for a given Task Name.

http://localhost:<PORT>/atoti/pyapi/load-controller/status

We will pass our Task Name to the /status endpoint through the query parameter processingId:

http://localhost:<PORT>/atoti/pyapi/load-controller/status?processingId=TaskNameHexString

This will give us the following JSON Response:

{
    "Task Name": "TaskNameHexString",
    "Time Taken (MS)": 2569,
    "Status": "SUCCESS",
    "Events": [
        {
            "Data Root": "file:/C:/Users/YourName/AppData/Local/Temp/atoti-zj1dg1gy/tmpwhbybivj.parquet",
            "Source": "PARQUET",
            "Lines Loaded": 2,
            "Duration (MS)": 452,
            "Errors": 0
        }
    ]
}

This response is the same as the response sent to us when we initiated the TaskNameHexString task.