atoti_observability

Plugin enhancing observability of Atoti sessions. Projects using Atoti Python SDK will have this kind of architecture: OpenTelemetry distributed tracing allows these different services to contribute their own spans to the same trace. This makes it possible to follow a request across process boundaries, identify bottlenecks, and understand the end-to-end latency of operations. With this plugin enabled, Atoti Server will also expose some metrics through OpenTelemetry. A quick way to get started with observability is to:

Run a Jaeger all-in-one Docker container (or any other OpenTelemetry collector).
Install one of the Python OpenTelemetry exporters.
Install atoti-observability (this plugin).
Call opentelemetry.trace.set_tracer_provider() at the start of the Python application.
Create spans in the application code for each major step.

See the project template for a working example.

When using different ports than the OpenTelemetry’s default ones, do not forget to set os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"] before calling atoti.Session.start() so that the Atoti Server launched in a subprocess inherits it and exports its telemetry data to the right place.

This plugin automatically adds the OpenTelemetry Java agent to the server process but this can be changed by passing another -javaagent:* to the java_options.

Example

Here, the OpenTelemetry environment is already configured:

>>> import os
>>> from opentelemetry.sdk.environment_variables import OTEL_EXPORTER_OTLP_ENDPOINT
>>> OTEL_EXPORTER_OTLP_ENDPOINT in os.environ
True

Creating a simple trace with multiple spans:

>>> from opentelemetry.trace import get_tracer
>>> TRACER = get_tracer("example")

>>> with TRACER.start_as_current_span("root") as span:
...     foo = 1
...     with TRACER.start_as_current_span("intermediate"):
...         bar = 2
...         with TRACER.start_as_current_span("leaf"):
...             baz = 3
>>> _print(span)
⏺ root           [atoti.python_sdk]
└── intermediate [atoti.python_sdk]
    └── leaf     [atoti.python_sdk]

Defining a function to avoid duplicating logic below:

>>> def query(session: tt.Session, /):
...     cities_df = pd.DataFrame(
...         columns=["City", "Price"],
...         data=[
...             ("Berlin", 150.0),
...             ("London", 240.0),
...             ("New York", 270.0),
...             ("Paris", 200.0),
...         ],
...     )
...     table = session.read_pandas(cities_df, keys={"City"}, table_name="Example")
...     cube = session.create_cube(table)
...     level = cube.levels["City"]
...     measure = cube.measures["Price.SUM"]
...     with TRACER.start_as_current_span("example query") as span:
...         _ = cube.query(measure, levels=[level])
...     return span

Calling atoti.Cube.query() generates some Python spans:

>>> span = query(session)
>>> _print(span)
⏺ example query                     [atoti.python_sdk]
└── Cube.query                      [atoti.python_sdk]
    ├── generate_mdx                [atoti.python_sdk]
    └── cellset_to_mdx_query_result [atoti.python_sdk]
        ├── Level.data_type         [atoti.python_sdk]
        └── Measure.data_type       [atoti.python_sdk]

Using a session with the observability plugin enabled will also export the server spans:

>>> span = query(session_with_observability_plugin)
>>> _print(span)
⏺ example query                                        [atoti.python_sdk]
└── Cube.query                                         [atoti.python_sdk]
    ├── GET /activeviam/pivot/rest/v10/cube/discovery  [atoti.server]
    │   └── unsecured request                          [atoti.server]
    ├── generate_mdx                                   [atoti.python_sdk]
    ├── POST /activeviam/pivot/rest/v10/cube/query/mdx [atoti.server]
    │   └── unsecured request                          [atoti.server]
    │       └── Json query service                     [atoti.server]
    │           └── select query                       [atoti.server]
    │               └── Async query executor           [atoti.server]
    │                   └── just-in-time query         [atoti.server]
    │                       └── Database query         [atoti.server]
    └── cellset_to_mdx_query_result                    [atoti.python_sdk]
        ├── Level.data_type                            [atoti.python_sdk]
        └── Measure.data_type                          [atoti.python_sdk]

`ObservabilityConfig`

​Example

Example