atoti_observability#

Plugin enhancing observability of Atoti sessions.

Projects using Atoti Python SDK will have this kind of architecture:

flowchart LR upstream["Upstream service"] style upstream stroke-dasharray: 5 5 subgraph python["Python application"] atoti-python-sdk["Atoti Python SDK"] other-libs["Other third-party libraries"] style other-libs stroke-dasharray: 5 5 end atoti-server["Atoti Server"] other-services["Other services"] style other-services stroke-dasharray: 5 5 upstream -.-> python atoti-python-sdk --> atoti-server other-libs -.-> other-services

OpenTelemetry distributed tracing allows these different services to contribute their own spans to the same trace. This makes it possible to follow a request across process boundaries, identify bottlenecks, and understand the end-to-end latency of operations.

With this plugin enabled, Atoti Server will also expose some metrics through OpenTelemetry.

A quick way to get started with observability is to:

  1. Run a Jaeger all-in-one Docker container (or any other OpenTelemetry collector).

  2. Install one of the Python OpenTelemetry exporters.

  3. Install atoti-observability (this plugin).

  4. Call opentelemetry.trace.set_tracer_provider() at the start of the Python application.

  5. Create spans in the application code for each major step.

See the project template for a working example.

Note

When using different ports than the OpenTelemetry’s default ones, do not forget to set os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"] before calling atoti.Session.start() so that the Atoti Server launched in a subprocess inherits it and exports its telemetry data to the right place.

Tip

This plugin automatically adds the OpenTelemetry Java agent to the server process but this can be changed by passing another -javaagent:* to the java_options.

Example

Here, the OpenTelemetry environment is already configured:

>>> import os
>>> from opentelemetry.sdk.environment_variables import OTEL_EXPORTER_OTLP_ENDPOINT
>>> OTEL_EXPORTER_OTLP_ENDPOINT in os.environ
True

Creating a simple trace with multiple spans:

>>> from opentelemetry.trace import get_tracer
>>> TRACER = get_tracer("example")

>>> with TRACER.start_as_current_span("root") as span:
...     foo = 1
...     with TRACER.start_as_current_span("intermediate"):
...         bar = 2
...         with TRACER.start_as_current_span("leaf"):
...             baz = 3
>>> _print(span)
⏺ root           [atoti.python_sdk]
└── intermediate [atoti.python_sdk]
    └── leaf     [atoti.python_sdk]

Defining a function to avoid duplicating logic below:

>>> def query(session: tt.Session, /):
...     cities_df = pd.DataFrame(
...         columns=["City", "Price"],
...         data=[
...             ("Berlin", 150.0),
...             ("London", 240.0),
...             ("New York", 270.0),
...             ("Paris", 200.0),
...         ],
...     )
...     table = session.read_pandas(cities_df, keys={"City"}, table_name="Example")
...     cube = session.create_cube(table)
...     level = cube.levels["City"]
...     measure = cube.measures["Price.SUM"]
...     with TRACER.start_as_current_span("example query") as span:
...         _ = cube.query(measure, levels=[level])
...     return span

Calling atoti.Cube.query() generates some Python spans:

>>> span = query(session)
>>> _print(span)
⏺ example query                     [atoti.python_sdk]
└── Cube.query                      [atoti.python_sdk]
    ├── generate_mdx                [atoti.python_sdk]
    └── cellset_to_mdx_query_result [atoti.python_sdk]
        ├── Level.data_type         [atoti.python_sdk]
        └── Measure.data_type       [atoti.python_sdk]

Using a session with the observability plugin enabled will also export the server spans:

>>> span = query(session_with_observability_plugin)
>>> _print(span)
⏺ example query                                        [atoti.python_sdk]
└── Cube.query                                         [atoti.python_sdk]
    ├── GET /activeviam/pivot/rest/v10/cube/discovery  [atoti.server]
    ├── generate_mdx                                   [atoti.python_sdk]
    ├── POST /activeviam/pivot/rest/v10/cube/query/mdx [atoti.server]
    │   └── Json query service                         [atoti.server]
    │       └── select query                           [atoti.server]
    │           └── Async query executor               [atoti.server]
    │               └── just-in-time query             [atoti.server]
    │                   └── Database query             [atoti.server]
    └── cellset_to_mdx_query_result                    [atoti.python_sdk]
        ├── Level.data_type                            [atoti.python_sdk]
        └── Measure.data_type                          [atoti.python_sdk]

ObservabilityConfig