atoti_observability#

Plugin enhancing observability of Atoti sessions.

Projects using Atoti Python SDK will have this kind of architecture:

flowchart LR upstream["Upstream service"] style upstream stroke-dasharray: 5 5 subgraph python["Python application"] atoti-python-sdk["Atoti Python SDK"] other-libs["Other third-party libraries"] style other-libs stroke-dasharray: 5 5 end atoti-server["Atoti Server"] other-services["Other services"] style other-services stroke-dasharray: 5 5 upstream -.-> python atoti-python-sdk --> atoti-server other-libs -.-> other-services

OpenTelemetry distributed tracing allows these different services to contribute their own spans to the same trace. This makes it possible to follow a request across process boundaries, identify bottlenecks, and understand the end-to-end latency of operations.

With this plugin enabled, Atoti Server will also expose some metrics through OpenTelemetry.

A quick way to get started with observability is to:

  1. Run a Jaeger all-in-one Docker container (or any other OpenTelemetry collector).

  2. Install one of the Python OpenTelemetry exporters.

  3. Install atoti-observability (this plugin).

  4. Call opentelemetry.trace.set_tracer_provider() at the start of the Python application.

  5. Create spans in the application code for each major step.

Note

When using different ports than the OpenTelemetry’s default ones, do not forget to set os.environ["OTEL_EXPORTER_OTLP_ENDPOINT"] before calling atoti.Session.start() so that the Atoti Server launched in a subprocess inherits it and exports its telemetry data to the right place.

Tip

This plugin automatically adds the OpenTelemetry Java agent to the server process but this can be changed by passing another -javaagent:* to the java_options.

Example

Here, the OpenTelemetry environment is already configured:

>>> import os
>>> from opentelemetry.sdk.environment_variables import OTEL_EXPORTER_OTLP_ENDPOINT
>>> OTEL_EXPORTER_OTLP_ENDPOINT in os.environ
True

Creating a tracer for this example (usually an application would use the same tracer for all its spans):

>>> from opentelemetry.trace import get_tracer
>>> tracer = get_tracer("example")

Creating a simple trace with multiple spans:

>>> with tracer.start_as_current_span("root") as root_span:
...     foo = 1
...     with tracer.start_as_current_span("intermediate"):
...         bar = 2
...         with tracer.start_as_current_span("leaf"):
...             baz = 3

>>> print(span_tree)
⏺ root [atoti.python_sdk.tests]
└── intermediate [atoti.python_sdk.tests]
    └── leaf [atoti.python_sdk.tests]

With some spans contributed by Atoti Python SDK:

>>> def _create_trace(session: tt.Session, /) -> int:
...     cities_df = pd.DataFrame(
...         columns=["City", "Price"],
...         data=[
...             ("Berlin", 150.0),
...             ("London", 240.0),
...             ("New York", 270.0),
...             ("Paris", 200.0),
...         ],
...     )
...     table = session.read_pandas(cities_df, keys={"City"}, table_name="Example")
...     cube = session.create_cube(table)
...     level = cube.levels["City"]
...     measure = cube.measures["Price.SUM"]
...     with tracer.start_as_current_span("query") as root_span:
...         _ = cube.query(measure, levels=[level])
...     return root_span
>>> root_span = _create_trace(session)

>>> print(span_tree)
⏺ query [atoti.python_sdk.tests]
├── generate_mdx [atoti.python_sdk.tests]
└── cellset_to_mdx_query_result [atoti.python_sdk.tests]

The server spans can be seen when using a session with the observability plugin enabled:

>>> root_span = _create_trace(session_with_observability_plugin)

>>> print(span_tree)
⏺ query [atoti.python_sdk.tests]
├── GET /activeviam/pivot/rest/v10/cube/discovery [atoti.server]
├── generate_mdx [atoti.python_sdk.tests]
├── POST /activeviam/pivot/rest/v10/cube/query/mdx [atoti.server]
│   └── Json query service [atoti.server]
│       └── select query [atoti.server]
│           └── Async query executor [atoti.server]
│               └── just-in-time query [atoti.server]
│                   └── Database query [atoti.server]
└── cellset_to_mdx_query_result [atoti.python_sdk.tests]
    ├── POST /graphql [atoti.server]
    │   └── query [atoti.server]
    └── POST /graphql [atoti.server]
        └── query [atoti.server]

ObservabilityConfig