> ## Documentation Index
> Fetch the complete documentation index at: https://docs.activeviam.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Scaling with distribution

<a id="module-docs.pages.scaling_with_distribution" />

<Note>
  This feature is not part of the community edition: it needs to be [unlocked](./unlocking_all_features).
</Note>

A [`Session`](../api/atoti.session#atoti.Session) runs on a single machine and the volume of data it can handle is thus limited by the machine’s CPU and memory.
Multiple sessions, each running on a dedicated machine, can be configured to join a cluster and contribute to a [`QueryCube`](../api/atoti.distribution.query_cube#atoti.QueryCube) in a [`QuerySession`](../api/atoti.distribution.query_session#atoti.QuerySession).
Query cubes distribute parts of the execution of incoming queries to data cubes in the cluster, breaking away from the limitations imposed by the single-machine model.

*Important*: Distribution comes at a cost: the communication required between machines in the cluster adds latency and overhead.
Atoti supports [NUMA](https://en.wikipedia.org/wiki/Non-uniform_memory_access) and is designed to scale vertically on large servers.
Go for vertical scaling if possible since it simplifies deployment and monitoring.
If vertical scaling is not option, then this page is for you.

Here, to fit in a guide, the data cubes and the query cube will run on the same machine but, in production, the code must be rearranged so that sessions run on dedicated machines that can communicate with each other.

## Setting up the first data cube

To have something to distribute to, let’s start a cube that will act as our first data cube.

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> import atoti as tt
>>> import pandas as pd
```

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> CUBE_NAME = "World"
```

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> def define_data_model(data_session: tt.Session, /) -> None:
...     table = data_session.create_table(
...         "Facts",
...         data_types={"ID": "String", "Region": "String", "Value": "long"},
...         keys={"ID"},
...     )
...     cube = data_session.create_cube(table, name=CUBE_NAME)
...     h, m = cube.hierarchies, cube.measures
...     m["Portion"] = m["Value.SUM"] / tt.parent_value(
...         m["Value.SUM"], degrees={h["Region"]: 1}
...     )
```

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> def load_data(data_session: tt.Session, /, *, region: str) -> None:
...     table = data_session.tables["Facts"]
...     dataframe = pd.DataFrame(
...         [(f"region-{i}", region, i) for i in range(1, 11)],
...         columns=list(table),
...     )
...     table.load(dataframe)
```

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> north_data_session = tt.Session.start()
>>> define_data_model(north_data_session)
>>> load_data(north_data_session, region="North")
```

This session only has a single region:

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> MDX = f'''SELECT
...   [Facts].[Region].Members ON ROWS,
...   {{[Measures].[contributors.COUNT]}} ON COLUMNS
...   FROM [{CUBE_NAME}]'''
```

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> north_data_session.query_mdx(MDX, keep_totals=True)
       contributors.COUNT
Region
Total                  10
North                  10
```

## Setting up the query cube and the cluster

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> query_session = tt.QuerySession.start()
```

In Atoti, a *cluster* defines how multiple sessions can communicate with each other.
Multiple data cubes and query cubes can contribute to the same cluster.

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> from pathlib import Path
>>> from secrets import token_urlsafe
>>> from tempfile import mkdtemp
```

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> from atoti_jdbc import JdbcPingDiscoveryProtocol
```

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> cluster_definition = tt.ClusterDefinition(
...     application_names={CUBE_NAME},
...     authentication_token=token_urlsafe(),
...     discovery_protocol=JdbcPingDiscoveryProtocol(
...         f"jdbc:h2:{Path(mkdtemp('atoti-cluster')).as_posix()}",
...         username="sa",
...         password="",
...     ),
... )
```

We define a query cube configured to contribute to the cluster:

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> CLUSTER_NAME = "My cluster"
```

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> query_session.session.clusters[CLUSTER_NAME] = cluster_definition
>>> query_session.query_cubes[CUBE_NAME] = tt.QueryCubeDefinition(
...     query_session.session.clusters[CLUSTER_NAME],
...     distributing_levels={("Facts", "Region", "Region")},
... )
```

The query cube is ready to execute queries, but since no data cubes joined the cluster yet, there is no data:

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> query_session.session.query_mdx(
...     f'''SELECT {{[Measures].[contributors.COUNT]}} ON COLUMNS FROM [{CUBE_NAME}]''',
...     keep_totals=True,
... ).empty
True
```

## Making the data cube join the cluster

The only change required to make the query cube useful is to make the data cube join its cluster:

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> north_data_session.clusters[CLUSTER_NAME] = cluster_definition
```

Data cubes join clusters asynchronously.
As a result, their contribution will not be observable for some time.
Polling can be used to wait for join completion:

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> from time import sleep
```

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> def wait_for_data_node(
...     query_session: tt.QuerySession,
...     /,
...     *,
...     max_retries: int = 60,
...     region: str,
...     sleep_duration: float = 1,
... ) -> None:
...     remaining_retries = max_retries
...     mdx = f'''SELECT {{[Measures].[contributors.COUNT]}} ON COLUMNS FROM [{CUBE_NAME}] WHERE [Facts].[Region].[Region].[{region}]'''
...
...     def data_node_joined():
...         try:
...             return not query_session.session.query_mdx(mdx).empty
...         except:
...             return False
...
...     while not data_node_joined():
...         remaining_retries -= 1
...         if not remaining_retries:
...             raise RuntimeError(f"No contributor found for `{region}` region.")
...         sleep(sleep_duration)
```

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> wait_for_data_node(query_session, region="North")
```

At this point, we can observe in the query cube the same hierarchies as the ones defined in the data cube:

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> query_session.session.cubes[CUBE_NAME].hierarchies  
```

<ul><li>Dimensions<ul><li>Facts<ul><li>ID<ol><li>ID</li></ol>    </li><li>Region<ol><li>Region</li></ol>    </li></ul></li></ul></li></ul>

And the measures too:

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> query_session.session.cubes[CUBE_NAME].measures  
```

<ul><li>Measures<ul><li>Portion<ul><li>formatter: #,###.00</li></ul></li><li>Value.MEAN<ul><li>formatter: #,###.00</li></ul></li><li>Value.SUM<ul><li>formatter: #,###</li></ul></li><li>contributors.COUNT<ul><li>formatter: None</li></ul></li><li>contributors.COUNT.World<ul><li>formatter: #,###</li></ul></li><li>update.TIMESTAMP<ul><li>formatter: None</li></ul></li></ul></li></ul>

The data is also available:

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> query_session.session.query_mdx(MDX, keep_totals=True)
       contributors.COUNT
Region
Total                  10
North                  10
```

The [`query_cube_ids`](../api/atoti.Cube.query_cube_ids#atoti.Cube.query_cube_ids) and [`data_cube_ids`](../api/atoti.QueryCube.data_cube_ids#atoti.QueryCube.data_cube_ids) properties give opaque IDs that can be used to check how many cubes of the other kind are connected:

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> len(north_data_session.cubes[CUBE_NAME].query_cube_ids)
1
```

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> len(query_session.query_cubes[CUBE_NAME].data_cube_ids)
1
```

## Adding more data cubes

Distribution is only useful if multiple data cubes contribute to a query cube so let’s do that:

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> south_data_session = tt.Session.start(tt.SessionConfig(ready=False))
>>> south_data_session.clusters[CLUSTER_NAME] = cluster_definition
>>> define_data_model(south_data_session)
>>> load_data(south_data_session, region="South")
```

We set [`ready`](../api/atoti.config.session_config#atoti.SessionConfig.ready) to `False` so that the data cube would not join the cluster before it is fully defined.

Setting [`ready`](../api/atoti.Session.ready#atoti.Session.ready) to `True` will now make it join the cluster.

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> south_data_session.ready = True
>>> wait_for_data_node(query_session, region="South")
```

```pycon theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}}
>>> query_session.session.query_mdx(MDX, keep_totals=True)
       contributors.COUNT
Region
Total                  20
North                  10
South                  10
```