> ## Documentation Index
> Fetch the complete documentation index at: https://docs.activeviam.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Migration notes

This page explains the changes required to migrate to the stated version
of the Atoti Limits.

## Migrate to 3.3.1

No migration needed.

## Migrate to 3.3.0

Upgrading from version *3.2.0*, see the [Atoti Limits 3.3.0 Release Notes](../../release-notes#330).

Atoti Limits is using Atoti Server 6.0.14-sb3 and Atoti UI 5.1.x.

For new features and fixes included in these releases, please see the [Atoti UI documentation](https://docs.activeviam.com/products/atoti/ui/5.1/)
and [Atoti UI Migration Notes](https://docs.activeviam.com/products/atoti/ui/5.1/docs/changelog), and
the [release notes for Atoti Server](https://docs.activeviam.com/products/atoti/server/6.0.14-sb3/docs/release/changelog/index.).

### Headline announcement

* **Persistence** : Atoti Limits can now work with persistent data sources. See [the migration guide](../../../dev/persistence/limits-data-migration) for more information on how to migrate your limits.
* **Migration to Configuration Properties** : All properties previously retrieved via `@Value` annotations have been migrated to `@ConfigurationProperties` classes. For details, see [Configuration Properties](../../../user-ref/properties/config-properties).
* **Custom validators and validation error handling**: `IValidator` subinterfaces have been created to make swapping the default validator(s) easier
  for a custom solution. `IValidationError` and `IValidationErrorHandler` have been created to allow for creating custom errors and custom behavior
  when handling validation errors. See [Adding Custom Validation](../../../dev/dev-extensions/custom-validation) for more details and information on how to implement these.
* **Validation error reporting**: The default behavior for validation errors has been updated to produce a summary of the validation errors that occurred, including line numbers and column indexes if the source is a file.
* **DLC Configuration Properties** : Data Load Controller property names have been updated, and are now handled by Spring via `@ConfigurationProperties`. See [DLC Configuration Properties](#dlc-configuration-properties).
* **Custom data-loading** : Interfaces have been introduced to facilitate overriding the default Data Load Controller and CSV Source configuration beans.
  Interfaces were also added to facilitate overriding the default Tuple Publisher implementations.
* **Atoti Server upgrade** : Atoti Limits has been upgraded to Atoti Server 6.0.14-sb3.
* **Custom Evaluations** : The `IEvaluationService` and `IEvaluationTaskManager` interfaces have been introduced to facilitate custom evaluation logic. See [Adding Custom Evaluation Logic](../../../dev/dev-extensions/custom-evaluation) for more information.
* **Migration scripts** : We have added a new `limits-migrations` module for migrating CSV files. See [CSV and properties migrations to 3.3.0](#csv-and-properties-migrations-to-330).

### Breaking changes

<Warning>
  Please note that Atoti Server versions 5.9 and 5.10 are now out-of-support. Therefore, we
  no longer support connections
  to servers of these versions starting with this release of Atoti Limits.
</Warning>

#### Managed Object ID behavior

The behavior has changed for how IDs are used in Atoti Limits. Previously, for each structure,
limit, or incident, a `*key` field was generated using a hash of that object’s key fields. Now we expect
the following:

* when the user creates a structure/limit/incident using the DLC or the UI file upload, we
  expect the ID to be populated in the file.
* when the user creates a structure/limit/incident, we don’t expect the ID to be
  populated and will auto-generate it.

This behavior was introduced to simplify the user experience, to enhance data integrity, and to prevent
potential hash-collisions.

#### ConfigurationProperties

New properties have been added as Spring `@ConfigurationProperties` with defaults. This means that the properties may not appear in the [application.yml](../../../user-ref/properties/property-files/application-yml) but have a value. The value can be overridden by adding your own value in the [application.yml](../../../user-ref/properties/property-files/application-yml) file.

#### Datasource configurations

We have updated the default datasource configurations for the `limits application`, `activiti`, the `audit log` and the `workflow processes` to point to the same `limits-application` datasource.

#### Key generation

Previously, a hash function was used to generate the key fields mentioned in the previous section. This could result in hash collisions.
We now use an auto-incrementing `IIdentityGenerator` to generate these IDs. The classes in the previous section now implement a `getDeterminator` function that returns a unique string for each object.

#### Entity field type changes

The following entity fields have been converted from arrays to singular values:

<table><thead><tr><th>Entity</th><th>Field</th><th>Old Type</th><th>New Type</th></tr></thead><tbody><tr><td>LimitEntity</td><td>limitValue</td><td>Double Array</td><td>Double</td></tr><tr><td>LimitsWorkflowProcessInstanceEntity</td><td>limitValues</td><td>Double Array</td><td>Double</td></tr><tr><td>LimitsWorkflowProcessInstanceEntity</td><td>scopeArray</td><td>String Array</td><td>String</td></tr></tbody></table>

If you persist these in a database then the column type should be updated accordingly.

#### Split of `IAlertTaskManager`

The `IAlertTaskManager` interface has been replaced with two interfaces to separate evaluation logic from scheduling logic. If you have a custom implementation
of `IAlertTaskManager`, you will need to replace it with implementations of one or both of `IEvaluationService` and `IEvaluationTaskManager`. For more information on the
new interfaces and how to implement them, see [Adding Custom Evaluation Logic](../../../dev/dev-extensions/custom-evaluation).

#### `*ActivePivot*` classes renamed to `*AtotiServer*`

Classes that follow the naming convention `*ActivePivot*` have been renamed to
match `*AtotiServer*`.
If you have any custom code that references these classes, you will need to update them.

#### UI Settings change

The `availableApplicationServers` setting has changed from a record to an array of server keys. AsOfDate settings that used to sit under this property are now supplied by the server, and therefore no longer need to be provided in the UI settings.

### CSV and properties migrations to 3.3.0

The migration script migrates the `limit_structures.csv` files and `limits.csv` files from
Atoti Limits 3.2.0 to 3.3.0. It also migrates old properties to their new format and outputs
them to the properties directory with the name `application-3-3.yml`. It does not add new properties
or remove old properties. You can use this new properties file as the starting point for your
`application.yml`.

#### How it works

This script expects the following program arguments (in this order):

1. The target version of Atoti Limits (should be `3.3.0`).
2. The source limit structures input file path.
3. The target limit structures output file path.
4. The source limits input file path.
5. The target limits output file path.
6. The path to the directory storing `*.properties`, `*.yaml` and `*.yml` property files.

#### Steps

1. Run `mvn clean install` on `limits-migrations`.
2. Run `java -jar path/to/limits-migrator-tool-exec.jar 3.3 path/to/source/structures.csv path/to/target/structures.csv path/to/source/limits.csv path/to/target/limits.csv path/to/properties/folder`.
   Your files are now converted to the new format and can be found in the target directory.

<Note>
  The output directory must already exist, but the file will be created.
</Note>

### Input file formats

#### Modified

<table><thead><tr><th>Modification</th><th>File</th><th>Field</th><th>Optional</th><th>Description</th></tr></thead><tbody><tr><td>Removed field</td><td><a href="../../../user-ref/input-files/limit_structures">limit\_structures.csv</a></td><td><code>User ID</code></td><td>N</td><td><code>User ID</code> has been removed from all input files. Please remove the <code>User ID</code> column from your <code>limit\_structures.csv</code> files.</td></tr><tr><td>Renamed field</td><td><a href="../../../user-ref/input-files/limit_structures">limit\_structures.csv</a></td><td><code>Reference ID</code></td><td>N</td><td><code>Reference ID</code> has been renamed to <code>Structure ID</code>. Please update the <code>Reference ID</code> column in your <code>limit\_structures.csv</code> files.</td></tr><tr><td>Renamed field</td><td><a href="../../../user-ref/input-files/limits_approve">limit\_approve.csv</a></td><td><code>Structure Reference ID</code></td><td>N</td><td><code>Structure Reference ID</code> has been renamed to <code>Structure ID</code>. Please update the <code>Structure Reference ID</code> column in your <code>limit\_approve.csv</code> files.</td></tr><tr><td>Renamed field</td><td><a href="../../../user-ref/input-files/limits_approve">limit\_approve.csv</a></td><td><code>parentLimitKey</code></td><td>N</td><td><code>parentLimitKey</code> has been renamed to <code>Source Limit ID</code>. Please update the <code>parentLimitKey</code> column in your <code>limit\_approve.csv</code> files.</td></tr></tbody></table>

### Configuration

#### Configuration properties

##### Properties modified

Updated property names:

###### DLC Configuration Properties

<table><thead><tr><th>Old Property Name</th><th>New Property Name</th><th>Description</th><th>Default Value</th></tr></thead><tbody><tr><td>input.data.root.dir.path</td><td>limits.dlc.root-dir</td><td>Directory path to the data files.</td><td>./src/test/resources/data-samples</td></tr><tr><td>csvSource.subdirectory.dataset</td><td>limits.dlc.sub-directory-paths.root</td><td>Path extension to specific server data directories.</td><td>/data</td></tr><tr><td>csvSource.subdirectory.dataset.whatif</td><td>limits.dlc.sub-directory-paths.whatif</td><td>Path extension to directory containing the What-If CSV files.</td><td>/whatif</td></tr><tr><td>csvSource.subdirectory.dataset.stage</td><td>limits.dlc.sub-directory-paths.stage</td><td>Path extension to directory containing the Stage CSV files.</td><td>/stage</td></tr><tr><td>default.csvSource.parser.threads</td><td>limits.dlc.parser-threads</td><td>Specifies the number of threads to be used for data loading, used for configuring the <code>DlcCSVSourceConfiguration</code>.</td><td>4</td></tr><tr><td>default.csvSource.buffer.size</td><td>limits.dlc.buffer-size</td><td>CSV buffer size in KB, used for configuring the <code>DlcCSVSourceConfiguration</code>.</td><td>1024</td></tr><tr><td>as.of.date.file.path.matcher</td><td>limits.dlc.path-matchers.asofdate</td><td>Regular expression to match <code>as\_of\_date.csv</code> files.</td><td><code>glob:\*\*/\*as\_of\_date\*.csv</code></td></tr><tr><td>limits.definitions.file.path.matcher</td><td>limits.dlc.path-matchers.limitstructures</td><td>Regular expression to match <code>limit\_structures.csv</code> files.</td><td><code>glob:\*\*/\*limit\_structures\*.csv</code></td></tr><tr><td>limits.approve.parameters.file.path.matcher</td><td>limits.dlc.path-matchers.limits</td><td>Regular expression to match <code>limits\_approve.csv</code> files.</td><td><code>glob:\*\*/\*limits\_approve\*.csv</code></td></tr><tr><td>alerts.definitions.file.path.matcher</td><td>limits.dlc.path-matchers.incidents</td><td>Regular expression to match <code>incident.csv</code> files.</td><td><code>glob:\*\*/\*incident\*.csv</code></td></tr></tbody></table>

###### Connected Atoti Server Properties

The following properties are set on the connected Atoti server and are used to configure the connection to Atoti Limits.

<table><thead><tr><th>Old Property Name</th><th>New Property Name</th><th>Description</th><th>Default Value</th></tr></thead><tbody><tr><td>ap.url</td><td>limits-connected-server.url</td><td>The URL of the Atoti server’s cube discovery</td><td /></tr><tr><td>ap.auth</td><td>limits-connected-server.auth</td><td>The base-64 encoded authentication used to auto-configure the authorization for the module to connect to the Atoti server.</td><td /></tr><tr><td>ap.auth.username</td><td>limits-connected-server.username</td><td>The username used to auto-configure the authorization for Atoti Limits to connect to the Atoti server.</td><td /></tr><tr><td>ap.configuration.kpi.path</td><td>limits-connected-server.kpi-path</td><td>The path to get the KPI permissions.</td><td /></tr><tr><td>ap.asOfDate.dim</td><td>limits-connected-server.as-of-date</td><td>The slicing date dimension present in the Atoti Server cube. Notation is Level\@Hierarchy\@Dimension.</td><td /></tr><tr><td>level-path.url</td><td>limits-connected-server.level-path-url</td><td>The URL of the level path rest service.</td><td /></tr><tr><td>limits.connect.fixed.delay</td><td>limits.connection.delay</td><td>The time delay in milliseconds between consecutive attempts to connect to the module.</td><td>10000</td></tr><tr><td>limits.connect.attempts</td><td>limits.connection.attempts</td><td>The number of attempts to connect to the module before quitting. If less than zero, the Atoti server will continuously try to connect. The Atoti server will also try to reconnect if Atoti Limits is stopped and restarted.</td><td>-1</td></tr><tr><td>auto-config.enabled</td><td>limits.auto-config-enabled</td><td>If false, the auto-configuration methods will not be fired.</td><td>true</td></tr><tr><td>atoti.server.version</td><td>limits-connected-server.atoti-server-version</td><td>The version of Atoti Server the connected server is running.</td><td /></tr></tbody></table>

The following properties are used by Atoti Limits to manage connections to connected Atoti servers. These can be auto-configured if `limits.auto-config-enabled` is true. If you manually configure the connection between Atoti Limits and a connected Atoti server, you will need to update these properties.

<table><thead><tr><th>Old Property Name</th><th>New Property Name</th><th>Description</th><th>Default Value</th></tr></thead><tbody><tr><td>ap.url.map</td><td>limits.connected-server.url.map</td><td>Map of the Solution name to http url.</td><td /></tr><tr><td>ap.version.map</td><td>limits.connected-server.version.map</td><td>Map of the Solution name to its server version: \{ \< server name >: \< version >, … }.</td><td /></tr><tr><td>ap.configuration.kpi.path.map</td><td>limits.connected-server.kpi-path.map</td><td>Atoti Server configuration KPI path to get the KPI permissions used during startup to delete the stale KPIs</td><td /></tr><tr><td>ap.auth.map</td><td>limits.connected-server.auth.map</td><td>Map of the Solution name to authentication token.</td><td /></tr><tr><td>ap.asOfDate.dim.map</td><td>limits.connected-server.as-of-date.map</td><td>Map of the Solution name to AsOfDate cube location.</td><td /></tr><tr><td>level-path.url</td><td>limits.connected-server.level-path.map</td><td>Map of the Solution name to Level Path REST endpoint URL.</td><td /></tr></tbody></table>

#### Property files

##### Files Modified

###### [application.yml](../../../user-ref/properties/property-files/application-yml)

New properties:

<table><thead><tr><th>Property Name</th><th>Comment</th><th>Value</th></tr></thead><tbody><tr><td>limits.application.datasource</td><td>Properties prefixed by this are related to the datasource for the <strong>limits application</strong></td><td /></tr><tr><td>limits.application.datasource.url</td><td>Connection url</td><td>jdbc:h2:mem:limits-application;DB\_CLOSE\_DELAY=-1;</td></tr><tr><td>limits.application.datasource.username</td><td>Connection username</td><td>app</td></tr><tr><td>limits.application.datasource.password</td><td>Connection password</td><td /></tr><tr><td>limits.application.datasource.properties.hibernate.dialect</td><td>Hibernate dialect</td><td>org.hibernate.dialect.H2Dialect</td></tr><tr><td>limits.application.datasource.properties.hibernate.format\_sql</td><td>Enables formatting of SQL logged to the console.</td><td>false</td></tr><tr><td>limits.application.datasource.properties.hibernate.hbm2ddl.auto</td><td>Setting for how Spring should handle the database table on startup. Potential values include: create, create\_drop, none, validate, update, drop, validate and truncate.</td><td>update</td></tr><tr><td>limits.application.datasource.properties.hibernate.globally\_quoted\_identifiers</td><td>Escapes all database identifiers, so we don’t have to put column and table names in quotations.</td><td>true</td></tr><tr><td>limits.application.datasource.hikari.connectionTimeout</td><td>Maximum timeout in milliseconds user will wait for a connection from connection pool.</td><td>30000</td></tr><tr><td>limits.application.datasource.hikari.idleTimeout</td><td>Maximum time a connection can remain idle in connection pool.</td><td>60000</td></tr><tr><td>limits.application.datasource.hikari.minimumIdle</td><td>Minimum number of idle connections in a connection pool.</td><td>1</td></tr><tr><td>limits.application.datasource.hikari.maximumPoolSize</td><td>Maximum size of the connection pool, including both idle and in-use connections.</td><td>10</td></tr><tr><td>limits.application.datasource.hikari.poolName</td><td>Name of the connection pool.</td><td>limits-application</td></tr><tr><td>application.datasource</td><td>Properties prefixed by this are related to the datasource for the <strong>limits workflow processes</strong></td><td /></tr><tr><td>application.datasource.url</td><td>Connection url</td><td>jdbc:h2:mem:limits-application;DB\_CLOSE\_DELAY=-1</td></tr><tr><td>application.datasource.username</td><td>Connection username</td><td>app</td></tr><tr><td>application.datasource.password</td><td>Connection password</td><td /></tr><tr><td>application.datasource.properties.hibernate.dialect</td><td>Hibernate dialect</td><td>org.hibernate.dialect.H2Dialect</td></tr><tr><td>application.datasource.properties.hibernate.format\_sql</td><td>Enables formatting of SQL logged to the console.</td><td>false</td></tr><tr><td>application.datasource.properties.hibernate.hbm2ddl.auto</td><td>Setting for how Spring should handle the database table on startup. Potential values include: create, create\_drop, none, validate, update, drop, validate and truncate.</td><td>update</td></tr><tr><td>application.datasource.properties.hibernate.globally\_quoted\_identifiers</td><td>Escapes all database identifiers, so we don’t have to put column and table names in quotations.</td><td>true</td></tr><tr><td>application.datasource.hikari.connectionTimeout</td><td>Maximum timeout in milliseconds user will wait for a connection from connection pool.</td><td>30000</td></tr><tr><td>application.datasource.hikari.idleTimeout</td><td>Maximum time a connection can remain idle in connection pool.</td><td>60000</td></tr><tr><td>application.datasource.hikari.minimumIdle</td><td>Minimum number of idle connections in a connection pool.</td><td>1</td></tr><tr><td>application.datasource.hikari.maximumPoolSize</td><td>Maximum size of the connection pool, including both idle and in-use connections.</td><td>10</td></tr><tr><td>application.datasource.hikari.poolName</td><td>Name of the connection pool.</td><td>application-process-instance</td></tr><tr><td>limits.structure.templates</td><td>A map of LimitStructureTemplate definitions by templateName. See <a href="../../../dev/dev-extensions/custom-limit-structure-templates">Adding Custom Limit Structure Templates</a></td><td>see <a href="../../../user-ref/properties/property-files/application-yml">application.yml</a></td></tr><tr><td>spring.h2.console.enabled</td><td>True if the <a href="https://www.h2database.com/html/quickstart.html">H2 console</a> is available. This is useful for investigating JDBC connections.</td><td><code>false</code></td></tr><tr><td>spring.data.rest.base-path</td><td>The base URL for Spring Data REST services which are autoconfigured for JPA repositories.</td><td><code>limits/rest/v2/spring/jpa</code></td></tr><tr><td>spring.liquibase.enabled</td><td>True if Liquibase database schema migrations listed in <code>limits-starter/src/main/resources/liquibase/master-changelog.yaml</code> should be applied on startup.</td><td><code>false</code></td></tr><tr><td>spring.liquibase.change-log</td><td>The location of the changelog file if Liquibase database schema migrations are applied on startup.</td><td><code>classpath:/liquibase/master-changelog.yaml</code></td></tr></tbody></table>

#### Files Deleted

The following `*.properties` property files have been deleted. Properties still in use have been moved to [`@ConfigurationProperties` classes](../../../user-ref/properties/config-properties/limits-activeviam) or [application.yml](../../../user-ref/properties/property-files/application-yml):

* `env-default.properties`
* `hibernate.properties`
* `jwt.properties`
* `limits.properties`
* `limits_test.properties`
* `tracing.properties`

### Datastores

#### Modified stores

<table><thead><tr><th>Modification</th><th>Store</th><th>Field</th><th>Type</th><th>Description</th></tr></thead><tbody><tr><td>Deleted</td><td><a href="../../../user-ref/datastore/limit-structures">Limit Structures</a></td><td><code>UserId</code></td><td>String</td><td>The field has been deleted. It contained the ID or name of the user who created a given limit structure.</td></tr><tr><td>Renamed</td><td><a href="../../../user-ref/datastore/limit-structures">Limit Structures</a></td><td><code>Reference ID</code></td><td>string</td><td><code>Reference ID</code> has been renamed to <code>Structure ID</code>.</td></tr><tr><td>Deleted</td><td><a href="../../../user-ref/datastore/limit-structures">Limit Structures</a></td><td><code>structureKey</code></td><td>string</td><td><code>structureKey</code> has been deleted. <code>Structure ID</code> will be used instead.</td></tr><tr><td>Renamed</td><td><a href="../../../user-ref/datastore/limits-store">Limits</a></td><td><code>Reference ID</code></td><td>string</td><td><code>Reference ID</code> has been renamed to <code>Structure ID</code>.</td></tr><tr><td>Renamed</td><td><a href="../../../user-ref/datastore/limits-store">Limits</a></td><td><code>parentLimitKey</code></td><td>string</td><td><code>parentLimitKey</code> has been renamed to <code>Source Limit ID</code>.</td></tr><tr><td>Deleted</td><td><a href="../../../user-ref/datastore/limits-store">Limits</a></td><td><code>limitKey</code></td><td>string</td><td><code>limitKey</code> has been deleted. <code>Limit ID</code> will be used instead.</td></tr><tr><td>Deleted</td><td><a href="../../../user-ref/datastore/limits-store">Limits</a></td><td><code>structureKey</code></td><td>string</td><td><code>structureKey</code> has been deleted. <code>Structure ID</code> will be used instead.</td></tr><tr><td>Deleted</td><td><a href="../../../user-ref/datastore/limits-store">Limits</a></td><td><code>Limit Name</code></td><td>string</td><td>The unused <code>Limit Name</code> column has been deleted.</td></tr><tr><td>Renamed</td><td><a href="../../../user-ref/datastore/incident-store">Incidents</a></td><td><code>incidentKey</code></td><td>string</td><td><code>incidentKey</code> has been renamed to <code>Incident ID</code>.</td></tr><tr><td>Renamed</td><td><a href="../../../user-ref/datastore/incident-store">Incidents</a></td><td><code>limitKey</code></td><td>string</td><td><code>limitKey</code> has been renamed to <code>Limit ID</code>.</td></tr></tbody></table>

### Cube schema

The `limitKey` hierarchy has been renamed to [`limitId`](../../../user-ref/cube/limitid).

### Measures

No changes.

### Context values

No changes.

### Other changes

#### Changes to ILimitsProcessInstanceWorkflowService

If you implement a custom `ILimitsProcessInstanceWorkflowService`, please implement the new `decorateWorkflowObject` method.
This is intended to decorate the `LimitDTO` and `IncidentDTO` objects with the values of workflow variables.
You will also need to modify the `update` method to the new signature which includes the additional `publishTuples` and `refreshKpis` parameters.

#### Custom data-loading

These classes were moved from the `limits-starter` module to the `limits-activeviam` module:

* `LimitsDlcConfigurationProperties`
* `CSVSourceConfig`
* `ACSVSourceConfig`
* `DataLoadControllerConfig`
* `InitialLoad`
* `LimitTuplePublisher`
* `AsOfDateTuplePublisher`
* `IncidentTuplePublisher`

If you have customized any of these classes, you need to migrate your code. Interfaces have been provided to override all classes listed above except`LimitsDlcConfigurationProperties` and `InitialLoad`.