Migration notes 3.3

This page explains the changes required to migrate to the stated version of the Atoti Limits.

Migrate to 3.3.1

No migration needed.

Migrate to 3.3.0

Upgrading from version 3.2.0, see the Atoti Limits 3.3.0 Release Notes.

Atoti Limits is using Atoti Server 6.0.14-sb3 and Atoti UI 5.1.x.

For new features and fixes included in these releases, please see the Atoti UI documentation and Atoti UI Migration Notes, and the release notes for Atoti Server.

Headline announcement

• Persistence : Atoti Limits can now work with persistent data sources. See the migration guide for more information on how to migrate your limits.
• Migration to Configuration Properties : All properties previously retrieved via @Value annotations have been migrated to @ConfigurationProperties classes. For details, see Configuration Properties.
• Custom validators and validation error handling: IValidator subinterfaces have been created to make swapping the default validator(s) easier for a custom solution. IValidationError and IValidationErrorHandler have been created to allow for creating custom errors and custom behavior when handling validation errors. See Adding Custom Validation for more details and information on how to implement these.
• Validation error reporting: The default behavior for validation errors has been updated to produce a summary of the validation errors that occurred, including line numbers and column indexes if the source is a file.
• DLC Configuration Properties : Data Load Controller property names have been updated, and are now handled by Spring via @ConfigurationProperties. See DLC Configuration Properties.
• Custom data-loading : Interfaces have been introduced to facilitate overriding the default Data Load Controller and CSV Source configuration beans. See Custom Data Load Controller Config and Custom CSV Source Config for more information. Interfaces were also added to facilitate overriding the default Tuple Publisher implementations. sSee Custom Tuple Publisher for more information.
• Atoti Server upgrade : Atoti Limits has been upgraded to Atoti Server 6.0.14-sb3.
• Custom Evaluations : The IEvaluationService and IEvaluationTaskManager interfaces have been introduced to facilitate custom evaluation logic. See Adding Custom Evaluation Logic for more information.
• Migration scripts : We have added a new limits-migrations module for migrating CSV files. See CSV and properties migrations to 3.3.0.

Breaking changes

Managed Object ID behavior

The behavior has changed for how IDs are used in Atoti Limits. Previously, for each structure, limit, or incident, a *key field was generated using a hash of that object’s key fields. Now we expect the following:

• when the user creates a structure/limit/incident using the DLC or the UI file upload, we expect the ID to be populated in the file.
• when the user creates a structure/limit/incident, we don’t expect the ID to be populated and will auto-generate it.

This behavior was introduced to simplify the user experience, to enhance data integrity, and to prevent potential hash-collisions.

ConfigurationProperties

New properties have been added as Spring @ConfigurationProperties with defaults. This means that the properties may not appear in the application.yml but have a value. The value can be overridden by adding your own value in the application.yml file.

Datasource configurations

We have updated the default datasource configurations for the limits application, activiti, the audit log and the workflow processes to point to the same limits-application datasource.

Key generation

Previously, a hash function was used to generate the key fields mentioned in the previous section. This could result in hash collisions. We now use an auto-incrementing IIdentityGenerator to generate these IDs. The classes in the previous section now implement a getDeterminator function that returns a unique string for each object.

Entity field type changes

The following entity fields have been converted from arrays to singular values:

Entity	Field	Old Type	New Type
LimitEntity	limitValue	Double Array	Double
LimitsWorkflowProcessInstanceEntity	limitValues	Double Array	Double
LimitsWorkflowProcessInstanceEntity	scopeArray	String Array	String

If you persist these in a database then the column type should be updated accordingly.

Split of IAlertTaskManager

The IAlertTaskManager interface has been replaced with two interfaces to separate evaluation logic from scheduling logic. If you have a custom implementation of IAlertTaskManager, you will need to replace it with implementations of one or both of IEvaluationService and IEvaluationTaskManager. For more information on the new interfaces and how to implement them, see Adding Custom Evaluation Logic.

*ActivePivot* classes renamed to *AtotiServer*

Classes that follow the naming convention *ActivePivot* have been renamed to match *AtotiServer*. If you have any custom code that references these classes, you will need to update them.

UI Settings change

The availableApplicationServers setting has changed from a record to an array of server keys. AsOfDate settings that used to sit under this property are now supplied by the server, and therefore no longer need to be provided in the UI settings.

CSV and properties migrations to 3.3.0

The migration script migrates the limit_structures.csv files and limits.csv files from Atoti Limits 3.2.0 to 3.3.0. It also migrates old properties to their new format and outputs them to the properties directory with the name application-3-3.yml. It does not add new properties or remove old properties. You can use this new properties file as the starting point for your application.yml.

How it works

This script expects the following program arguments (in this order):

1. The target version of Atoti Limits (should be 3.3.0).
2. The source limit structures input file path.
3. The target limit structures output file path.
4. The source limits input file path.
5. The target limits output file path.
6. The path to the directory storing *.properties, *.yaml and *.yml property files.

Steps

1. Run mvn clean install on limits-migrations.
2. Run java -jar path/to/limits-migrator-tool-exec.jar 3.3 path/to/source/structures.csv path/to/target/structures.csv path/to/source/limits.csv path/to/target/limits.csv path/to/properties/folder.
   Your files are now converted to the new format and can be found in the target directory.
   

   note

   The output directory must already exist, but the file will be created.

Input file formats

Modified

Modification	File	Field	Optional	Description
Removed field	limit_structures.csv	User ID	N	User ID has been removed from all input files. Please remove the User ID column from your limit_structures.csv files.
Renamed field	limit_structures.csv	Reference ID	N	Reference ID has been renamed to Structure ID. Please update the Reference ID column in your limit_structures.csv files.
Renamed field	limit_approve.csv	Structure Reference ID	N	Structure Reference ID has been renamed to Structure ID. Please update the Structure Reference ID column in your limit_approve.csv files.
Renamed field	limit_approve.csv	parentLimitKey	N	parentLimitKey has been renamed to Source Limit ID. Please update the parentLimitKey column in your limit_approve.csv files.

Configuration

Configuration properties

Properties modified

Updated property names:

DLC Configuration Properties

Old Property Name	New Property Name	Description	Default Value
input.data.root.dir.path	limits.dlc.root-dir	Directory path to the data files.	./src/test/resources/data-samples
csvSource.subdirectory.dataset	limits.dlc.sub-directory-paths.root	Path extension to specific server data directories.	/data
csvSource.subdirectory.dataset.whatif	limits.dlc.sub-directory-paths.whatif	Path extension to directory containing the What-If CSV files.	/whatif
csvSource.subdirectory.dataset.stage	limits.dlc.sub-directory-paths.stage	Path extension to directory containing the Stage CSV files.	/stage
default.csvSource.parser.threads	limits.dlc.parser-threads	Specifies the number of threads to be used for data loading, used for configuring the DlcCSVSourceConfiguration.	4
default.csvSource.buffer.size	limits.dlc.buffer-size	CSV buffer size in KB, used for configuring the DlcCSVSourceConfiguration.	1024
as.of.date.file.path.matcher	limits.dlc.path-matchers.asofdate	Regular expression to match as_of_date.csv files.	glob:**/*as_of_date*.csv
limits.definitions.file.path.matcher	limits.dlc.path-matchers.limitstructures	Regular expression to match limit_structures.csv files.	glob:**/*limit_structures*.csv
limits.approve.parameters.file.path.matcher	limits.dlc.path-matchers.limits	Regular expression to match limits_approve.csv files.	glob:**/*limits_approve*.csv
alerts.definitions.file.path.matcher	limits.dlc.path-matchers.incidents	Regular expression to match incident.csv files.	glob:**/*incident*.csv

Limits Properties

Old Property Name	New Property Name	Description	Default Value
limits.rest.url	limits.rest-url	The URL where the REST APIs of Atoti Limits are located	http://localhost:3090/limits/rest/v2/

Connected Atoti Server Properties

The following properties are set on the connected Atoti server and are used to configure the connection to Atoti Limits.

Old Property Name	New Property Name	Description	Default Value
ap.url	limits-connected-server.url	The URL of the Atoti server’s cube discovery
ap.auth	limits-connected-server.auth	The base-64 encoded authentication used to auto-configure the authorization for the module to connect to the Atoti server.
ap.auth.username	limits-connected-server.username	The username used to auto-configure the authorization for Atoti Limits to connect to the Atoti server.
ap.configuration.kpi.path	limits-connected-server.kpi-path	The path to get the KPI permissions.
ap.asOfDate.dim	limits-connected-server.as-of-date	The slicing date dimension present in the Atoti Server cube. Notation is Level@Hierarchy@Dimension.
level-path.url	limits-connected-server.level-path-url	The URL of the level path rest service.
limits.connect.fixed.delay	limits.connection.delay	The time delay in milliseconds between consecutive attempts to connect to the module.	10000
limits.connect.attempts	limits.connection.attempts	The number of attempts to connect to the module before quitting. If less than zero, the Atoti server will continuously try to connect. The Atoti server will also try to reconnect if Atoti Limits is stopped and restarted.	-1
auto-config.enabled	limits.auto-config-enabled	If false, the auto-configuration methods will not be fired.	true
atoti.server.version	limits-connected-server.atoti-server-version	The version of Atoti Server the connected server is running.

The following properties are used by Atoti Limits to manage connections to connected Atoti servers. These can be auto-configured if limits.auto-config-enabled is true. If you manually configure the connection between Atoti Limits and a connected Atoti server, you will need to update these properties.

Old Property Name	New Property Name	Description	Default Value
ap.url.map	limits.connected-server.url.map	Map of the Solution name to http url.
ap.version.map	limits.connected-server.version.map	Map of the Solution name to its server version: { < server name >: < version >, … }.
ap.configuration.kpi.path.map	limits.connected-server.kpi-path.map	Atoti Server configuration KPI path to get the KPI permissions used during startup to delete the stale KPIs
ap.auth.map	limits.connected-server.auth.map	Map of the Solution name to authentication token.
ap.asOfDate.dim.map	limits.connected-server.as-of-date.map	Map of the Solution name to AsOfDate cube location.
level-path.url	limits.connected-server.level-path.map	Map of the Solution name to Level Path REST endpoint URL.

CSV Topic Configuration Properties

Old Property Name	New Property Name	Description	Default Value
default.topic.accept.incomplete.lines	limits.topic.csv.incomplete-lines	Specifies whether incomplete CSV lines are accepted.	false
default.topic.accept.overflow.lines	limits.topic.csv.overflow-lines	Specifies whether overflowing CSV lines are accepted.	false
default.topic.parser.process.quotes	limits.topic.csv.process-quotes	If true, the parser will parse values in double quotes. Otherwise, quotes are treated as literals	true
default.topic.parser.column.separator	limits.topic.csv.column-separator	Parser column separator.	‘,’
default.topic.file.number.skipped.lines	limits.topic.csv.skipped-lines	The number of lines to skip at the start of the file.	1

Activiti Mail Server Properties

Old Property Name	New Property Name	Description	Default Value
activiti.mail.host	limits.activiti.mail.host	URL of the mail server.	smtp.gmail.com
activiti.mail.port	limits.activiti.mail.port	Port of the mail server.	465
activiti.mail.username	limits.activiti.mail.username	username to use for authentication to the mail server.
activiti.mail.password	limits.activiti.mail.password	password to use for authentication to the mail server.
activiti.mail.ssl	limits.activiti.mail.ssl	Use SSL when connecting to the mail server.	true
activiti.mail.tls	limits.activiti.mail.tls	Use TLS when connection to the mail server.	true
activiti.mail.breach.url	limits.activiti.mail.breach-url	The URL where the raw binary data can be fetched from.	http://localhost:3000/#/dashboard/d46

Incident Properties

Old Property Name	New Property Name	Description	Default Value
incident.output.data.dir.path	limits.incident.output-data-dir	The output directory for incident files generated after an evaluation.	./src/test/resources/data-samples/data
incident.evaluation.parallel.tasks	limits.incident.evaluation.parallel-tasks	Number of threads to use when evaluating limits on-demand.	10
limit.evaluation.include-passes	limits.incident.evaluation.include-passes	If true, Atoti Limits stores passes (i.e. evaluations that do not result in a breach or warning) in the datastore during limit evaluation.	false
limit.evaluation.error.include-stacktrace	limits.incident.evaluation.error.include-stacktrace	true if the stack trace should be available in the evaluation response to the UI.	false

Scope Properties

Old Property Name	New Property Name	Description	Default Value
scope.forceUnambiguousLevelPaths	limits.scope.force-unambiguous-level-paths	Boolean flag to enable if we want to force Scopes to be Unambiguous on Multi-Level Hierarchies. Please see limits.scope.force-unambiguous-level-paths for more.	false
scope.autofill-unambiguous-level-paths	limits.scope.autofill-unambiguous-level-paths	Boolean flag to enable if we want to autofill Scopes to its Unambiguous Multi-Level values. Please see limits.scope.autofill-unambiguous-level-paths for more.	true
scope.isUsingExplicitMatching	limits.scope.is-using-explicit-matching	Flag to enable explicit scope matching. If true, a limit will only be visible in the business cube at the exact location the limit’s scope is defined on and no lower. This does not affect the location evaluations and breaches are checked at.	true

Miscellaneous Properties

Old Property Name	New Property Name	Description	Default Value
sample-rate-cron	limits.alert-task.sample-rate-cron	Cron expression for Intraday Alert task’s cron job. Default is to run the evaluation every 5 minutes. An example cron expression tool: https://www.freeformatter.com/cron-expression-generator-quartz.html. Note that cron expressions are evaluated differently in Java compared to Linux. See this explanation by Baeldung. We require the Spring cron syntax i.e. 6 space-separated fields.	0 */5 * * * *
task-scheduler-thread-pool-size	limits.task-scheduler.thread-pool-size	Thread pool size for task scheduler	4
canTheApproverBeTheSameAsTheCreator	limits.workflow-rules.can-approver-be-same-as-creator	If true, the creator of a limit may also approve it.	false
discovery-manager.polling-delay	limits.discovery-manager.polling-delay	Milliseconds that the connected cube description is valid for before needing to be refreshed. If the cube is expected to change frequently, this should be decreased.	30000

Property files

Files Modified

application.yml

New properties:

Property Name	Comment	Value
limits.application.datasource	Properties prefixed by this are related to the datasource for the limits application
limits.application.datasource.url	Connection url	jdbc:h2:mem:limits-application;DB_CLOSE_DELAY=-1;
limits.application.datasource.username	Connection username	app
limits.application.datasource.password	Connection password
limits.application.datasource.properties.hibernate.dialect	Hibernate dialect	org.hibernate.dialect.H2Dialect
limits.application.datasource.properties.hibernate.format_sql	Enables formatting of SQL logged to the console.	false
limits.application.datasource.properties.hibernate.hbm2ddl.auto	Setting for how Spring should handle the database table on startup. Potential values include: create, create_drop, none, validate, update, drop, validate and truncate.	update
limits.application.datasource.properties.hibernate.globally_quoted_identifiers	Escapes all database identifiers, so we don’t have to put column and table names in quotations.	true
limits.application.datasource.hikari.connectionTimeout	Maximum timeout in milliseconds user will wait for a connection from connection pool.	30000
limits.application.datasource.hikari.idleTimeout	Maximum time a connection can remain idle in connection pool.	60000
limits.application.datasource.hikari.minimumIdle	Minimum number of idle connections in a connection pool.	1
limits.application.datasource.hikari.maximumPoolSize	Maximum size of the connection pool, including both idle and in-use connections.	10
limits.application.datasource.hikari.poolName	Name of the connection pool.	limits-application
application.datasource	Properties prefixed by this are related to the datasource for the limits workflow processes
application.datasource.url	Connection url	jdbc:h2:mem:limits-application;DB_CLOSE_DELAY=-1
application.datasource.username	Connection username	app
application.datasource.password	Connection password
application.datasource.properties.hibernate.dialect	Hibernate dialect	org.hibernate.dialect.H2Dialect
application.datasource.properties.hibernate.format_sql	Enables formatting of SQL logged to the console.	false
application.datasource.properties.hibernate.hbm2ddl.auto	Setting for how Spring should handle the database table on startup. Potential values include: create, create_drop, none, validate, update, drop, validate and truncate.	update
application.datasource.properties.hibernate.globally_quoted_identifiers	Escapes all database identifiers, so we don’t have to put column and table names in quotations.	true
application.datasource.hikari.connectionTimeout	Maximum timeout in milliseconds user will wait for a connection from connection pool.	30000
application.datasource.hikari.idleTimeout	Maximum time a connection can remain idle in connection pool.	60000
application.datasource.hikari.minimumIdle	Minimum number of idle connections in a connection pool.	1
application.datasource.hikari.maximumPoolSize	Maximum size of the connection pool, including both idle and in-use connections.	10
application.datasource.hikari.poolName	Name of the connection pool.	application-process-instance
limits.structure.templates	A map of LimitStructureTemplate definitions by templateName. See Adding Custom Limit Structure Templates	see application.yml
spring.h2.console.enabled	True if the H2 console is available. This is useful for investigating JDBC connections.	false
spring.data.rest.base-path	The base URL for Spring Data REST services which are autoconfigured for JPA repositories.	limits/rest/v2/spring/jpa
spring.liquibase.enabled	True if Liquibase database schema migrations listed in limits-starter/src/main/resources/liquibase/master-changelog.yaml should be applied on startup.	false
spring.liquibase.change-log	The location of the changelog file if Liquibase database schema migrations are applied on startup.	classpath:/liquibase/master-changelog.yaml

Files Deleted

The following *.properties property files have been deleted. Properties still in use have been moved to @ConfigurationProperties classes or application.yml:

• env-default.properties
• hibernate.properties
• jwt.properties
• limits.properties
• limits_test.properties
• tracing.properties

Datastores

Modified stores

Modification	Store	Field	Type	Description
Deleted	Limit Structures	UserId	String	The field has been deleted. It contained the ID or name of the user who created a given limit structure.
Renamed	Limit Structures	Reference ID	string	Reference ID has been renamed to Structure ID.
Deleted	Limit Structures	structureKey	string	structureKey has been deleted. Structure ID will be used instead.
Renamed	Limits	Reference ID	string	Reference ID has been renamed to Structure ID.
Renamed	Limits	parentLimitKey	string	parentLimitKey has been renamed to Source Limit ID.
Deleted	Limits	limitKey	string	limitKey has been deleted. Limit ID will be used instead.
Deleted	Limits	structureKey	string	structureKey has been deleted. Structure ID will be used instead.
Deleted	Limits	Limit Name	string	The unused Limit Name column has been deleted.
Renamed	Incidents	incidentKey	string	incidentKey has been renamed to Incident ID.
Renamed	Incidents	limitKey	string	limitKey has been renamed to Limit ID.

Cube schema

The limitKey hierarchy has been renamed to limitId.

Measures

No changes.

Context values

No changes.

Other changes

Changes to ILimitsProcessInstanceWorkflowService

If you implement a custom ILimitsProcessInstanceWorkflowService, please implement the new decorateWorkflowObject method. This is intended to decorate the LimitDTO and IncidentDTO objects with the values of workflow variables. You will also need to modify the update method to the new signature which includes the additional publishTuples and refreshKpis parameters.

Custom data-loading

These classes were moved from the limits-starter module to the limits-activeviam module:

• LimitsDlcConfigurationProperties
• CSVSourceConfig
• ACSVSourceConfig
• DataLoadControllerConfig
• InitialLoad
• LimitTuplePublisher
• AsOfDateTuplePublisher
• IncidentTuplePublisher

If you have customized any of these classes, you need to migrate your code. Interfaces have been provided to override all classes listed above exceptLimitsDlcConfigurationProperties and InitialLoad.

Additional resources

See the following resources on add custom data-loading logic:

• Custom Tuple Publisher: overriding the default Tuple Publishers.
• Custom Data-Load Controller Config: overriding the default Data Load Controller config bean.
• Custom ICSVSourceConfig: overriding the CSV source config.