☰
Navigation :

Migration notes 3.3-BETA

This page explains the changes required to migrate to the stated version of the Atoti Limits.

Migrate to 3.3.0-BETA

Upgrading from version 3.2.0, see the Atoti Limits 3.3.0-BETA Release Notes.

Atoti Limits is using Atoti Server 6.0.13-sb3 and Atoti UI 5.1.x.

For new features and fixes included in these releases, please see the Atoti UI documentation and Atoti UI Migration Notes, and the release notes for Atoti Server.

Headline announcement

  • Persistence : Atoti Limits can now work with persistent data sources. See the migration guide for more information on how to migrate your limits.
  • Migration to Configuration Properties : All properties previously retrieved via @Value annotations have been migrated to @ConfigurationProperties classes. For details, see Configuration Properties.
  • Custom validators and validation error handling: IValidator subinterfaces have been created to make swapping the default validator(s) easier for a custom solution. IValidationError and IValidationErrorHandler have been created to allow for creating custom errors and custom behavior when handling validation errors. See Adding Custom Validation for more details and information on how to implement these.
  • Validation error reporting: The default behavior for validation errors has been updated to produce a summary of the validation errors that occurred, including line numbers if the source is a file.
  • DLC Configuration Properties : Data Load Controller property names have been updated, and are now handled by Spring via @ConfigurationProperties. See DLC Configuration Properties.
  • Custom data-loading : Interfaces have been introduced to facilitate overriding the default Data-Load Controller and CSV Source configuration beans; see Custom Data-Load Controller Config and see Custom CSV Source Config for more information. Interfaces were also added to facilitate overriding the default Tuple Publisher implementations; see Custom Tuple Publisher for more information.
  • Atoti Server upgrade : Atoti Limits has been upgraded to Atoti Server 6.0.13-sb3.

Breaking changes

ConfigurationProperties

New properties have been added as Spring @ConfigurationProperties with defaults. This means that the properties may not appear in the application.yml but have a value. The value can be overridden by adding your own value in the application.yml file.

Datasource configurations

We have updated the default datasource configurations for the limits application, activiti, the audit log and the workflow processes to point to the same limits-application datasource.

Key field type changes

The following class fields have been converted from integers to strings. This is in preparation for a change to using the default reference IDs in the CSV files if provided. This will not impact you unless you have custom code that types these fields.

  • LimitStructureDTO::limitStructureKey
  • LimitDTO::limitKey
  • LimitDTO::parentLimitKey
  • IncidentDTO::incidentKey
  • IncidentDTO::limitKey

Key generation

Previously a hash function was used to generate the key fields mentioned in the previous section. This could result in hash collisions. We now use an auto-incrementing IIdentityGenerator to generate these IDs. The classes in the previous section now implement a getDeterminator function that returns a unique string for each object.

Input file formats

Modified

Modification File Field Optional Description
Removed field limit_structures.csv User ID N User ID has been removed from all input files. Please remove the User ID column from all limit_structures.csv files.

Configuration

Configuration properties

Properties modified

Updated property names:

DLC Configuration Properties
Old Property Name New Property Name Description Default Value
input.data.root.dir.path limits.dlc.root-dir Directory path to the data files. ./src/test/resources/data-samples
csvSource.subdirectory.dataset limits.dlc.sub-directory-paths.root Path extension to specific server data directories. /data
csvSource.subdirectory.dataset.whatif limits.dlc.sub-directory-paths.whatif Path extension to directory containing the What-If CSV files. /whatif
csvSource.subdirectory.dataset.stage limits.dlc.sub-directory-paths.stage Path extension to directory containing the Stage CSV files. /stage
default.csvSource.parser.threads limits.dlc.parser-threads Specifies the number of threads to be used for data loading, used for configuring the DlcCSVSourceConfiguration. 4
default.csvSource.buffer.size limits.dlc.buffer-size CSV buffer size in KB, used for configuring the DlcCSVSourceConfiguration. 1024
as.of.date.file.path.matcher limits.dlc.path-matchers.asofdate Regular expression to match as_of_date.csv files. glob:**/*as_of_date*.csv
limits.definitions.file.path.matcher limits.dlc.path-matchers.limitstructures Regular expression to match limit_structures.csv files. glob:**/*limit_structures*.csv
limits.approve.parameters.file.path.matcher limits.dlc.path-matchers.limits Regular expression to match limits_approve.csv files. glob:**/*limits_approve*.csv
alerts.definitions.file.path.matcher limits.dlc.path-matchers.incidents Regular expression to match incident.csv files. glob:**/*incident*.csv
Limits Properties
Old Property Name New Property Name Description Default Value
limits.rest.url limits.rest-url The URL where the REST APIs of Atoti Limits are located http://localhost:3090/limits/rest/v2/
Connected Atoti Server Properties

The following properties are set on the connected Atoti server and are used to configure the connection to Atoti Limits.

Old Property Name New Property Name Description Default Value
ap.url limits-connected-server.url The URL of the Atoti server’s cube discovery
ap.auth limits-connected-server.auth The base-64 encoded authentication used to auto-configure the authorization for the module to connect to the Atoti server.
ap.auth.username limits-connected-server.username The username used to auto-configure the authorization for Atoti Limits to connect to the Atoti server.
ap.configuration.kpi.path limits-connected-server.kpi-path The path to get the KPI permissions.
ap.asOfDate.dim limits-connected-server.as-of-date The slicing date dimension present in the Atoti Server cube. Notation is Level@Hierarchy@Dimension.
level-path.url limits-connected-server.level-path-url The URL of the level path rest service.
limits.connect.fixed.delay limits.connection.delay The time delay in milliseconds between consecutive attempts to connect to the module. 10000
limits.connect.attempts limits.connection.attempts The number of attempts to connect to the module before quitting. If less than zero, the Atoti server will continuously try to connect. The Atoti server will also try to reconnect if Atoti Limits is stopped and restarted. -1
auto-config.enabled limits.auto-config-enabled If false, the auto-configuration methods will not be fired. true
atoti.server.version limits-connected-server.atoti-server-version The version of Atoti Server the connected server is running.

The following properties are used by Atoti Limits to manage connections to connected Atoti servers. These can be auto-configured if limits.auto-config-enabled is true. If you manually configure the connection between Atoti Limits and a connected Atoti server, you will need to update these properties.

Old Property Name New Property Name Description Default Value
ap.url.map limits.connected-server.url.map Map of the Solution name to http url.
ap.version.map limits.connected-server.version.map Map of the Solution name to its server version: { < server name >: < version >, … }.
ap.configuration.kpi.path.map limits.connected-server.kpi-path.map Atoti Server configuration KPI path to get the KPI permissions used during startup to delete the stale KPIs
ap.auth.map limits.connected-server.auth.map Map of the Solution name to authentication token.
ap.asOfDate.dim.map limits.connected-server.as-of-date.map Map of the Solution name to AsOfDate cube location.
level-path.url limits.connected-server.level-path.map Map of the Solution name to Level Path REST endpoint URL.
CSV Topic Configuration Properties
Old Property Name New Property Name Description Default Value
default.topic.accept.incomplete.lines limits.topic.csv.incomplete-lines Specifies whether incomplete CSV lines are accepted. false
default.topic.accept.overflow.lines limits.topic.csv.overflow-lines Specifies whether overflowing CSV lines are accepted. false
default.topic.parser.process.quotes limits.topic.csv.process-quotes If true, the parser will parse values in double quotes. Otherwise, quotes are treated as literals true
default.topic.parser.column.separator limits.topic.csv.column-separator Parser column separator. ‘,’
default.topic.file.number.skipped.lines limits.topic.csv.skipped-lines The number of lines to skip at the start of the file. 1
Activiti Mail Server Properties
Old Property Name New Property Name Description Default Value
activiti.mail.host limits.activiti.mail.host URL of the mail server. smtp.gmail.com
activiti.mail.port limits.activiti.mail.port Port of the mail server. 465
activiti.mail.username limits.activiti.mail.username username to use for authentication to the mail server.
activiti.mail.password limits.activiti.mail.password password to use for authentication to the mail server.
activiti.mail.ssl limits.activiti.mail.ssl Use SSL when connecting to the mail server. true
activiti.mail.tls limits.activiti.mail.tls Use TLS when connection to the mail server. true
activiti.mail.breach.url limits.activiti.mail.breach-url The URL where the raw binary data can be fetched from. http://localhost:3000/#/dashboard/d46
Content Server Properties

These properties are auto-configured if limits.auto-config-enabled is true. If you manually configure the connection between Limits and a connected Atoti server, you will need to update these properties.

Old Property Name New Property Name Description Default Value
content.server.url limits.content.server.url URL of the content server to be used with Limits (and Limit’s UI). It is usually exposed at the server’s /content endpoint.
content.server.auth limits.content.server.auth The Base64 Encoded Authentication to use to authenticate requests to the Content Server. The Authentication should be for admin access and the passcode should be in the format USER_NAME:PASSWORD
content.server.auth.username limits.content.server.username The username used to auto-configure the authorization for the module to connect to the Atoti Content server
content.server.name limits.content.server.name The name of the content server to which Limits is connecting.
Incident Properties
Old Property Name New Property Name Description Default Value
incident.output.data.dir.path limits.incident.output-data-dir The output directory for incident files generated after an evaluation. ./src/test/resources/data-samples/data
incident.evaluation.parallel.tasks limits.incident.evaluation.parallel-tasks Number of threads to use when evaluating limits on-demand. 10
limit.evaluation.include-passes limits.incident.evaluation.include-passes If true, Atoti Limits stores passes (i.e. evaluations that do not result in a breach or warning) in the datastore during limit evaluation. false
limit.evaluation.error.include-stacktrace limits.incident.evaluation.error.include-stacktrace true if the stack trace should be available in the evaluation response to the UI. false
Scope Properties
Old Property Name New Property Name Description Default Value
scope.forceUnambiguousLevelPaths limits.scope.force-unambiguous-level-paths Boolean flag to enable if we want to force Scopes to be Unambiguous on Multi-Level Hierarchies. Please see limits.scope.force-unambiguous-level-paths for more. false
scope.autofill-unambiguous-level-paths limits.scope.autofill-unambiguous-level-paths Boolean flag to enable if we want to autofill Scopes to its Unambiguous Multi-Level values. Please see limits.scope.autofill-unambiguous-level-paths for more. true
scope.isUsingExplicitMatching limits.scope.is-using-explicit-matching Flag to enable explicit scope matching. If true, a limit will only be visible in the business cube at the exact location the limit’s scope is defined on and no lower. This does not affect the location evaluations and breaches are checked at. true
Miscellaneous Properties
Old Property Name New Property Name Description Default Value
sample-rate-cron limits.alert-task.sample-rate-cron Cron expression for Intraday Alert task’s cron job. Default is to run the evaluation every 5 minutes. An example cron expression tool: https://www.freeformatter.com/cron-expression-generator-quartz.html. Note that cron expressions are evaluated differently in Java compared to Linux. See this explanation by Baeldung. We require the Spring cron syntax i.e. 6 space-separated fields. 0 */5 * * * *
task-scheduler-thread-pool-size limits.task-scheduler.thread-pool-size Thread pool size for task scheduler 4
canTheApproverBeTheSameAsTheCreator limits.workflow-rules.can-approver-be-same-as-creator If true, the creator of a limit may also approve it. false
discovery-manager.polling-delay limits.discovery-manager.polling-delay Milliseconds that the connected cube description is valid for before needing to be refreshed. If the cube is expected to change frequently, this should be decreased. 30000

Property files

Files Modified
application.yml

New properties:

Property Name Comment Value
limits.application.datasource Properties prefixed by this are related to the datasource for the limits application
limits.application.datasource.url Connection url jdbc:h2:mem:limits-application;DB_CLOSE_DELAY=-1;
limits.application.datasource.username Connection username app
limits.application.datasource.password Connection password
limits.application.datasource.properties.hibernate.dialect Hibernate dialect org.hibernate.dialect.H2Dialect
limits.application.datasource.properties.hibernate.format_sql Enables formatting of SQL logged to the console. false
limits.application.datasource.properties.hibernate.hbm2ddl.auto Setting for how Spring should handle the database table on startup. Potential values include: create, create_drop, none, validate, update, drop, validate and truncate. update
limits.application.datasource.properties.hibernate.globally_quoted_identifiers Escapes all database identifiers, so we don’t have to put column and table names in quotations. true
limits.application.datasource.hikari.connectionTimeout Maximum timeout in milliseconds user will wait for a connection from connection pool. 30000
limits.application.datasource.hikari.idleTimeout Maximum time a connection can remain idle in connection pool. 60000
limits.application.datasource.hikari.minimumIdle Minimum number of idle connections in a connection pool. 1
limits.application.datasource.hikari.maximumPoolSize Maximum size of the connection pool, including both idle and in-use connections. 10
limits.application.datasource.hikari.poolName Name of the connection pool. limits-application
application.datasource Properties prefixed by this are related to the datasource for the limits workflow processes
application.datasource.url Connection url jdbc:h2:mem:limits-application;DB_CLOSE_DELAY=-1
application.datasource.username Connection username app
application.datasource.password Connection password
application.datasource.properties.hibernate.dialect Hibernate dialect org.hibernate.dialect.H2Dialect
application.datasource.properties.hibernate.format_sql Enables formatting of SQL logged to the console. false
application.datasource.properties.hibernate.hbm2ddl.auto Setting for how Spring should handle the database table on startup. Potential values include: create, create_drop, none, validate, update, drop, validate and truncate. update
application.datasource.properties.hibernate.globally_quoted_identifiers Escapes all database identifiers, so we don’t have to put column and table names in quotations. true
application.datasource.hikari.connectionTimeout Maximum timeout in milliseconds user will wait for a connection from connection pool. 30000
application.datasource.hikari.idleTimeout Maximum time a connection can remain idle in connection pool. 60000
application.datasource.hikari.minimumIdle Minimum number of idle connections in a connection pool. 1
application.datasource.hikari.maximumPoolSize Maximum size of the connection pool, including both idle and in-use connections. 10
application.datasource.hikari.poolName Name of the connection pool. application-process-instance
limits.structure.templates A map of LimitStructureTemplate definitions by templateName. See Adding Custom Limit Structure Templates see application.yml
spring.h2.console.enabled True if the H2 console is available. This is useful for investigating JDBC connections. false
spring.data.rest.base-path The base URL for Spring Data REST services which are autoconfigured for JPA repositories. limits/rest/v2/spring/jpa

Files Deleted

The default Hibernate properties previously defined in hibernate.properties have been removed.

Datastores

Modified stores

Modification Store Field Type Description
Deleted Limit Structures UserId String The field has been removed. It contained the ID or name of the user who created a given limit structure.
Modified Limit Structures structureKey String This field has been converted from an integer to a string.
Modified Limits limitKey String This field has been converted from an integer to a string.
Modified Limits structureKey String This field has been converted from an integer to a string.
Modified Limits parentLimitKey String This field has been converted from an integer to a string.
Modified Incidents incidentKey String This field has been converted from an integer to a string.
Modified Incidents limitKey String This field has been converted from an integer to a string.

Cube schema

No changes.

Measures

No changes.

Context values

No changes.

Other changes

Changes to IAlertTaskManager

If you have a custom implementation of IAlertTaskManager, you need to add the postEvaluation method to your implementation. This method was added to handle post-evaluation tasks, such as handling validation errors or writing results to a file. The method signature is as follows:

/**
   * Operations to perform after the evaluation of all limits.
   *
   * @param incidents the incidents resulting from the evaluation
   */
  void postEvaluation(Collection<IncidentDTO> incidents);

You will also need to remove the getIncidentCrudService method as it is no longer used and has been removed.

Changes to ILimitsProcessInstanceWorkflowService

If you implement a custom ILimitsProcessInstanceWorkflowService, please implement the new decorateWorkflowObject method. This is intended to decorate the LimitDTO and IncidentDTO objects with the values of workflow variables.

Custom data-loading

These classes were moved from the limits-starter module to the limits-activeviam module:

  • LimitsDlcConfigurationProperties
  • CSVSourceConfig
  • ACSVSourceConfig
  • DataLoadControllerConfig
  • InitialLoad
  • LimitTuplePublisher
  • AsOfDateTuplePublisher
  • IncidentTuplePublisher

If you have customized any of these classes, you need to migrate your code. Interfaces have been provided to override all classes listed above exceptLimitsDlcConfigurationProperties and InitialLoad.

Additional resources

See the following resources on add custom data-loading logic: