Migration notes 3.3-BETA
This page explains the changes required to migrate to the stated version of the Atoti Limits.
Migrate to 3.3.0-BETA
Upgrading from version 3.2.0, see the Atoti Limits 3.3.0-BETA Release Notes.
Atoti Limits is using Atoti Server 6.0.13-sb3 and Atoti UI 5.1.x.
For new features and fixes included in these releases, please see the Atoti UI documentation and Atoti UI Migration Notes, and the release notes for Atoti Server.
Headline announcement
- Persistence : Atoti Limits can now work with persistent data sources. See the migration guide for more information on how to migrate your limits.
- Migration to Configuration Properties : All properties previously retrieved via
@Value
annotations have been migrated to@ConfigurationProperties
classes. For details, see Configuration Properties. - Custom validators and validation error handling:
IValidator
subinterfaces have been created to make swapping the default validator(s) easier for a custom solution.IValidationError
andIValidationErrorHandler
have been created to allow for creating custom errors and custom behavior when handling validation errors. See Adding Custom Validation for more details and information on how to implement these. - Validation error reporting: The default behavior for validation errors has been updated to produce a summary of the validation errors that occurred, including line numbers if the source is a file.
- DLC Configuration Properties : Data Load Controller property names have been updated, and are now handled by Spring via
@ConfigurationProperties
. See DLC Configuration Properties. - Custom data-loading : Interfaces have been introduced to facilitate overriding the default Data-Load Controller and CSV Source configuration beans; see Custom Data-Load Controller Config and see Custom CSV Source Config for more information. Interfaces were also added to facilitate overriding the default Tuple Publisher implementations; see Custom Tuple Publisher for more information.
- Atoti Server upgrade : Atoti Limits has been upgraded to Atoti Server 6.0.13-sb3.
Breaking changes
ConfigurationProperties
New properties have been added as Spring @ConfigurationProperties
with defaults. This means that the properties may not appear in the application.yml but have a value. The value can be overridden by adding your own value in the application.yml file.
Datasource configurations
We have updated the default datasource configurations for the limits application
, activiti
, the audit log
and the workflow processes
to point to the same limits-application
datasource.
Key field type changes
The following class fields have been converted from integers to strings. This is in preparation for a change to using the default reference IDs in the CSV files if provided. This will not impact you unless you have custom code that types these fields.
LimitStructureDTO::limitStructureKey
LimitDTO::limitKey
LimitDTO::parentLimitKey
IncidentDTO::incidentKey
IncidentDTO::limitKey
Key generation
Previously a hash function was used to generate the key fields mentioned in the previous section. This could result in hash collisions.
We now use an auto-incrementing IIdentityGenerator
to generate these IDs. The classes in the previous section now implement a getDeterminator
function that returns a unique string for each object.
Input file formats
Modified
Modification | File | Field | Optional | Description |
---|---|---|---|---|
Removed field | limit_structures.csv | User ID |
N | User ID has been removed from all input files. Please remove the User ID column from all limit_structures.csv files. |
Configuration
Configuration properties
Properties modified
Updated property names:
DLC Configuration Properties
Old Property Name | New Property Name | Description | Default Value |
---|---|---|---|
input.data.root.dir.path | limits.dlc.root-dir | Directory path to the data files. | ./src/test/resources/data-samples |
csvSource.subdirectory.dataset | limits.dlc.sub-directory-paths.root | Path extension to specific server data directories. | /data |
csvSource.subdirectory.dataset.whatif | limits.dlc.sub-directory-paths.whatif | Path extension to directory containing the What-If CSV files. | /whatif |
csvSource.subdirectory.dataset.stage | limits.dlc.sub-directory-paths.stage | Path extension to directory containing the Stage CSV files. | /stage |
default.csvSource.parser.threads | limits.dlc.parser-threads | Specifies the number of threads to be used for data loading, used for configuring the DlcCSVSourceConfiguration . |
4 |
default.csvSource.buffer.size | limits.dlc.buffer-size | CSV buffer size in KB, used for configuring the DlcCSVSourceConfiguration . |
1024 |
as.of.date.file.path.matcher | limits.dlc.path-matchers.asofdate | Regular expression to match as_of_date.csv files. |
glob:**/*as_of_date*.csv |
limits.definitions.file.path.matcher | limits.dlc.path-matchers.limitstructures | Regular expression to match limit_structures.csv files. |
glob:**/*limit_structures*.csv |
limits.approve.parameters.file.path.matcher | limits.dlc.path-matchers.limits | Regular expression to match limits_approve.csv files. |
glob:**/*limits_approve*.csv |
alerts.definitions.file.path.matcher | limits.dlc.path-matchers.incidents | Regular expression to match incident.csv files. |
glob:**/*incident*.csv |
Limits Properties
Old Property Name | New Property Name | Description | Default Value |
---|---|---|---|
limits.rest.url | limits.rest-url | The URL where the REST APIs of Atoti Limits are located | http://localhost:3090/limits/rest/v2/ |
Connected Atoti Server Properties
The following properties are set on the connected Atoti server and are used to configure the connection to Atoti Limits.
Old Property Name | New Property Name | Description | Default Value |
---|---|---|---|
ap.url | limits-connected-server.url | The URL of the Atoti server’s cube discovery | |
ap.auth | limits-connected-server.auth | The base-64 encoded authentication used to auto-configure the authorization for the module to connect to the Atoti server. | |
ap.auth.username | limits-connected-server.username | The username used to auto-configure the authorization for Atoti Limits to connect to the Atoti server. | |
ap.configuration.kpi.path | limits-connected-server.kpi-path | The path to get the KPI permissions. | |
ap.asOfDate.dim | limits-connected-server.as-of-date | The slicing date dimension present in the Atoti Server cube. Notation is Level@Hierarchy@Dimension. | |
level-path.url | limits-connected-server.level-path-url | The URL of the level path rest service. | |
limits.connect.fixed.delay | limits.connection.delay | The time delay in milliseconds between consecutive attempts to connect to the module. | 10000 |
limits.connect.attempts | limits.connection.attempts | The number of attempts to connect to the module before quitting. If less than zero, the Atoti server will continuously try to connect. The Atoti server will also try to reconnect if Atoti Limits is stopped and restarted. | -1 |
auto-config.enabled | limits.auto-config-enabled | If false, the auto-configuration methods will not be fired. | true |
atoti.server.version | limits-connected-server.atoti-server-version | The version of Atoti Server the connected server is running. |
The following properties are used by Atoti Limits to manage connections to connected Atoti servers. These can be auto-configured if limits.auto-config-enabled
is true. If you manually configure the connection between Atoti Limits and a connected Atoti server, you will need to update these properties.
Old Property Name | New Property Name | Description | Default Value |
---|---|---|---|
ap.url.map | limits.connected-server.url.map | Map of the Solution name to http url. | |
ap.version.map | limits.connected-server.version.map | Map of the Solution name to its server version: { < server name >: < version >, … }. | |
ap.configuration.kpi.path.map | limits.connected-server.kpi-path.map | Atoti Server configuration KPI path to get the KPI permissions used during startup to delete the stale KPIs | |
ap.auth.map | limits.connected-server.auth.map | Map of the Solution name to authentication token. | |
ap.asOfDate.dim.map | limits.connected-server.as-of-date.map | Map of the Solution name to AsOfDate cube location. | |
level-path.url | limits.connected-server.level-path.map | Map of the Solution name to Level Path REST endpoint URL. |
CSV Topic Configuration Properties
Old Property Name | New Property Name | Description | Default Value |
---|---|---|---|
default.topic.accept.incomplete.lines | limits.topic.csv.incomplete-lines | Specifies whether incomplete CSV lines are accepted. | false |
default.topic.accept.overflow.lines | limits.topic.csv.overflow-lines | Specifies whether overflowing CSV lines are accepted. | false |
default.topic.parser.process.quotes | limits.topic.csv.process-quotes | If true , the parser will parse values in double quotes. Otherwise, quotes are treated as literals |
true |
default.topic.parser.column.separator | limits.topic.csv.column-separator | Parser column separator. | ‘,’ |
default.topic.file.number.skipped.lines | limits.topic.csv.skipped-lines | The number of lines to skip at the start of the file. | 1 |
Activiti Mail Server Properties
Old Property Name | New Property Name | Description | Default Value |
---|---|---|---|
activiti.mail.host | limits.activiti.mail.host | URL of the mail server. | smtp.gmail.com |
activiti.mail.port | limits.activiti.mail.port | Port of the mail server. | 465 |
activiti.mail.username | limits.activiti.mail.username | username to use for authentication to the mail server. | |
activiti.mail.password | limits.activiti.mail.password | password to use for authentication to the mail server. | |
activiti.mail.ssl | limits.activiti.mail.ssl | Use SSL when connecting to the mail server. | true |
activiti.mail.tls | limits.activiti.mail.tls | Use TLS when connection to the mail server. | true |
activiti.mail.breach.url | limits.activiti.mail.breach-url | The URL where the raw binary data can be fetched from. | http://localhost:3000/#/dashboard/d46 |
Content Server Properties
These properties are auto-configured if limits.auto-config-enabled
is true. If you manually configure the connection between Limits and a connected Atoti server, you will need to update these properties.
Old Property Name | New Property Name | Description | Default Value |
---|---|---|---|
content.server.url | limits.content.server.url | URL of the content server to be used with Limits (and Limit’s UI). It is usually exposed at the server’s /content endpoint. | |
content.server.auth | limits.content.server.auth | The Base64 Encoded Authentication to use to authenticate requests to the Content Server. The Authentication should be for admin access and the passcode should be in the format USER_NAME:PASSWORD | |
content.server.auth.username | limits.content.server.username | The username used to auto-configure the authorization for the module to connect to the Atoti Content server | |
content.server.name | limits.content.server.name | The name of the content server to which Limits is connecting. |
Incident Properties
Old Property Name | New Property Name | Description | Default Value |
---|---|---|---|
incident.output.data.dir.path | limits.incident.output-data-dir | The output directory for incident files generated after an evaluation. | ./src/test/resources/data-samples/data |
incident.evaluation.parallel.tasks | limits.incident.evaluation.parallel-tasks | Number of threads to use when evaluating limits on-demand. | 10 |
limit.evaluation.include-passes | limits.incident.evaluation.include-passes | If true , Atoti Limits stores passes (i.e. evaluations that do not result in a breach or warning) in the datastore during limit evaluation. |
false |
limit.evaluation.error.include-stacktrace | limits.incident.evaluation.error.include-stacktrace | true if the stack trace should be available in the evaluation response to the UI. |
false |
Scope Properties
Old Property Name | New Property Name | Description | Default Value |
---|---|---|---|
scope.forceUnambiguousLevelPaths | limits.scope.force-unambiguous-level-paths | Boolean flag to enable if we want to force Scopes to be Unambiguous on Multi-Level Hierarchies. Please see limits.scope.force-unambiguous-level-paths for more. | false |
scope.autofill-unambiguous-level-paths | limits.scope.autofill-unambiguous-level-paths | Boolean flag to enable if we want to autofill Scopes to its Unambiguous Multi-Level values. Please see limits.scope.autofill-unambiguous-level-paths for more. | true |
scope.isUsingExplicitMatching | limits.scope.is-using-explicit-matching | Flag to enable explicit scope matching. If true, a limit will only be visible in the business cube at the exact location the limit’s scope is defined on and no lower. This does not affect the location evaluations and breaches are checked at. | true |
Miscellaneous Properties
Old Property Name | New Property Name | Description | Default Value |
---|---|---|---|
sample-rate-cron | limits.alert-task.sample-rate-cron | Cron expression for Intraday Alert task’s cron job. Default is to run the evaluation every 5 minutes. An example cron expression tool: https://www.freeformatter.com/cron-expression-generator-quartz.html. Note that cron expressions are evaluated differently in Java compared to Linux. See this explanation by Baeldung. We require the Spring cron syntax i.e. 6 space-separated fields. | 0 */5 * * * * |
task-scheduler-thread-pool-size | limits.task-scheduler.thread-pool-size | Thread pool size for task scheduler | 4 |
canTheApproverBeTheSameAsTheCreator | limits.workflow-rules.can-approver-be-same-as-creator | If true, the creator of a limit may also approve it. | false |
discovery-manager.polling-delay | limits.discovery-manager.polling-delay | Milliseconds that the connected cube description is valid for before needing to be refreshed. If the cube is expected to change frequently, this should be decreased. | 30000 |
Property files
Files Modified
application.yml
New properties:
Property Name | Comment | Value |
---|---|---|
limits.application.datasource | Properties prefixed by this are related to the datasource for the limits application | |
limits.application.datasource.url | Connection url | jdbc:h2:mem:limits-application;DB_CLOSE_DELAY=-1; |
limits.application.datasource.username | Connection username | app |
limits.application.datasource.password | Connection password | |
limits.application.datasource.properties.hibernate.dialect | Hibernate dialect | org.hibernate.dialect.H2Dialect |
limits.application.datasource.properties.hibernate.format_sql | Enables formatting of SQL logged to the console. | false |
limits.application.datasource.properties.hibernate.hbm2ddl.auto | Setting for how Spring should handle the database table on startup. Potential values include: create, create_drop, none, validate, update, drop, validate and truncate. | update |
limits.application.datasource.properties.hibernate.globally_quoted_identifiers | Escapes all database identifiers, so we don’t have to put column and table names in quotations. | true |
limits.application.datasource.hikari.connectionTimeout | Maximum timeout in milliseconds user will wait for a connection from connection pool. | 30000 |
limits.application.datasource.hikari.idleTimeout | Maximum time a connection can remain idle in connection pool. | 60000 |
limits.application.datasource.hikari.minimumIdle | Minimum number of idle connections in a connection pool. | 1 |
limits.application.datasource.hikari.maximumPoolSize | Maximum size of the connection pool, including both idle and in-use connections. | 10 |
limits.application.datasource.hikari.poolName | Name of the connection pool. | limits-application |
application.datasource | Properties prefixed by this are related to the datasource for the limits workflow processes | |
application.datasource.url | Connection url | jdbc:h2:mem:limits-application;DB_CLOSE_DELAY=-1 |
application.datasource.username | Connection username | app |
application.datasource.password | Connection password | |
application.datasource.properties.hibernate.dialect | Hibernate dialect | org.hibernate.dialect.H2Dialect |
application.datasource.properties.hibernate.format_sql | Enables formatting of SQL logged to the console. | false |
application.datasource.properties.hibernate.hbm2ddl.auto | Setting for how Spring should handle the database table on startup. Potential values include: create, create_drop, none, validate, update, drop, validate and truncate. | update |
application.datasource.properties.hibernate.globally_quoted_identifiers | Escapes all database identifiers, so we don’t have to put column and table names in quotations. | true |
application.datasource.hikari.connectionTimeout | Maximum timeout in milliseconds user will wait for a connection from connection pool. | 30000 |
application.datasource.hikari.idleTimeout | Maximum time a connection can remain idle in connection pool. | 60000 |
application.datasource.hikari.minimumIdle | Minimum number of idle connections in a connection pool. | 1 |
application.datasource.hikari.maximumPoolSize | Maximum size of the connection pool, including both idle and in-use connections. | 10 |
application.datasource.hikari.poolName | Name of the connection pool. | application-process-instance |
limits.structure.templates | A map of LimitStructureTemplate definitions by templateName. See Adding Custom Limit Structure Templates | see application.yml |
spring.h2.console.enabled | True if the H2 console is available. This is useful for investigating JDBC connections. | false |
spring.data.rest.base-path | The base URL for Spring Data REST services which are autoconfigured for JPA repositories. | limits/rest/v2/spring/jpa |
Files Deleted
The default Hibernate properties previously defined in hibernate.properties have been removed.
Datastores
Modified stores
Modification | Store | Field | Type | Description |
---|---|---|---|---|
Deleted | Limit Structures | UserId | String | The field has been removed. It contained the ID or name of the user who created a given limit structure. |
Modified | Limit Structures | structureKey | String | This field has been converted from an integer to a string. |
Modified | Limits | limitKey | String | This field has been converted from an integer to a string. |
Modified | Limits | structureKey | String | This field has been converted from an integer to a string. |
Modified | Limits | parentLimitKey | String | This field has been converted from an integer to a string. |
Modified | Incidents | incidentKey | String | This field has been converted from an integer to a string. |
Modified | Incidents | limitKey | String | This field has been converted from an integer to a string. |
Cube schema
No changes.
Measures
No changes.
Context values
No changes.
Other changes
Changes to IAlertTaskManager
If you have a custom implementation of IAlertTaskManager
, you need to add the postEvaluation
method to your implementation.
This method was added to handle post-evaluation tasks, such as handling validation errors or writing results to a file.
The method signature is as follows:
/**
* Operations to perform after the evaluation of all limits.
*
* @param incidents the incidents resulting from the evaluation
*/
void postEvaluation(Collection<IncidentDTO> incidents);
You will also need to remove the getIncidentCrudService
method as it is no longer used and has
been removed.
Changes to ILimitsProcessInstanceWorkflowService
If you implement a custom ILimitsProcessInstanceWorkflowService
, please implement the new decorateWorkflowObject
method.
This is intended to decorate the LimitDTO
and IncidentDTO
objects with the values of workflow variables.
Custom data-loading
These classes were moved from the limits-starter
module to the limits-activeviam
module:
LimitsDlcConfigurationProperties
CSVSourceConfig
ACSVSourceConfig
DataLoadControllerConfig
InitialLoad
LimitTuplePublisher
AsOfDateTuplePublisher
IncidentTuplePublisher
If you have customized any of these classes, you need to migrate your code. Interfaces have been provided to override all classes listed above exceptLimitsDlcConfigurationProperties
and InitialLoad
.
Additional resources
See the following resources on add custom data-loading logic:
- Custom Tuple Publisher: overriding the default Tuple Publishers.
- Custom Data-Load Controller Config: overriding the default Data Load Controller config bean.
- Custom ICSVSourceConfig: overriding the CSV source config.