Migrate to 6.1
It is strongly recommended to begin by reviewing the what's new page, which explains some of the migration steps described here.
Important It is also strongly recommended to begin by clearing every use of deprecated methods from the Atoti library before attempting to migrate. Indeed, these methods have been, for the most part, removed in the 6.1 version. It will prove much easier to perform these small migration steps while the project still compiles, and to test that it still performs as expected, before going to the next step of this migration to 6.1.
Project configuration changes
Atoti has been upgraded to:
- Java 21
- Spring 6
- Spring Boot 3.2 These updates are necessary in order to follow the ecosystem's release cycle, particularly when it comes to security updates.
Spring Modules
Considerable effort has gone into simplifying and streamlining the project configuration process in version 6.1.
Spring Boot Starters have been developed for Atoti, allowing the import of themed Maven modules instead of manually consolidating numerous configuration classes in the application.
The starters represent an "override or opt-out" of the configurations philosophy, rather than an "opt-in by importing" one that was previously implemented. More details can be found in the Starters documentation.
Migration steps will be provided in subsequent sections of this page.
A considerable part of the Atoti modules that need to be imported may be replaced by one or two
starters, increasing the maintainability of the project's pom.xml
file.
Atoti Modules
After updating the project's .pom files to use this new version of Atoti Server, some dependencies will need to be modified to align with the changes in the Atoti Server modules.
Modules com.activeviam.tech:composer-impl
and com.activeviam.tech:composer-intf
have been merged
into a single com.activeviam.tech:composer-core
module.
The relationship between the datastore and the sources (CSV, JDBC) has been inverted. These sources now depend on the datastore, and not the other way around. To utilize a specific source, the corresponding artifact must be separately imported:
- CSV source:
com.activeviam.source:csv-source
- JDBC source:
com.activeviam.source:jdbc-source
- Parquet source:
com.activeviam:parquet-source
(not impacted by this change, it already had to be imported separately)
This inversion will reduce the size of the final jar
by cutting out the transitive dependencies of
the sources that the project does not require, and thus does not import.
Module com-activeviam-activepivot:activepivot-copper2-impl
has been renamed
com-activeviam-activepivot:activepivot-copper
.
Module com-activeviam-activepivot:activepivot-copper2-test
has been renamed
com-activeviam-activepivot:activepivot-copper-test
.
It only contains internal
and private_
packages, and should not be imported. A proper public
testing module has been created for 6.1:
com.activeviam:atoti-server-test
, and will be discussed later on in this page.
JungSchemaPrinter
has been moved to its own maven module
com.activeviam.tech:datastore-schema-printer
, in the package
com.activeviam.database.datastore.schemaprinter.api
.
Other Dependencies
With 6.1, most of the dependencies of Atoti Server have been upgraded to their latest versions. To stay up to date with the security updates of these dependencies, this often implied major version changes. It is recommended to remove any version number that have been manually set in the project pom, and find a dependency convergence once again during this upgrade to 6.1.
Atoti Server's version lifecycle being longer than most OSS version lifecycle, dependency upgrades will continue throughout the lifetime of version 6.1.
ActiveMonitor
ActiveMonitor no longer relies on org.apache.velocity.tools:velocity-tools-generic
.
This dependency allowed for dynamic message templates.
Users can elect to re-add such this extension by overriding the Spring Bean returning
VelocityTemplateEngine
, and setting their own extensions through setExtensions(...)
.
Example:
public class MessagingConfigurationOverride {
@Bean
public VelocityTemplateEngine velocityTemplateEngine() {
final VelocityTemplateEngine engine = new VelocityTemplateEngine();
final Map<String, Object> extensions = new HashMap<>();
extensions.put("numberTool", new NumberTool());
engine.setExtensions(extensions);
return engine;
}
private void usage() {
final VelocityTemplateEngine engine = new VelocityTemplateEngine();
final String template = "${number} as ${numberTool.integer($number)}";
final Map<String, Object> model = new HashMap<>();
model.put("number", 1.2);
engine.render(template, model); // Returns "1.2 as 1"
}
It is recommended to replace the ActiveMonitor product by the Limits product, which concentrates most of the development efforts.
Sources
Following the AWS end-of-support announcement for the Java SDK v1 effective December 31, 2025, the AWS cloud source has been updated to use the AWS Java SDK v2.
When using client side encryption, the metadata header x-amz-unencrypted-content-length
must be
updated to x-amz-meta-unencrypted-content-length
when accessing S3.
AWS recommends using the x-amz-meta-
prefix for custom metadata headers, as it will not conflict
with any other header they might add in the future. The AWS Cloud source has been updated to follow
this recommendation.
Apache Arrow has been bumped from version 12 to version 17.
Distribution
The JGroups dependency has been upgraded to the latest version available (5.3).
List of noticeable changes in JGroups XML config files:
max_bundle_size
property does not exist anymoreenable_diagnostics
property does not exist anymoreuse_fork_join_pool
property does not exist anymorepbcast.STABLE.stability_delay
property does not exist anymoreFD
protocol does not exist anymore. One may useFD_ALL
andFD_HOST
instead, as described here- in
AUTH
, token parameters should now be prefixed withauth_token.
Also,MD5Token
does not exist anymore org.jgroups.aws.s3.NATIVE_S3_PING
protocol was renamed toaws.S3_PING
. See more info here
Public and Internal APIs
After upgrading the maven modules, the next step of the migration is to be able to compile the project.
Most of the classes of the Atoti Server library have been moved to new packages, following the process described in the what's new page. The Java API migration tool can be used to facilitate the migration. It will automatically update all imports with the new packages and class names.
The migration tool's output will provide a list of all the private APIs that the project relies on after this automatic migration. This list should be kept somewhere. If it still has relevant entries after manually finishing this migration to 6.1, please do contact ActiveViam's support.
REST Services
The latest version of the Atoti REST API is 9.
Miscellaneous
Renamed elements
A lot of classes have been renamed as well as being moved to another packages. While the imports were automatically updated by the migration script, uses of these classes within the source code were not, for technical reasons.
Now that the imports compile properly, they can be used as the source of truth regarding how the code base should be migrated.
Acronyms
The following acronyms have been modified to follow new code style standards.
Previous Version | New Version |
---|---|
CSV | Csv |
JDBC | Jdbc |
POJO | Pojo |
Branding
Most, if not all, of the references to the previous company name, QuartetFS, also abbreviated QFS, have been removed.
As previously stated, the corresponding imports should have been automatically updated, making the changes to the code base rather straightforward.
Annotations used for code injection were re-branded:
Previous Version | New Version |
---|---|
QuartetExtendedPlugin | AtotiExtendedPlugin |
QuartetExtendedPluginValue | AtotiExtendedPluginValue |
QuartetPlugin | AtotiPlugin |
QuartetPluginValue | AtotiPluginValue |
QuartetType | AtotiType |
Moved and/or replaced properties
ActiveViam Properties
The class ActiveViamProperties
, which references all the application properties that can be set
to alter the configuration of the engine, has been updated:
Key | Previous Version | New Version | Comment |
---|---|---|---|
CHUNK_ALLOCATOR_KEY_PROPERTY | activeviam.chunkAllocatorClass | activeviam.chunkAllocatorKey | The values taken by this property were changed from class names to keys. The possible values are: "mmap", "array", "direct", "slab", "direct_buffer", "heap_buffer". "slab" is still the recommended and default. |
DATA_CUBE_REST_ENDPOINT_PORT_PROPERTY | activeviam.distribution.endpoint.port | See the changes to distribution properties | |
DATA_CUBE_REST_ENDPOINT_SUFFIX_PROPERTY | activeviam.distribution.endpoint.suffix | See the changes to distribution properties | |
DATA_CUBE_REST_ENDPOINT_HOST_PROPERTY | activeviam.distribution.endpoint.host | See the changes to distribution properties | |
DATA_CUBE_REST_ENDPOINT_PROTOCOL_PROPERTY | activeviam.distribution.endpoint.protocol | See the changes to distribution properties | |
EXTERNAL_DATABASE_QUERY_TIMEOUT_LIMIT | activeviam.directquery.externalDatabaseQueryTimeoutLimit | Database dependant. See classes com.activeviam.directquery.XXX.api.XXXClientSettings where XXX is a database product name. | |
ENABLE_DIRECTQUERY_AUTOVECTORIZER | activeviam.directquery.enableAutoVectorizer | com.activeviam.database.api.settings.DiscovererSettings | See also com.activeviam.directquery.api.schema.Vectorization and the how-to guide |
DIRECTQUERY_AUTOVECTORIZER_DELIMITER | activeviam.directquery.autoVectorizerDelimiter | com.activeviam.database.api.settings.DiscovererSettings | See also com.activeviam.directquery.api.schema.Vectorization and the how-to guide |
DIRECTQUERY_AUTOVECTORIZER_THRESHOLD | activeviam.directquery.autoVectorizerThreshold | com.activeviam.database.api.settings.DiscovererSettings | See also com.activeviam.directquery.api.schema.Vectorization and the how-to guide |
MANAGER_FEEDING_TIMEOUT | activeviam.directquery.cubeFeedingTimeoutInSeconds | Database dependant. See classes com.activeviam.directquery.XXX.api.XXXDatabaseSettings where XXX is a database product name. | |
DIRECTQUERY_AUTO_ADAPTIVE_JIT | activeviam.directquery.autoAdaptiveJit | Removed. | |
DIRECT_QUERY_SUB_QUERY_LIMIT | activeviam.directquery.subQueryLimit | Database dependant. See classes com.activeviam.directquery.XXX.api.XXXDatabaseSettings where XXX is a database product name. | |
DIRECT_QUERY_GET_BY_KEY_BEHAVIOR | activeviam.directquery.GetByKeyBehavior | Database dependant. See classes com.activeviam.directquery.XXX.api.XXXDatabaseSettings where XXX is a database product name. | |
SNOWFLAKE_MAX_RESULTSET_SIZE | activeviam.directquery.snowflake.maxresultsetsize | com.activeviam.directquery.snowflake.api.SnowflakeClientSettings | |
AGGRESSIVE_AXIS_POSITION_LIMIT_CHECK_PROPERTY | activeviam.mdx.result.aggresiveAxisPositionLimitCheck | Moved to the Context Value MdxContext to be configurable on a per-query basis |
Application Properties
activeviam.jwt.*
properties have been renamed into atoti.jwt.*
properties. Moreover, two properties have been
renamed:
Previous Version | New Version |
---|---|
activeviam.jwt.generate | atoti.jwt.enabled |
activeviam.jwt.check.user_details | atoti.jwt.check_user_details |
Static constants
Previous Version | New Version | Comment |
---|---|---|
com.activeviam.copper.ProviderCoordinate#GLOBAL_PROVIDER_NAME | com.activeviam.activepivot.core.intf.api.description.IAggregateProviderDefinition#GLOBAL_PROVIDER_NAME | |
com.quartetfs.biz.pivot.query.aggregates.impl.StoredMeasureHandler#PLUGIN_TYPE | com.activeviam.activepivot.core.intf.api.realtime.IAggregatesContinuousHandler#BASIC_HANDLER_PLUGIN_KEY | Handler is now internal. |
com.quartetfs.biz.pivot.query.aggregates.impl.StoredPrimitiveMeasureHandler#PLUGIN_TYPE | Handler is now internal. Re-implement IAggregatesContinuousHandler if needed. | |
com.quartetfs.biz.pivot.query.aggregates.impl.TransactionStreamFullRefreshHandler#PLUGIN_TYPE | com.activeviam.activepivot.core.intf.api.realtime.IAggregatesContinuousHandler#FORCE_FULL_REFRESH_PLUGIN_KEY | Handler is now internal. |
com.quartetfs.biz.pivot.query.aggregates.impl.MultiAnalysisHierarchyMeasureHandler#PLUGIN_TYPE | com.activeviam.activepivot.core.intf.api.realtime.IAggregatesContinuousHandler#MULTI_ANALYSIS_HIERARCHY_MEASURE_HANDLER | |
com.quartetfs.biz.pivot.query.aggregates.impl.CommitIsolatedStoreHandler#PLUGIN_TYPE | com.activeviam.activepivot.core.intf.api.realtime.IAggregatesContinuousHandler#createCommitOnIsolatedStoreHandlerPluginKey(String) | CommitIsolatedStoreHandler is a variable plugin value. |
com.quartetfs.biz.pivot.query.aggregates.impl.TransactionStream#PLUGIN_TYPE | com.activeviam.activepivot.core.intf.api.realtime.IStream#ACTIVEPIVOT_PLUGIN_KEY | TransactionStream is internal. |
com.quartetfs.biz.pivot.query.aggregates.impl.CommitStoreStream#PLUGIN_TYPE | com.activeviam.activepivot.core.intf.api.realtime.IStream#createCommitOnStoreStreamPluginKey(String) | CommitStoreStream is variable plugin value. |
com.qfs.messenger.impl.LocalMessenger#PLUGIN_KEY | com.activeviam.activepivot.core.intf.api.description.IMessengerDefinition#LOCAL_PLUGIN_KEY | |
com.qfs.messenger.impl.NettyMessenger#PLUGIN_KEY | com.activeviam.activepivot.core.intf.api.description.IMessengerDefinition#NETTY_PLUGIN_KEY | |
com.quartetfs.biz.pivot.cube.hierarchy.IAnalysisHierarchy#LEVEL_TYPES_PROPERTY | com.activeviam.activepivot.core.intf.api.description.IAxisLevelDescription#ANALYSIS_LEVEL_TYPE_PROPERTY | Used on a per-level basis instead of a per-hierarchy basis |
com.quartetfs.biz.pivot.cube.hierarchy.ILevel#AXIS | com.activeviam.activepivot.core.intf.api.cube.hierarchy.IHierarchy#AXIS | |
com.quartetfs.biz.pivot.cube.hierarchy.ILevel#BRANCH_LEVEL_NAME | com.activeviam.activepivot.core.intf.api.cube.hierarchy.IHierarchy#BRANCH_LEVEL_NAME | |
com.quartetfs.biz.pivot.cube.hierarchy.ILevel#EPOCH_LEVEL_NAME | com.activeviam.activepivot.core.intf.api.cube.hierarchy.IHierarchy#EPOCH_LEVEL_NAME | |
com.quartetfs.biz.pivot.cube.hierarchy.ILevel#ALLMEMBER | com.activeviam.activepivot.core.intf.api.cube.hierarchy.IHierarchy#ALLMEMBER | |
com.quartetfs.biz.pivot.cube.hierarchy.axis.impl.AxisHierarchyBase#AUTO_CONTRIBUTE_UNKNOWN_MEMBER_PROPERTY | com.activeviam.activepivot.core.intf.api.description.IAxisHierarchyDescription#AUTO_CONTRIBUTE_UNKNOWN_MEMBER_PROPERTY | |
com.quartetfs.biz.pivot.cube.hierarchy.axis.impl.AxisHierarchyBase#AUTO_CONTRIBUTE_UNKNOWN_MEMBER_ALWAYS | com.activeviam.activepivot.core.intf.api.description.IAxisHierarchyDescription#AUTO_CONTRIBUTE_UNKNOWN_MEMBER_ALWAYS | |
com.quartetfs.biz.pivot.cube.hierarchy.axis.impl.AxisHierarchyBase#AUTO_CONTRIBUTE_UNKNOWN_MEMBER_IF_EMPTY | com.activeviam.activepivot.core.intf.api.description.IAxisHierarchyDescription#AUTO_CONTRIBUTE_UNKNOWN_MEMBER_IF_EMPTY | |
com.quartetfs.biz.pivot.cube.hierarchy.axis.impl.AxisHierarchyBase#AUTO_CONTRIBUTE_UNKNOWN_MEMBER_NEVER | com.activeviam.activepivot.core.intf.api.description.IAxisHierarchyDescription#AUTO_CONTRIBUTE_UNKNOWN_MEMBER_NEVER |
Spring Boot Starters
As mentioned in a previous section, Version 6.1 has introduced Spring Boot
Starters to ease the configuration of an Atoti project. These starters offer a default configuration
for an Atoti project.
A considerable part of the org.springframework.context.annotation.Configuration
objects imported
in a 6.0 project can be removed from the org.springframework.context.annotation.Import
of the
application configuration.
Services
ActivePivotXmlaServletConfig
has been completely removed. The underlying servlet has been
transformed into a fully fledged REST controller. Importing the starter
com.activeviam.springboot:atoti-server-starter
will automatically configure the XMLA endpoint.
Extensions of ActivePivotXmlaServletConfig
should not be migrated. All configuration options have
been ported to Spring properties.
LocalI18nConfig
has been completely removed. Importing the starter
com.activeviam.springboot:atoti-server-starter
will automatically configure internationalization
to use the file mentioned in the i18n
sub folder of the resource folder.
Importing the starters com.activeviam.springboot:atoti-ui-starter
and
com.activeviam.springboot:atoti-admin-ui-starter
will automatically setup Atoti UI and Atoti Admin
UI. ActiveUIResourceServerConfig
and AdminUIResourceServerConfig
no longer need to be imported.
ActiveViamPropertyFromSpringConfig
, used to resolve ActiveViamProperties
from Spring properties,
no longer needs to be imported. Importing the starter
com.activeviam.springboot:atoti-server-starter
will automatically forward the Spring properties
to the Atoti project.
Importing the starter com.activeviam.springboot:atoti-server-starter
will automatically configure
ActivePivotServicesConfig
. It no longer needs to be imported.
Finally, the starter will automatically import all the REST services that can be exposed with an
Atoti project. Classes like ActiveViamRestServicesConfig
, ActivePivotRestServicesConfig
or
ActivePivotWebSocketServicesConfig
no longer need to be imported.
Importing the starter com.activeviam.springboot:atoti-server-starter
will automatically provide
a basic configuration of the IDataExportService
. Builders are available for customization through
DataExportServiceBuilder
Database Service
The implementation of Database Service and its components have been reviewed to avoid exposing internal components. This resulted in changes in the way this service converts conditions and Update-Where operations.
They now operate directly on the JSON data, with the help of two side objects. One object providing some context to the operation, like the table field being processed. The other object provides convenience methods to transform data values from the JSON.
Converting a condition factory
As mentioned above, IConditionFactory
changed to receive the JSON data as well as the context of the condition and the helper object to transform data values.
As an example, the following code shows how to migrate a "greater than" condition.
Implementation in 6.0:
class ZeroCondition extends PluginValue implements IConditionFactory {
@Override
public String key() {
return "$gt";
}
@Override
public ICondition compile(
final String field, final JsonNode jsonNode, final JsonConditionCompiler jsonCompiler) {
// Extract the value from JSON using the configured parser for the table field
final Object value = jsonCompiler.convertValue(field, jsonNode);
// Create the equivalent database condition
final String[] path = field.split("/", -1);
return BaseConditions.greater(FieldPath.of(path), value);
}
}
Implementation in 6.1:
@AtotiPluginValue(intf = IConditionFactory.class)
class GreaterCondition implements IConditionFactory {
@Override
public String key() {
return "$gt";
}
@Override
public ICondition compile(
final JsonNode jsonValue,
final IConditionContext fieldExpression,
final IServiceJsonHelper jsonHelper) {
final var fieldDetails = fieldExpression.getTestedFieldDetails();
// Extract the value from JSON using the configured parser for the table field
final Object value = jsonHelper.readValue(fieldDetails.getTableFieldReference(), jsonValue);
// Create the equivalent database condition
return BaseConditions.greater(fieldDetails.getFieldReference(), value);
}
}
As seen in the conversion example, the JSON data remains.
The nullable String
representing the field tested by the condition is now described in the condition context. This object conveniently reports it as a FieldPath
, from the root table in the query, as well as the actual StoreField
in the database.
Finally, the compiler was replaced by a helper service focusing on parsing data from JSON.
A usage not visible in this example is the case of nested condition, like a not(equal(...))
. This can be achieved through the condition context, using the method IConditionContext#convertCondition
. This accepts the JSON data defining the nested condition as well as an optional FieldPath
, if operating on another path.
Converting an Update-Where procedure factory
As mentioned above, IUpdateWhereProcedureFactory
changed to receive the JSON data as well as the context of the procedure and the helper object to transform data values.
As an example, the following code defines a custom procedure changing the value of a table field when the value matches a given needle value.
Implementation in 6.0:
class NullIfProcedure extends PluginValue implements IUpdateWhereProcedureFactory {
@Override
public String key() {
return "$nullIf";
}
@Override
public UpdateWhereSelectionAndProcedure buildProcedure(
String ignored, JsonNode value, JsonUpdateWhereCompiler updatewherebuilder) {
final var testedField = value.get("field").asText();
final var needle = updatewherebuilder.convertValue(testedField, value);
return new UpdateWhereSelectionAndProcedure(
new IUpdateWhereProcedure() {
int position;
@Override
public void init(final IRecordFormat selectionFormat, final IRecordFormat recordFormat) {
this.position = recordFormat.getFieldIndex("target");
}
@Override
public void execute(final IArrayReader selectedRecord, final IArrayWriter recordWriter) {
final var value = selectedRecord.read(this.position);
if (needle.equals(value)) {
recordWriter.write(this.position, null);
}
}
},
new Selection(
updatewherebuilder.getTableName(),
List.of(AliasedField.create("target", DeprecatedDatabaseApi.toPath(field)))),
Set.of(testedField));
}
@Override
public void checkJsonNode(final JsonNode node) {}
}
Implementation in 6.1:
@AtotiPluginValue(intf = IUpdateWhereProcedureFactory.class)
class NullIfProcedure implements IUpdateWhereProcedureFactory {
@Override
public String key() {
return "$nullIf";
}
@Override
public UpdateWhereSelectionAndProcedure buildProcedure(
final JsonNode value, final IUpdateWhereContext context, final IServiceJsonHelper helper) {
final var testedField = value.get("field").asText();
final var needle =
helper.readValue(new StoreField(context.getTableName(), testedField), value.get("value"));
return new UpdateWhereSelectionAndProcedure(
new IUpdateWhereProcedure() {
int position;
@Override
public void init(final IRecordFormat selectionFormat, final IRecordFormat recordFormat) {
this.position = selectionFormat.getFieldIndex("target");
}
@Override
public void execute(final IArrayReader selectedRecord, final IArrayWriter recordWriter) {
final var value = selectedRecord.read(this.position);
if (needle.equals(value)) {
recordWriter.write(this.position, null);
}
}
},
new Selection(
context.getTableName(),
List.of(AliasedField.create("target", FieldPath.of(testedField)))),
Set.of(testedField));
}
@Override
public void checkJsonNode(final JsonNode node) {}
}
The migration mostly consists of accessing the update information from the context, like the table name or the updated field. The nullable String
representing the field to update is now described in the context. This object conveniently reports it as a FieldPath
, from the root table in the query, as well as the actual StoreField
in the database.
And the compiler was replaced by a helper service focusing on parsing data from JSON.
Security
Importing the starter com.activeviam.springboot:atoti-server-starter
will automatically configure
JwtConfig
. It no longer needs to be imported.
NoSecurityDatabaseServiceConfig
, is the default database rest service security
when importing the starter com.activeviam.springboot:atoti-server-starter
: the default
implementation of IDatabaseServiceConfiguration
provides No-ops and no-restrictions.
FullAccessBranchPermissionsManagerConfig
has been removed from the public API.
Importing the starter com.activeviam.springboot:atoti-server-starter
automatically configures a
branch permission manager giving full permissions to all users.
Finally, importing the starter com.activeviam.springboot:atoti-server-starter
automatically
configures an opinionated security for the standard Atoti REST services. This is fully designed for
extension.
The following packages contain the classes used to define this security:
com.activeviam.springboot.atoti.server.starter.private_.security
, in the starter, contains the opinionated default security. It contains all the beans that can be overridden.com.activeviam.web.spring.api.security.dsl
contains theHttpConfigurer
used to parameterize the security of each endpoint.
All the security related to Atoti's REST endpoints, and all the security related to Atoti's communication with Atoti UI, is automatically handled. This results in a lot of
@Bean
being obsolete in 6.1, with only the Spring related ones that need to be kept.
Defining the application
Building an Atoti Server application operating on top of a Datastore from descriptions can now be done with the following lines of code:
// This assumes that `datastoreDescription` is defined as a `IDatastoreSchemaDescription`
// and that `managerDescription` is defined as a `IManagerDescription`.
// In this example, no permission manager is used to control access to branches
StartBuilding.application()
.withDatastore(datastoreDescription)
.withManager(managerDescription)
.withoutBranchRestrictions()
.build();
It can easily be converted to Spring beans. The following snippet provides a working example of how
to expose IActivePivotManager
as a Spring Bean :
@Configuration
@RequiredArgsConstructor
public class ActivePivotWithDatastoreConfig implements IDatastoreConfig, IActivePivotConfig {
private final IActivePivotManagerDescriptionConfig apManagerConfig;
private final IDatastoreSchemaDescriptionConfig datastoreDescriptionConfig;
private final IActivePivotBranchPermissionsManagerConfig branchPermissionsManagerConfig;
@Bean
protected ApplicationWithDatastore applicationWithDatastore() {
return StartBuilding.application()
.withDatastore(this.datastoreDescriptionConfig.datastoreSchemaDescription())
.withManager(this.apManagerConfig.managerDescription())
.withEpochPolicy(this.apManagerConfig.epochManagementPolicy())
.withBranchPermissionsManager(
this.branchPermissionsManagerConfig.branchPermissionsManager())
.build();
}
@Bean
@Override
public IActivePivotManager activePivotManager() {
return applicationWithDatastore().getManager();
}
@Bean
@Override
public IDatastore database() {
return applicationWithDatastore().getDatastore();
}
}
This replaces the configuration class ActivePivotWithDatastoreConfig
.
Atoti Components
Registry
In addition to the changes made to the annotations mentioned in the branding section,
Registry initialization has been streamlined.
There is a single entry point: Registry#initialize(RegistryContributions)
that does not require
any knowledge about Atoti's registry mechanism.
RegistryContributions
comes with a static builder that provides the exact same options as the
previous ContributionProvider
classes, making the migration seamless.
The return type of IPluginValue#key()
is now String
instead of Object
.
Partitioning
Partitioning string description no longer support the word "hash", which has been replaced with
"modulo" for clarity's sake.
"hashX(fieldName)"
partitioning description must be changed to "moduloX(fieldName)"
.
Numa Node Selectors have been simplified. Implementations of INumaSelectorDescription
are
available for definition.
The API to retrieve number of processors available in the current machine, often used to define a partitioning, has been moved:
Previous Version | New Version |
---|---|
com.qfs.platform.IPlatform#getProcessorCount | com.activeviam.tech.numalib.api.PlatformUtil#getProcessorCount |
This API should be used instead of Runtime.getRuntime().availableProcessors()
. See, for instance,
https://bugs.openjdk.org/browse/JDK-6942632.
Content Service
Building a Content Service
The Content Service used to be defined as an internal detail of the ActivePivotContentService
,
before being extracted and published as a Bean.
This was automatically performed in com.qfs.server.cfg.content.IActivePivotContentServiceConfig
.
With Atoti Server 6.1, the Content Service is promoted, and now represents a central Bean that must
be defined by users.
Builders are available
at com.activeviam.tech.contentserver.storage.api.builder.ContentServiceBuilder
.
In 6.0, this might have looked like this:
public class ContentServiceConfig implements IActivePivotContentServiceConfig {
@Override
@Bean
public IActivePivotContentService activePivotContentService() {
return new ActivePivotContentServiceBuilder().build();
}
@Override
@Bean
public IContentService contentService() {
return activePivotContentService().getContentService().getUnderlying();
}
}
In 6.1:
@Override
@Bean
public IContentService contentService() {
// Service defined here as a main Bean
return ContentServiceBuilder.create().inMemory().build();
}
@Override
@Bean
public IActivePivotContentService activePivotContentService() {
return new ActivePivotContentServiceBuilder()
.with(contentService())
.withCacheForEntitlements(10)
.needInitialization("ROLE_ADMIN", "ROLE_ADMIN")
.build();
}
Additional examples for different configuration can be found below.
In Memory:
final var contentService = IContentService.builder().inMemory().build();
Backed with a database:
final Properties hibernateProperties = new Properties();
hibernateProperties.setProperty(AvailableSettings.SHOW_SQL, "false");
hibernateProperties.setProperty(AvailableSettings.FORMAT_SQL, "false");
// ... any property wanted
final var config = new org.hibernate.cfg.Configuration().addProperties(hibernateProperties);
final var contentService =
IContentService.builder().withPersistence().configuration(config).build();
Rooted in a specific directory of another Content Service:
final var contentService = IContentService.builder().inMemory().build();
final var prefixedService = IContentService.prefixed(contentService, "root/dir");
The classes **FullAccessBranchPermissionsManagerConfig**
and
ContentServiceBranchPermissionsManager
have been removed. A new builder is available to create
the permission manager using the Content Service. This builder can be used to create the entire
configuration class.
The following snippet illustrates how to do so:
/**
* Sandbox configuration class creating the manager of branch permissions.
*
* @author ActiveViam
*/
@Configuration
@RequiredArgsConstructor
public class ActivePivotBranchPermissionsManagerConfig
implements IActivePivotBranchPermissionsManagerConfig {
private final IContentServiceConfig contentServiceConfig;
@Bean
@Override
public IBranchPermissionsManager branchPermissionsManager() {
final CachedBranchPermissionsManager manager =
new CachedBranchPermissionsManager(
ContentServiceBranchPermissionsManagerBuilder.create()
.contentService(this.contentServiceConfig.contentService())
.allowedBranchCreators(Set.of(ROLE_ADMIN, ROLE_USER))
.defaultBranchOwners(Set.of(ROLE_ADMIN))
.build());
manager.setBranchPermissions(
IEpoch.MASTER_BRANCH_NAME,
new BranchPermissions(
Collections.singleton(ROLE_ADMIN), IBranchPermissions.ALL_USERS_ALLOWED));
return manager;
}
}
Content
It is no longer possible to interact with IContextValues
from the IActivePivotContentService
.
They are no longer stored there by Active Pivot.
Methods surrounding the use of context values have been removed, including getContextValue
,
setContextValue
, removeContextValue
, etc...
An IEntitlementProvider
should be used instead.
The only exceptions to this rule are KPIs and Calculated Members.
With the removed support for Context Values, the locales configured per users, previously stored
inside the MdxContext
, have been upgraded to dedicated content service entries:
IActivePivotContentService#getUserLocale()
has been added.
Moreover, the changes due to the creation of the public APIs have forced a change in the structure of KPIs and calculated members and their serialization.
To ensure that the content of an existing content service is compatible with Atoti Server 6.1, and that UI elements such as dashboards are not lost, a migration tool is available in the sandbox project to help with this task.
com.activeviam.migration.api.ContentServiceMigrator
, in the sandbox application, provides a
method migrate
to perform this migration. To create an instance of this class, one must provide
the instance of the IContentService
to migrate.
Creating a backup of the content service before this database migration should be considered before running this tool.
ContentServiceMigrationApp
, in the sandbox application, uses the ContentServiceMigrator
and the
content service configuration defined in the project to start the migration.
Databases
DirectQuery
A new API has been introduced in version 6.1 to define external databases and applications leveraging the DirectQuery feature. This updated API retains the core functionality of the 6.0 version, and also allows for easier transitions between the different database connectors, and for the ability to define a table without requiring a schema discovery.
com.activeviam.directquery.api.schema.SchemaDescription
and
com.activeviam.directquery.application.api.Application
are common to all external databases.
Database specific implementations of the Spring config have been replaced by a common config
ADirectQueryApplicationConfig
. AClickhouseConfig
, for instance, has been removed.
A user guide is available on the DirectQuery Getting Started page.
ClickHouse
The ClickHouse connector only supports ClickHouse version 24.8 LTS and above. It will also be compatible with future LTS versions that will release during the 6.1 lifecycle.
Databricks
The Databricks connector only supports Databricks runtime version 15.4 LTS and above. It will also be compatible with future LTS versions that will release during the 6.1 lifecycle.
Datastore
DatastoreSchemaDescription
's constructors now require a List<? extends IStoreDescription>
instead
of a Collection<? extends IStoreDescription>
.
Deprecated method ICursor.rewind()
has been removed. Cursors are now considered performing one-way
pass over the underlying data.
Methods using the ID of a table in the datastore transaction API have been removed. The table name should be used instead:
Previous Version | New Version | Comment |
---|---|---|
IDatastoreTransactionStatistics.getStoreTransactionStatistics(int) | IDatastoreTransactionStatistics.getStoreTransactionStatistics(String) | |
ITransactionManager.startTransaction(int[]) | ITransactionManager.startTransaction(String...) | |
ITransactionalWriter.add(int, Object[]) | ITransactionalWriter.add(String, Object...) | |
ITransactionalWriter.addAll(int, Collection<Object[]>) | ITransactionalWriter.addAll(String, Collection<Object[]>) | |
ITransactionManager.addRecords(int, IRecordBlock<? extends IRecordReader>) | ||
ITransactionalWriter.remove(int, Object[]) | ITransactionalWriter.remove(String, Object...) | |
ITransactionalWriter.removeAll(int, Collection<Object[]>) | ITransactionalWriter.removeAll(String, Collection<Object[]>) | |
ITransactionManager.removeRecords(int, IRecordBlock<? extends IRecordReader>) | ||
IDatastoreSchemaTransactionInformation.getLockedStoreIds() | ||
SchemaPrinter.printStoresSizes() | DatabasePrinter.printTableSizes |
CSV source
Previous Version | New Version | Comment |
---|---|---|
ICsvTopic<Path> com.qfs.msg.csv.filesystem.impl.FileSystemCsvTopicFactory#createTopic(String, String, ICSVParserConfiguration) | com.activeviam.source.csv.api.FileSystemCsvTopicFactory#createTopic(String, ICSVParserConfiguration, String...) | The topic can be linked to multiple files at once. |
ICsvTopic<Path> com.qfs.msg.csv.filesystem.impl.FileSystemCsvTopicFactory#createDirectoryTopic(String, String, String, ICSVParserConfiguration) | com.activeviam.source.csv.api.FileSystemCsvTopicFactory#createDirectoryTopic(String, ICSVParserConfiguration, String, String) | For consistency with the above change. |
ICsvTopic<Path> com.qfs.msg.csv.filesystem.impl.FileSystemCsvTopicFactory#createPoolingDirectoryTopic(String, String, String, ICSVParserConfiguration, int) | com.activeviam.source.csv.api.FileSystemCsvTopicFactory#createPoolingDirectoryTopic(String, ICSVParserConfiguration, String, String, int) | For consistency with the above change. |
The CsvSource
class is now private. com.activeviam.source.csv.api.CsvSourceFactory
offers
various builders to create an ICsvSource
.
ActivePivot
Hierarchies
Defining a filter for an entire cube is now done through the fluent builder withFactFilter
, rather
than using withFilter
. This distinguishes the API from the other filtering methods, such as
filtering a partial provider.
ISelectionDescriptionBuilder.withField(String name)
only takes field names, and does not accept
field descriptions.
Interface ILevel
is a legacy wrapper around ILevelInfo
, and is now internal. All the necessary
information can be retrieved from the ILevelInfo
. From this change derive other modifications
Previous Version | New Version |
---|---|
List<? extends ILevel> IHierarchy#getLevels | List<ILevelInfo> IHierarchy#getLevels |
ILevel#AXIS | IHierarchy#AXIS |
ILevel#BRANCH_LEVEL_NAME | IHierarchy#BRANCH_LEVEL_NAME |
ILevel#EPOCH_LEVEL_NAME | IHierarchy#EPOCH_LEVEL_NAME |
ILevel#ALLMEMBER | IHierarchy#ALLMEMBER |
ILevel HierarchiesUtil#getLevel | ILevelInfo HierarchiesUtil#getLevel |
ILevel AAdvancedPostProcessor#getLevel | ILevelInfo AAdvancedPostProcessor#getLevel |
Levels, hierarchies and dimensions are respectively and uniquely identified using the
LevelIdentifier
, HierarchyIdentifier
and DimensionIdentifier
classes.
Most APIs have been migrated from accepting String
arguments to using the corresponding
identifiers.
These objects have been introduced to replace String
descriptions using @
symbol, which reduced
the robustness of Atoti's APIs by accepting partial descriptions.
On that note, Copper no longer supports partially defined elements anymore as well. And Copper joins no longer guess the table field to associate with a level: it is now required to properly specify the join mapping.
Previous Version | New Version |
---|---|
Copper.member("LEVEL") | Copper.member(Copper.level("DIMENSION", "HIERARCHY", "LEVEL")) |
Copper.hierarchy("HIERARCHY") | Copper.hierarchy("DIMENSION", "HIERARCHY") |
Copper.level("LEVEL") | Copper.level("DIMENSION", "HIERARCHY", "LEVEL") |
Copper.newHierarchy(...).fromValues(...).withMembers("STORE", "FIELD") | Copper.newHierarchy(...).fromValues(...).withMembers(new StoreField("STORE", "FIELD")) |
Copper.newHierarchy(...).fromStore(...).withLevel("LEVEL") | Copper.newHierarchy(...).fromStore(...).withLevel("LEVEL", FieldPath.of("LEVEL")) |
Copper.newHierarchy(...).fromStore(...).withLevel("LEVEL", "PATH/TO/FIELD") | Copper.newHierarchy(...).fromStore(...).withLevel("LEVEL", FieldPath.of("PATH", "TO", "FIELD") |
Window.orderBy("HIERARCHY") | Window.orderBy(Copper.hierarchy("DIMENSION", "HIERARCHY")) |
CopperStore.withMapping(CopperLevel) | CopperStore.withMapping(FieldPath, CopperLevel) |
com.quartetfs.biz.pivot.cube.hierarchy.axis.impl.AAnalysisHierarchy
has been removed.
com.quartetfs.biz.pivot.cube.hierarchy.axis.impl.AAnalysisHierarchyV2
has been renamed to
com.activeviam.activepivot.core.ext.api.cube.hierarchy.impl.AAnalysisHierarchy
.
Analysis hierarchies should be registered using
@AtotiExtendedPluginValue(intf = IAnalysisHierarchy.class, ..)
.
Aggregate Providers are now fully internal. Basic information is available through
IActivePivotVersion#getAggregateProviderStatistics
.
Measures
User Defined Measures (Post Processors)
Leading and trailing spaces are no longer trimmed in level descriptions. This allows to perform
queries against data sources with field names containing leading and/or trailing spaces. However,
this change may cause failures when passing level descriptions to post-processors in string-encoded
form (e.g. "level1@hierarchy1@dimension1,level2@hierarchy2@dimension2"
). This is the case for the
leafLevels
property ofABaseDynamicAggregationPostProcessor
and its subclasses. It is necessary
to ensure that no extra spaces between names and separators ('@'
and ','
) remain.
Examples:
" level @ hierarchy "
is now parsed as{" level ", " hierarchy ", null}
and should be rewritten to"level@hierarchy "
."L1@H1, L2@H2"
is now parsed as[{"L1", "H1", null}, {" L2", "H2", null}]
and should be rewritten to"L1@H1,L2@H2"
.
IPrefetcher.name()
is deprecated and will be eventually removed. Some implementations have lost
their constructor without a prefetcher name. Migration requires to pass the name as first argument.
Prefetcher names were introduced to ease the writing of a post-processor. It allows calls to
IAdvancedAggregatesRetriever#retrieveAggregates(String prefetcherName)
. Forcing a name increases
the API's clarity.
On the subject of clarity, abstract class ALocationShiftPostProcessor
has been updated.
Previous Version | New Version |
---|---|
EVALUATE_FOR_MEASURES_PROPERTY | HELPER_MEASURES_PROPERTY |
UNDERLYING_PREFETCHER_NAME | HELPER_PREFETCHER_NAME |
targetMeasures | helperMeasures |
getTargetMeasures(...) | initializeHelperMeasures(...) |
createUnderlyingMeasuresPrefetcher(...) | createHelperPrefetcher(...) |
IScopeLocation
is now internal. Some of its functionalities have been extracted to the objects
that used to expose the scope location.
Previous Version | New Version |
---|---|
IAdvancedAggregatesRetriever#getScope() | IAdvancedAggregatesRetriever#getLocation() |
IAdvancedAggregatesRetriever#getScope().createBuilder() | IAdvancedAggregatesRetriever#createPointLocationBuilder() |
IAggregatesRetrievalResult#getScope() | IAggregatesRetrievalResult#getLocation() |
IIterableAggregatesRetrievalResult#getScope() | IIterableAggregatesRetrievalResult#getLocation() |
Helper class LocationUtil
has been simplified and many methods have been removed. For instance,
LocationUtil#createRangeLocation
now only has one signature, down from four different
alternatives. Other notable changes include:
Previous Version | New Version |
---|---|
new ModifiedLocation(ILocation, int, Object[]) | LocationUtil.createModifiedLocation(ILocation, int, Object[]) |
new ModifiedLocation(ILocation, int[], Object[][]) | LocationUtil.createModifiedLocation(ILocation, int[], Object[][]) |
new LocationBuilder(ILocation, OperationFlag...) | LocationUtil.createLocationBuilder(ILocation, OperationFlag...) |
Location expansion, previously done using LocationUtil#expand
, has been revamped.
The associated methods now provide an iterator which generates locations on the fly, rather than
accumulating them all at once in a collection. This reduces the pressure that this method could
put on the Garbage Collector during a query.
The performance of the method was also improved.
LocationUtil#expandRangeLevels(ILocation, List)
should be used to obtain an iterator that
generates point locations from the given range location.
A location can also be partially expanded, keeping some coordinates as ranges. In that case,
LocationUtil#partialExpand(ILocation, List, List)
should be used.
Read these methods' documentations for more information.
Example:
// Time hierarchy: Year\Month\Date, Currency hierarchy: AllMember\Currency.
final Location location = new Location(new Object[][] {{null, null, null}, {ILevel.ALLMEMBER, null}});
final List<ILevelInfo> expansionLevels = List.of(monthLevel);
final Iterator<ILocation> iterator = LocationUtil.partialExpand(location, expansionLevels, hierarchies);
// Expands levels Year and Month, creating such locations as
// 2024\01\*|AllMember\*, 2024\02\*|AllMember\*, ... , 2023\01\*|AllMember\*, ...
iterator.forEachRemaining(location -> { ... })
IIterableAggregatesRetrievalResult#transferValues
cannot use an Object[]
anymore. An
IWritableRecord
must be created through
IIterableAggregatesRetrievalResult#createRecordFormat(int... measureIds).newRecord()
, and can be
used as a buffer, much like the Object[]
may have been before.
Use of Vectors
Vector implementations are now private.
Previous Version | New Version |
---|---|
new ArrayDoubleVector(double[]) | ArrayVectorUtils#doubleVector(double...) |
new ArrayFloatVector(float[]) | ArrayVectorUtils#floatVector(float...) |
new ArrayLongVector(long[]) | ArrayVectorUtils#longVector(long...) |
new ArrayIntegerVector(int[]) | ArrayVectorUtils#intVector(int...) |
new ArrayObjectVector(Object[]) | ArrayVectorUtils#objectVector(Object...) |
Real Time Support
AStoreStream
only exposes one constructor: AStoreStream(IMultiVersionActivePivot)
.
Implementations of IAggregatesContinuousHandler
are internal. Their plugin keys are available in
the interface.
The records sent to the AStoreStream
listener are now undictionarized. It is no longer necessary
to retrieve the values from dictionaries, meaning that they can be directly read from the given
records.
AFullRefreshHandler
has been removed. Extend AAggregatesContinuousHandler
instead, and implement
computeImpact(ILocation, EventT)
by calling the public method Impact.fullRefresh(location)
.
Schema rebuild
ScheduledActivePivotSchemaRebuilder
and PeriodicActivePivotSchemaRebuilder
have been removed.
The scheduling must now be handled in the project. The entry point is
IActivePivotManager.rebuild(String... pivotsId)
.
Queries
Previous Version | New Version |
---|---|
com.quartetfs.biz.pivot.query.impl.ActivePivotQueryRunner#create() | com.activeviam.activepivot.core.impl.api.query.IActivePivotQueryRunner#create() |
IActivePivotQueryRunner#withWildcardCoordinates(String...) | |
IActivePivotQueryRunner#withWildcardCoordinates(LevelIdentifier...) | |
IActivePivotQueryRunner#withContextValue(Class<T> class, T value) | IActivePivotQueryRunner#withContextValues(T...) |
DrillthroughExecutor.createLocationInterpreter(Properties) | DrillthroughExecutor.createLocationInterpreter(ISelection, Properties) |
In a Query Plan, Copper Joins used to be represented by nodes named External Retrieval
. With the
introduction of the Direct Query feature, this name was prone to confuse users, as "external" could
refer to a table that is not part of the selection, but also to a table that is not at all in
memory.
Since these retrievals simply represent fetching data from a database, they are now called
DatabaseRetrieval
. These retrievals, along with JitPrimitiveRetrievals
, may hit an in-memory
table, or an external table. That is up to the database configuration.
Because the ActivePivot instance has no knowledge of the implementation details of the underlying
database, it is not possible at the moment to offer a better alternative.
These changes are reflected in the REST API for cube queries.
MdxUtil
is now internal. Mdx queries can be executed using the class
com.activeviam.activepivot.server.impl.api.query.MdxQueryUtil
, or using the @Bean
for the
interface com.activeviam.activepivot.server.intf.api.webservices.IQueriesService
.
Distribution
Injections
The setup of a distributed pivot no longer requires any explicit injections in the application.
The starter com.activeviam.springboot:atoti-server-starter
, and the official testing framework,
automatically perform these injections.
More details are available in
this section.
The following calls are no longer needed (because Atoti's starter automatically performs this call):
- inject(IDistributedMessenger.class, plugin.key(), contextValueManager);
- inject(IDistributedSecurityManager.class, plugin.key(), userDetailsService);
Distribution Properties
- The
ActiveViamProperty
activeviam.distribution.endpoint.suffix
is removed and replaced withDataClusterDefinitionBuilder#withEndpointSuffix(String)
. - The
ActiveViamProperty
activeviam.distribution.endpoint.port
is removed and replaced withDataClusterDefinitionBuilder#withPortNumber(int)
. - The
ActiveViamProperty
activeviam.distribution.endpoint.host
is removed and replaced withDataClusterDefinitionBuilder#withAddress(String)
. - The
ActiveViamProperty
activeviam.distribution.endpoint.protocol
is removed and replaced withDataClusterDefinitionBuilder#withProtocol(String)
.
All can also be set through the fluent builders.
When defining the IDataClusterDefinition
through fluent builders, the method
withUniqueIdentifierInCluster
was renamed to withCubeIdentifierInCluster
.
The suffix and port number can be set at this time, through methods withEndpointSuffix(String)
and
withPort(int)
. These two ActiveViamProperties
were redundant with Spring Boot's properties
server.port
and server.servlet.context-path
, and have been removed. To avoid a loss of
functionality, they can now be forwarded to the cluster definition, by injecting the Spring
properties.
Queries
The IClusterDefinition#EXECUTE_IN_DATA_CUBE_PROPERTY
and
IDistributedPostProcessor#EXECUTE_IN_DATA_CUBE_PROPERTY
properties are no longer supported.
Planning a distributed query has been enhanced. Previously, queries were distributed from the leaves
of the calculation chain, up to the first post-processor that would not allow distribution.
This process relied on post-processor chains properly implementing IDistributedPostProcessor
or
IPartitionedPostProcessor
.
However, the introduction of distributed Copper meant that most of these decisions would be made by the engine, rather than being taken through custom user code.
Now, as long as there is a measure that provides a way to reduce partial results coming from multiple cubes into a single result, the entire sub-chain below that measure is distributed, unless another measure strictly specifies that it cannot be distributed, by:
- removing one of the distribution fields from the list of partitioning levels, retrieved through
IPostProcessor#setPartitioningLevels
. - implementing
IDistributedPostProcessor#canBeDistributed
and returning false for this query.
For instance, Copper.combine(measures)
simply applies the given lambda to the underlying measures:
it has no knowledge of whether it can be distributed.
- If all the underlying measures can be distributed, this combination can also be distributed, as long as one of its ancestors in the measure chain specifies how to reduce the partial results into a single final result.
- If one of its underlying measures cannot be distributed, then it also cannot be distributed.
This should result in queries being a lot more distributed than before.
Testing
As previously mentioned, module com-activeviam-activepivot:activepivot-copper-test
is now
internal,
with dedicated module com.activeviam:atoti-server-test
being created instead for project testing.
The associated documentation page details how these testers should be used.
The testing utility QueryCubeSync
is private. DistributionTestHelper
, also in the new test
module, represents the official public endpoint to create a distributed test that handles operations
such as awaiting cluster stability before running a distributed query.