Migrate to 6.0

danger

Starting from 6.0, many classes visible from Atoti jars are now considered internal classes.
These internal classes are located in packages containing internal or private_.
Examples:

• com.activeviam.database.sql.internal.schema.ISqlFieldDescription, indicated by the "..sql.internal.schema..."
• com.activeviam.database.bigquery.private_.BigqueryDataVersion, indicated by the "..bigquery.private_..."

These classes may be changed at any point in time, without regard for backward compatibility, without warning, without mention in the changelog, and without migration notes.
Users must not use them. This will never be an issue as they are never exposed to users The APIs always return their interfaces or wrap them. They are only visible through debuggers.

The main change in this version is the introduction of the Database API. This API will allow ActivePivot to work on top of the Datastore but also on top of external databases. The Datastore now implements the new Database API and the core components of ActivePivot also use it rather than the Datastore API.

Datastore

Operation	Before	Now
Compile a record query	datastore.getLatestVersion().getQueryManager().compile(query)	datastore.getQueryManager().listQuery()
Compile a GetByKey query	DatastoreQueryHelper.createGetByKeyQuery()	datastore.getQueryManager().getByKeyQuery()
Execute a query	datastore.getHead(branch).getQueryRunner().forQuery(query)	datastore.getHead(branch).getQueryRunner().xxxQuery()
Get the metadata	datastore.getQueryMetadata().getMetadata()	datastore.getEntityResolver()

• Field dictionaries are no longer accessible (datastore.getQueryMetadata().getDictionaries() before) but you should never need them. Indeed, the new way to retrieve a field's default value (datastore.getEntityResolver().findTable(tableName).getField(fieldName).getType().getDefaultValue()) always returns an un-dictionarized value, as opposed as before when using recordFormat.getDefault(fieldIndex) that returns the dictionarized value when the field is dictionarized.
• The ICursor interface now implements AutoCloseable. If cursors are used in your project, do not forget to close them, as they could leak database connections.
• Methods to create conditions in BaseConditions now require arguments of the safe type FieldPath instead of legacy field expression strings. These methods have also been renamed to start with a lower case as they are not classes: Equal is now equal, And is now and... Note that Lesser and LesserOrEqual have been renamed less and lessOrEqual. True() and False() have been removed and can be replaced by BaseConditions.TRUE and BaseConditions.FALSE.
• SelectionField is replaced by AliasedField. The equivalent of new SelectionField("alias", "reference/field") is now AliasedField.create("alias", FieldPath.of("reference", "field")).
• The constructors of NonNullableFieldsPostProcessor and DictionarizeFieldsPostProcessor now use ReachableField instead of String.
• The constructor of StoreDescriptionBuilder is now protected. Use StoreDescription.builder() instead.
• In StoreDescriptionBuilder, all operations on fields should be done before defining the partitioning. For example, you cannot call addField() after withPartitioning().
• The return type of DatastoreSchemaDescriptionUtil.createPath(String...) has changed. See the javadoc.
• DynamicConditions was removed. Its methods are in BaseConditions. Sub-interfaces of ICondition and existing implementations have been moved and are now internal classes. ICustomCondition remains a public class, though it has been moved into com.activeviam.datastore.condition package.
• The API to create Dynamic conditions has changed. It is now mandatory to name a parameter using as(String) method. Parameter indexes are no longer supported.
• Other changes:

Before	Now
datastoreVersion.getQueryRunner()	datastoreVersion.getDatastoreQueryRunner()
datastoreVersion.getQueryManager()	datastoreVersion.getDatastoreQueryManager()

Spring

The interface IActivePivotManagerDescriptionConfig no longer contains the datastore description. The IActivePivotManagerDescriptionConfig#userManagerDescription() was renamed managerDescription(). userSchemaDescription() was moved to and renamed IDatastoreSchemaDescriptionConfig#datastoreSchemaDescription(). One now has to define an IDatastoreSchemaDescriptionConfig that will contain the datastore description.

• ActivePivotConfig was renamed ActivePivotWithDatastoreConfig.
• IDatastoreConfig.datastore() was renamed IDatastoreConfig.database().

CXF & SOAP

Both SOAP services and CXF dependencies were completely dropped. Following this, all remote services require an AAuthenticator instead of a ClientPool.

Here is a list of replacements for each dropped service from com.qfs.webservices:

Webservice interface name	Replacement
IIdGenerator, IStreamingService, ILongPollingService	ActivePivot Websocket API
IQueriesService	com.qfs.pivot.rest.query.IQueriesRestService
IAdministrationService	com.qfs.pivot.rest.discovery.IDiscoveryRestService
IConfigurationService	com.qfs.pivot.rest.configuration.IConfigurationRestService
ILicensingService	No replacement

ActivePivot

Aggregation module changes

Cloneable aggregated values

The API to use when writing aggregation bindings with cloneable aggregated values has been rewritten to avoid user mistakes.

Previously, users achieved such a task by extending ABasicAggregationBindingCloneable or one of its sub-classes - AAggregationBindingCloneable, AVectorAggregationBinding, SumVectorAggregationBinding, ... It often requires to carefully write aggregated values using #write(int, Object) in addition to an obscure flag using #writeBoolean. Reads and updates would also require reading such flags and values, not to mutate a value shared through the system.

The new API focuses on user ease. When writing a value, implementors must choose between #writeReadOnlyAggregate(int, Object), preserving the original value as much as possible or #writeWritableAggregate(int, Object), informing this class that the provided value can be mutated.
Similarly, when reading an existing aggregate, implementors can access the possibly untouched value through #readReadOnlyAggregate(int) or automatically access a writable value using #readWritableAggregate(int).
At any point, implementors are free to call the write methods to replace the current aggregates.

With the described changes, aggregating values change to something like this (in pseudo-code):

void copy(int from, int to){
  // The aggregate is read-only because we pulled it directly from the input reader
  // We avoid creating a copy of the value that may not be necessary
-  this.output.write(to, this.input.read(from))
-  this.output.writeBoolean(to, false);
+  writeReadOnlyAggregate(to, readInput(from));
}

void aggregate(int from, int to) {
-  IVector aggregate = this.output.read(to);
-  if (this.output.readBoolean(to)) {
-    aggregate = cloneAggregate(aggregate);
-    this.output.write(to, aggregate);
-    this.output.writeBoolean(to, true);
-  }
+  IVector aggregate = readWritableAggregate(to);
  // We can safely modify the content of `aggregate`
  aggregate.plus(readInput(from));
}

danger

When implementing an aggregation that uses AVectorAggregationBinding to bind columns, ensure that the IChunkFactory creates marked chunk, for instance writing:

public class AggregationVector extends AAggregation<?> {

  @Override
  public IChunkFactory<?> createChunkFactory(
          final boolean isTransient,
          final IAllocationSettings allocationSettings) {
    return ChunkFactories
            .chunkMarkedVectorFactory(getAggregatedDataType(), isTransient, allocationSettings);
  }

}

The Aggregation Function documentation provides an up-to-date guide for writing this type of aggregation function.

Deprecated classes

The following abstract classes are deprecated and will be removed in a future version:

• AGenericBaseAggregationFunction
• AGenericAggregationFunction
• AGenericVectorAggregationFunction and AVectorAggregationFunction (See example)

Implementers must now write their aggregation functions using the standard unique API. The Aggregation Function documentation provides an up-to-date guide for writing an aggregation function.

Generalization of Aggregation Functions

The client facing (through the Registry) IAggregationFunction has been extensively modified. @QuartetPluginValue(intf = IAggregationFunction.class) and @QuartetPluginValue(intf = IUserDefinedAggregationFunction.class) must be changed to @QuartetPluginValue(intf = IGenericAggregationFunction.class).

A new IGenericAggregationFunction has been introduced, which allows you to create an aggregation based on an arbitrary number of data sources. IAggregationFunction represents a specialization of this new interface, for the creation of aggregations based on a single source of data. We recommend clients implement AAggregationFunction when implementing aggregation functions based on a single source of data.

The interface IGenericAggregationFunction contains the following methods:

• int getAggregatedType(int[] sourceTypes), which deduces the type of the aggregated values, based on the types of the sources of data. It must throw if the given types are not supported. It can be used to implement:
• IGenericAggregation createAggregation(List<String> sourceIdentifiers, int[] sourceTypes), which creates an aggregation specialized for the given sources of data.
• IGenericAggregationFunction withRemovalSupport(), which specializes the current aggregation function with capabilities to disaggregate from the resulting column.

The interface IAggregationFunction has lost the responsibility to create a chunk factory, which is now the prerogative of the IAggregation interface. IMultiSourceAggregationFunction was also introduced, with an associated base abstract class AMultiSourceAggregationFunction that mirrors AAggregationFunction.

IMultiSourceAggregation was also introduced, with an associated base abstract class AMultiSourceAggregation that mirrors AAggregation. We recommend clients implement AAggregation, AMultiSourceAggregation or AUserDefinedAggregation instead of implementing the interfaces, as they already give default implementations for most of the methods.

Post Processors

• Legacy post processors have been replaced by their corresponding new implementation. This means all V2 post processors introduced in 5.11 have become the base post processors to use. Migration notes are available in post processor migration section.
• Post Processors no longer need to implement Serializable.

Aggregate Provider Configuration

The GlobalMultipleAggregateProvider, which was serving as an aggregate provider and the holder of partial providers, has been removed. Defining the topmost provider with the type "MULTI" is no longer needed.

• GlobalMultipleAggregateProviderBase.UNDERLYINGPROVIDER_PLUGINKEY has been removed
• GlobalMultipleAggregateProviderBase.RANGE_SHARING has been replaced with IAggregateProviderHolderBase.RANGE_SHARING
• IBuildableAggregateProviderDescriptionBuilder contains the plugin keys for JIT, Leaf and Bitmap providers

Builders

• com.activeviam.pivot.builders.StartBuilding was removed and is replaced by com.activeviam.builders.StartBuilding.

• com.activeviam.builders.StartBuilding.entitlementsProvider() was moved to com.activeviam.pivot.security.builders.EntitlementsProvider.builder().

• The withProperty(String, String) methods on the dimension, hierarchy and level builders where respectively renamed withDimensionProperty, withHierarchyProperty and withLevelProperty to avoid ambiguities.

Copper

Parts of the Copper API causing boxing were removed. Migration notes are available in copper migration section.

• CopperStore.field(String) now takes a field name instead of an expression. If you used an expression, you must now use CopperStore.field(FieldPath).
• CopperStore.withMapping(String, CopperLevel) now takes a field name instead of an expression. If you used an expression, you must now use CopperStore.withMapping(FieldPath, CopperLevel). Same for CopperStore.withMapping(String, String).

Testing API

The CubeTesterBuilderExtension provided for in the Copper Testing API now required a Supplier<CubeTesterBuilder> as first argument. The migration can be straightforwardly done by replacing the provided testerBuilder instance by the ()->testerBuilder supplier in your test classes.

Partial providers

• The default implementation of IPartialProviderSelector.select has been removed, any custom implementation of IPartialProviderSelector must now implement this method.

Datastore Service

This service is now called the Database Service. The classes and interfaces related to the Datastore Service have been renamed. datastore was replaced by database to emphasize that this service works on any IDatabase. Here are some examples:

• IDatastoreServiceConfiguration was renamed IDatabaseServiceConfiguration.
• DatastoreRestServicesConfig was renamed DatabaseRestServicesConfig.
• DatastoreService was renamed ReadOnlyDatabaseService.
• DatastoreRestService was renamed DatabaseRestServiceController.
• JsonDatastoreQueryResult was renamed JsonDatabaseQueryResult.
• SecurityFromCubeToDatastoreFilterHook was removed.

The public vocabulary has also changed, with "database"/"table"/"join" being used instead of "datastore"/"store"/"reference". Examples of impacted classes:

• IDatastoreFieldDescription was renamed IDatabaseFieldDescription
• IDatastoreReferenceDescription was renamed IDatabaseJoinDescription
• JsonDatastoreReference was renamed JsonDatabaseJoin
• IStoreSecurity was renamed ITableSecurity
• IStorePermission was renamed ITablePermissions
• the endpoint datastore/data/stores was renamed database/data/tables
• the endpoint datastore/data/storeNames was renamed database/data/tableNames
• the endpoint datastore/discovery/references was renamed database/discovery/joins

Analysis Hierarchy

Constructor of AAnalysisHierarchy now requires the list of ILevelInfo of the corresponding hierarchy levels. In any cases, implementations of Java-based analysis hierarchy should be migrated to AAnalysisHierarchyV2. Either the description of the analysis must be given to the cube description, or an IAnalysisHierarchyDescriptionProvider plugin value should be created to go along with the custom Analysis Hierarchy. This provider description must have the same plugin key as the Analysis Hierarchy.

Introspecting Analysis Hierarchies

In order to define the introspecting levels of an analysis hierarchy, one must now set the selectionField of the levels of the created AnalysisHierarchyDescription according to the needs of the hierarchy.

Analysis hierarchies with at least now one level with a specified selectionField in their description are now implicitly introspecting hierarchies and must implement AMultiVersionAnalysisHierarchy#processIntrospectedMember([...]) to define their introspection logic.

Limits regarding the ordering between introspecting and non-introspecting levels for an Analysis Hierarchy are unchanged (impossible to define an introspecting level deeper than a non-introspecting one).

Hierarchy descriptions

Hierarchy descriptions now must use the IHierarchyDescription interface. Depending on the type of described hierarchy, the sub-interface to use will change:

• "Standard" hierarchy descriptions (e.g. hierarchies whose levels are filled by levels of the Selection) use the IAxisHierarchyDescription interface which remains unchanged.
• All Analysis hierarchy descriptions now use the IAnalysisHierarchyDescription interface and now must explicitly contain information relative to their levels.
• The fluent builder path for Analysis hierarchies has changed, from

withHierarchy(hierarchyName)
.withPluginKey(pluginKey)
.withProperty(propertyKey, propertyValue)

to :

withAnalysisHierarchy(IAnalysisHierarchyDescription)

or

withAnalysisHierarchy(hierarchyName, pluginKey)
        .withCustomizations(hierarchyDescription -> {
            hierarchyDescription.putProperty(propertyKey, propertyValue);
        })

The second option allows for customization of the created Analysis Hierarchy's description. This description will be created by the associated IAnalysisHierarchyDescriptionProvider.

• AAnalysisHierarchy.buildDiscriminatorPaths() was removed. It is replaced by a more efficient method: buildDiscriminatorPathsIterator(IDatabaseVersion).

-AxisLevelDescription(String levelName) was removed. A selectionField (formerly propertyName) must now be specified in the constructor to avoid unexpected behavior.

Aggregation Procedure

DatastorePrefetchRequest is replaced by DatabasePrefetchRequest. The field expressions are now represented with the FieldPath class instead of String.

Memory Allocation Monitoring

We are no longer reporting the off-heap memory allocated with java's internal MBeans. Instead, you can use the newly created MBeans located in DirectMemoryMonitoring for the information you wanted regarding memory consumption from ActivePivot. Additionally, MAC can be used for deeper investigations.

Other changes

• StartBuilding.selection(IDatastoreSchemaDescription) was removed. Use StartBuilding.selection(description..asDatabaseSchema()) instead.
• ADatastoreVersionAwareProperty was renamed ADatabaseVersionAwareProperty.
• com.qfs.store.IDatastoreVersionAware was renamed com.activeviam.database.api.IDatabaseVersionAware.
• IDatastoreSelectionSession was renamed ISchemaSelectionSession.
• IFilteredDatastoreSelectionSession was renamed IActivePivotSchemaSelectionSession.
• IDatastoreAware was changed and will be replaced by IDatabaseAware.
• AStoreStream.dictionaries is replaced with AStoreStream.resultFormat.getDictionary(...).
• All ActivePivot named threads are now prefixed by activeviam and suffixed by worker-i where i is the thread count number.
• Creating a QueriesTimeLimit with a Duration lower than 1 second will now throw a IllegalArgumentException.

Content Server Rest API

• The new Rest Bulk API for Content Server was developed. The old API for bulk operations was removed. It is no longer possible to use POST method with query parameters moveAll, putAll or deleteAll because mappings for them are deleted. Also, the query parameter metadataOnly is replaced by a parameter includeContent that is a logical opposite of metadataOnly. Do not forget to update the value of this parameter from True to False and vice versa when renaming.

CSV Source

• The class CSVSourceConfiguration is now created through CSVSourceConfigurationBuilder, and is given the method fileComparator that previously belonged to the class CSVSource.

Rest Services

All our Rest endpoints were migrated to Spring MVC. The migration implies a few changes:

• Removal of the JsonResponse wrapper. The data will now be directly in the body of the request and the status in the HTTP Status code of the request. The errors in AP are still returned as JsonError with the full stack trace

• Every rest endpoint is now prefaced by the activeviam namespace. For example : localhost:XXXX/activeviam/content/rest/v7/

• The datastore endpoint has been changed to database.

• The content server rest service now handle the path of the file requested or given through Query parameters not Path variables. The parameter for import, permissions, etc. needs now to be true and doesn't allow void.

• The Activemonitor's monitors site separator is now an underscore not a slash.

ActiveMonitor

• IParameterAwareActivePivotManagerDescriptionConfig was renamed IParameterAwareDatastoreSchemaDescriptionConfig. You will need to add a Spring configuration implementing IActivePivotManagerDescription in your project.

Properties

• The properties used for the external configuration of ActivePivot and ActiveMonitor applications in ActiveViamProperty and JwtConfig have been changed to uniformize their naming conventions.

  Properties are now either :

  • Prefixed by "activeviam"
  • Prefixed by an ActiveViam technical product name such as "activepivot" or "activemonitor"

List of the affected properties with the new corresponding naming :

• In JwtConfig :

Before	Now
qfs.jwt.expiration	activeviam.jwt.expiration
qfs.jwt.key.private	activeviam.jwt.key.private
qfs.jwt.key.public	activeviam.jwt.key.public
qfs.jwt.claim_key.authorities	activeviam.jwt.claim_key.authorities
qfs.jwt.claim_key.principal	activeviam.jwt.claim_key.principal
qfs.jwt.generate	activeviam.jwt.generate
qfs.jwt.check.user_details	activeviam.jwt.check.user_details

• In ActiveViamProperty :

Before	Now
contentServer.remote.import.timeout	activeviam.contentServer.remote.import.timeout
qfs.handler.stored.maxSize	activeviam.handler.stored.maxSize
qfs.mmap.threshold	activeviam.mmap.threshold
qfs.mmap.tracking	activeviam.mmap.tracking
qfs.compression.chunkset.disable	activeviam.compression.chunkset.disable
qfs.stream.configuration.gc.delay	activeviam.stream.configuration.gc.delay
qfs.mdx.formula.callStackDepthLimit	activeviam.mdx.formula.callStackDepthLimit
completer.log.suppressed	activeviam.completer.log.suppressed
qfs.contentservice.root	activeviam.contentservice.root
nativeMemoryCacheRatio	activeviam.nativeMemoryCacheRatio
qfs.slab.memory.allocation.strategy	activeviam.slab.memory.allocation.strategy
defaultChunkSize	activeviam.defaultChunkSize
minimalChunkSize	activeviam.minimalChunkSize
chunkGarbageCollectionFactor	activeviam.chunkGarbageCollectionFactor
chunkAllocatorClass	activeviam.chunkAllocatorClass
qfs.vectors.defaultBlockSize	activeviam.vectors.defaultBlockSize
qfs.vectors.garbageCollectionFactor	activeviam.vectors.garbageCollectionFactor
qfs.vectors.allocator.pool.size	activeviam.vectors.allocator.pool.size
qfs.vectors.swap.directory.numberFiles	activeviam.vectors.swap.directory.numberFiles
vectorDelimiter	activeviam.vectorDelimiter
queries.continuous.primitive.fullFork	activeviam.queries.continuous.primitive.fullFork
postprocessor.partitionOnRangeLevelsByDefault	activeviam.postprocessor.partitionOnRangeLevelsByDefault
qfs.distribution.gossip.router.port	activeviam.distribution.gossip.router.port
qfs.distribution.gossip.router.enable	activeviam.distribution.gossip.router.enable
logicalAddress	activeviam.logicalAddress
protocolPath	activeviam.protocolPath
qfs.distribution.netty.externalAddress	activeviam.distribution.netty.externalAddress
qfs.distribution.netty.portRange	activeviam.distribution.netty.portRange
nettyMessageMaxSize	activeviam.distribution.nettyMessageMaxSize
qfs.distribution.netty.bindAddress	activeviam.distribution.netty.bindAddress
qfs.streaming.MDX.attemptLimit	activeviam.streaming.MDX.attemptLimit
qfs.streaming.GET_AGGREGATES.attemptLimit	activeviam.streaming.GET_AGGREGATES.attemptLimit
xmlaDiscoveryApproxCardinality	activeviam.xmlaDiscoveryApproxCardinality
qfs.pool.policy	activeviam.pool.policy
qfs.pool.size	activeviam.pool.size
qfs.pool.scheduler.size	activeviam.pool.scheduler.size
quartet.property.separator	activeviam.property.separator
qfs.collectible.query.type	activeviam.collectible.query.type
qfs.mmap.sampling.depth	activeviam.mmap.sampling.depth
qfs.mmap.sampling.start	activeviam.mmap.sampling.start
qfs.mmap.sampling.percent	activeviam.mmap.sampling.percent
qfs.branch.master	activeviam.branch.master
qfs.pool.nodes	activeviam.pool.nodes
qfs.pool.procs	activeviam.pool.procs
qfs.activepivot.expiredqueries.polling	activeviam.activepivot.expiredqueries.polling
qfs.streaming.initialPublicationMode	activeviam.streaming.initialPublicationMode
qfs.exportservice.rootpath	activeviam.exportservice.rootpath
duringThreadNumber	activeviam.duringThreadNumber
maxDuringThreadNumber	activeviam.maxDuringThreadNumber
qfs.distribution.remote.pool.size	activeviam.distribution.remote.pool.size
qfs.distribution.log.size.threshold	activeviam.distribution.log.size.threshold
qfs.distribution.maxPendingDiscoveries	activeviam.distribution.maxPendingDiscoveries
qfs.conflation.maxQueueSize	activeviam.conflation.maxQueueSize
qfs.conflation.maxDelayTime	activeviam.conflation.maxDelayTime
repository.cache.isolated_transactions	activeviam.repository.cache.isolated_transactions
repository.poll.period	activeviam.repository.poll.period
repository.poll.period.max	activeviam.repository.poll.period.max
repository.poll.log.threshold	activeviam.repository.poll.log.threshold
repository.daemon.waitStableDistribution	activeviam.repository.daemon.waitStableDistribution
qfs.server.namespace.parent	NO LONGER USED (see Rest Services section)
continuousQueryMonitoring	activeviam.continuousQueryMonitoring
qfs.selection.listener.catchUpMaxTime	activeviam.selection.listener.catchUpMaxTime
com.activeviam.json.strongPrimitiveParsing	activeviam.json.strongPrimitiveParsing
com.activeviam.directquery.enableAutoVectorizer	activeviam.directquery.enableAutoVectorizer
com.activeviam.directquery.cubeFeedingTimeoutInSeconds	activeviam.directquery.cubeFeedingTimeoutInSeconds
com.activeviam.directquery.snowflake.maxresultsetsize	activeviam.directquery.snowflake.maxresultsetsize
mdx.negativecellset.limit	activeviam.mdx.negativecellset.limit
qfs.activecollector.renew.frequency	activeviam.activecollector.renew.frequency
activepivot.snl.url	activemonitor.activepivot.url
live.snl.url	activemonitor.ui.url
sentinel.poll.period	activemonitor.poll.period
sentinel.poll.period.max	activemonitor.poll.period.max
sentinel.poll.log.threshold	activemonitor.poll.log.threshold
sentinel.daemon.waitStableDistribution	activemonitor.daemon.waitStableDistribution
sentinel.periodicExecutors.poolSize	activemonitor.periodicExecutors.poolSize