Skip to main content

Migration notes

Migrate to 5.10

Datastore

  • Primary indexes are now called Unique indexes (All the rows in an index of this type must be unique).
  • TransactionManagerUtil.resetBranch was moved in ITransactionManager.
  • Rename ARemoveUnknowKeyListener to ARemoveUnknownKeyListener to fix typo.
  • Move NumaSelectorFromStoreDescription from com.qfs.desc to com.qfs.desc.impl package.
  • Move ADefaultEpochPolicy from com.qfs.multiversion to com.qfs.multiversion.impl package.
  • Rename ImmutableRecordList to ImmutableRecordBlock to match the naming pattern used by sibling classes.

ActivePivot

    • Change the hierarchy builder: hierarchyBuilder.factless().withStoreName(...) was replaced by hierarchyBuilder.fromStore(...). It cannot be called after factless() nor notFactless().

Copper

  • All published Copper measure must now be named. Expressions like Copper.sum("pnl").publish(context) have to be replaced with Copper.sum("pnl").withName("pnl.SUM").publish(context).
  • Copper.newSingleLevelHierarchy(...).from(CopperStoreField) was replaced by Copper.newHierarchy(...).fromField(CopperStoreField).
  • Copper.newSingleLevelHierarchy(...).from(CopperLevelValues) was replaced by Copper.newHierarchy(...).fromValues(CopperLevelValues).

Distribution

  • The JGroupsMessenger has been removed. Now, ActivePivot relies on two messenger types, the LocalMessenger and the NettyMessenger. Note that the NettyMessenger still uses JGroups for member discovery and group membership. More details on this topic in Distributed Messenger
  • The property NotificationMember#AWAIT_NOTIFICATIONS has been renamed to IDataClusterDefinition#AWAIT_NOTIFICATIONS_TIME.

Functional Interfaces

The functional interfaces in package com.qfs.func have been moved and renamed.

Old locationNew location
com.qfs.func.IBiIntPredicatecom.activeviam.util.function.BiIntPredicate
com.qfs.func.IFloatUnaryOperatorcom.activeviam.util.function.FloatUnaryOperator
com.qfs.func.IntBiFunctioncom.activeviam.util.function.IntBiFunction
com.qfs.func.ITriFunctioncom.activeviam.util.function.TriFunction
com.quartetfs.fwk.util.FloatBinaryOperatorcom.activeviam.util.function.FloatBinaryOperator
com.qfs.func.IEithercom.activeviam.util.function.IEither
com.qfs.func.IOptionalEithercom.activeviam.util.function.IOptionalEither
com.qfs.func.impl.Eithercom.activeviam.util.function.impl.Either
com.qfs.func.impl.OptionalEithercom.activeviam.util.function.impl.OptionalEither

Query Results Context Value

The default behavior for QueriesResultLimit context value have changed from QueriesResultLimit#withoutLimit() to QueriesResultLimit#defaultLimit() (the default transient and intermediate result limit amount to 100,000 and 1,000,000 point locations respectively). The default limits may appear low for certain projects, in this case the configuration can be easily overridden. This is explained in the respective documentation section.

ActiveViam Properties

  • activeviam.datastore.query.maxLookupsOnPrimary was renamed activeviam.datastore.query.maxLookupsOnUnique.
  • ActiveViamProperty.MAX_LOOKUPS_ON_PRIMARY_INDEX was renamed ActiveViamProperty.MAX_LOOKUPS_ON_UNIQUE_INDEX.

Spring Configuration Changes

The ActivePivot Configuration does not automatically contain the ContextValueFilter bean anymore, thus preventing conflicts with Spring Boot.

Azure Cloud Source migration

The Azure Cloud Source has migrated its dependency to the Azure Blob Storage SDK from v8 to v12.

A migration guide from v8 to v12 can be found on the Azure SDK repository.

Renamed Classes

All classes of the Azure Cloud Source have been changed to include the word Azure as a prefix. As a result, the following classes had their name changed:

5.95.10
CloudBlobPathAzureBlobPath
BlockBlobPathAzureBlockBlobPath
AppendBlobPathAzureAppendBlobPath
PageBlobPathAzurePageBlobPath

Changed Classes

  • The handling of client-side encryption has undergone some major changes to better match the new architecture of the Azure Blob Storage SDK. Changes about this are detailed in the below section.

  • IAzureCloudDirectory:

    • Now extends ICloudDirectory<BlobClientBase> instead of ICloudDirectory<CloudBlob>.
    • getTBlob(String) return types were updated with the new types from the Azure SDK.

  • Removed IAzureEntityPath.upload(InputStream, long, BlobRequestOptions).

    Prefer using getUnderlying() to get a reference to the underlying blob client and use blob-specific configuration for uploads.

  • Uploading with an unknown length is not supported anymore for page blobs, and will throw an UnsupportedOperationException if attempted.

  • Records#getGlobalDefaultValue directly uses the content type value passed as an int instead of ILiteralType.

Client-side Encryption

In the Azure Blob Storage SDK, client-side encryption is now implemented in the package com.azure:azure-storage-blob-cryptography.

With the SDK upgrade, client-side encryption cannot be configured at the Azure service client level anymore. It needs to be specified on a per-blob basis, before uploading or downloading content.

As a result, two new classes have been introduced to handle client-side encryption:

  • AzureEncryptedCloudDirectory: a variant of AzureCloudDirectory that additionally holds encryption keys for client-side encryption, and that can produce AzureEncryptedBlobPaths to existing and non-existing encrypted blobs.
  • AzureEncryptedBlobPath: a path to an encrypted blob; essentially a wrapper around an EncryptedBlobClient.

The other classes AzureBlobPath, AzureBlockBlobPath, AzureAppendBlobPath, AzurePageBlobPath and AzureCloudDirectory are all unaware of client-side encryption. They will not encrypt uploaded data and will not decrypt downloaded data.

They accept three additional arguments in their constructor compared to their non-encrypted counterparts:

  • a key-wrapping algorithm (String)
  • a key encryption key (com.azure.core.cryptography.AsyncKeyEncryptionKey)
  • a key encryption key resolver (com.azure.core.cryptography.AsyncKeyEncryptionKeyResolver)

If the directory or path only needs to perform either only downloads or only uploads, some arguments are not required and can be set to null: | operations | required | |---------------------|----------------------------------------------| | download | key encryption key resolver | | upload | key-wrapping algorithm, key encryption key | | download and upload | all arguments |

The possible key wrapping algorithms are specified in the class KeyWrapAlgorithm from com.azure:azure-security-keyvault-keys (the dependency is not included in the Azure Cloud Source).

Some snippets of AsyncKeyEncryptionKey creation from a local key or from an Azure Key Vault can be found in the readme of com.azure:azure-storage-blob-cryptography.

Important: the new implementation of client-side encryption in Azure Blob Storage SDK v12 only permits uploads to client-side-encrypted block blobs. The creation of client-side-encrypted blobs of other types (append and page) is not supported anymore.

Downloads of client-side encrypted page and append blobs that were already created through other means is still possible through the AzureEncryptedBlobPath.

Migrate to 5.9

Java 8 is no longer supported. ActivePivot 5.9 is compatible with Java 11.

ActivePivot

New Copper API

Shipped as an experimental feature in ActivePivot 5.8.3, the new Copper API is officially replacing the one released with ActivePivot 5.7.0, which is now abandoned.

Shift measure creation

The Copper shift measure creation methods have changed : CopperMeasure#at([...]) methods have been changed to a unified CopperMeasure#shift(CopperLevelsAt...) method. Find more information in our user guide for updated examples.

Parent Value

The method Copper.parentValue(measure, hierarchy) has been changed to measure.parentValueOn(hierarchy).

Setting the designated type of a measure

To improve clarity, the method CopperMeasure#cast([...]) has been renamed to CopperMeasure#withType([...]).

Level representation

LevelCoordinate has been renamed and is now LevelIdentifier.

Test jar

The Copper test utils that were previously packaged in a test-jar have been moved to their own module:

 <groupId>com.activeviam.activepivot</groupId>
 <artifactId>activepivot-copper-test</artifactId>

Spring Configuration Changes

Schema configuration

ActivePivotConfig now requires an IActivePivotManagerDescriptionConfig. IDatastoreDescriptionConfig was merged into IActivePivotManagerDescriptionConfig and IParameterAwareDatastoreDescriptionConfig was renamed IParameterAwareActivePivotManagerDescriptionConfig.

In IActivePivotManagerDescriptionConfig you must now implement userManagerDescription() (respectively named userSchemaDescription()) instead of managerDescription() (respec. schemaDescription()).

The way the description post-processors are applied has been changed. You are impacted if you explicitly build the Datastore or the ActivePivotManager in your project instead of using DatastoreConfig and ActivePivotConfig. In this case, define an IActivePivotManagerDescriptionConfig and give the results of the managerDescription() and schemaDescription() default methods to the builders.

IActivePivotConfig.activePivotManagerDescription() was removed and replaced by IActivePivotManagerDescriptionConfig.managerDescription().

CORS configuration

The CORS configuration now relies on Spring standards. Instead of our own ICorsFilterConfig that created a SpringCorsFilter, we moved to Spring CorsConfigurationSource. As internal components require a knowledge of the CORS configuration before we can define the actual CorsConfigurationSource, a new interface ICorsConfig has been introduced, detailing the information of a standard CORS configuration. This is used in internal services to create REST and WS services compatible with the configuration. This is used in the sandbox project to create the final CORS configuration.

ICorsConfig only requires you to define the list of allowed origins. The other methods provide the accepted and exposed headers the authorized methods. Those have default implementations compatible with the ActivePivot stack.

SpringCorsFilter has been deleted. Thanks to CorsConfigurationSource, Spring automatically creates the filter.

Miscellaneous

ActivePivotServicesConfig removes an internal attribute, creating a unused dependency to IDatastoreConfig. If needed, you can restore this dependency by extending the configuration class and restoring the attribute.

I18nConfig was renamed AI18nConfig. Many of the static methods have changed to instance methods to allow easier extensions and refinements.

Aggregation Procedures

Interface

The interface was slightly changed to support more features in Copper (getDatastorePrefetch and createContext). This change should be transparent for procedures extending AAnalysisAggregationProcedure.

Additional validation checks

To offer support for Aggregation Procedures depending on other Procedures, ActivePivot uses the property IAnalysisAggregationProcedureDescription#getUnderlyingLevels() to identify the dependencies between Procedures. This drives a change to the API, preventing Procedures to define their own handled levels in the list of underlying levels.

This constraint is enforced by the validation procedure of the description. Configurations may need to be updated. Particurlarly, in 5.8, you always had to define a handled hierarchy from builders. You can now avoid this with the method withoutHierarchies().

Option to disable the Epoch level

The epoch level of the epoch dimension can now be enabled or disabled. The epoch dimension is still enabled by default. The epoch level is now disabled by default. To enable it, use IEpochDimensionDescription.setEpochLevelEnabled(boolean) or IEpochDimensionBuilder.withEpochLevel.

The IEpochDimensionBuilder.withBranchesLevel method was renamed IEpochDimensionBuilder.withBranchLevel. The methods in IEpochDimensionDescription were also renamed from *epochsLevel* and *branchesLevel* (e.g. getEpochsLevelComparator or getBranchesLevelFormatter) to *epochLevel* and *branchLevel* (e.g. getEpochLevelComparator and getBranchLevelFormatter).

Distributed Cube no longer supports the epoch level

New default aggregation functions

Aggregation functions that maintain a history state, like AVG, MIN, or MAX are memory intensive. Thus, for the MIN and MAX aggregation functions, the new default behavior is append only. This means that disaggregation is not supported for these two functions. For disagregation support, please use MIN_HISTORY and MAX_HISTORY aggregation functions (e.g. value.MIN_HISTORY or value.MAX_HISTORY).

Default Distributed Message Size Limits

Previous ActivePivot versions had a maximum default distributed message size of 1GB for every message type. This limit has been reduced. The default maximum size is:

  • 64MB for IInitialDiscoveryMessage messages
  • 4MB for ITransactionCommittedMessage messages
  • 4MB for ScopedAggregatesRetrievalResultTransporter answers wrapped in BroadcastResult messages
  • 32MB for DrillthroughMessageWithHeadersAnswers answers wrapped in BroadcastResult messages
  • 2MB for any other message types

You can still override these values using the property NETTY_MAX_SIZE_PROPERTY of the ActiveViamProperty. In addition to the older configuration method using class names, it is possible to configure the message default size limit using the constants (enum) defined in NettyStreamUtils#MessageType as follows:

  • INITIAL_DISCOVERY_MESSAGE for initial discovery messages
  • ITRANSACTION_COMMITTED_MESSAGE for transaction commit message
  • SCOPEDAGGREGATES_RETRIEVAL_TRANSPORTER for single Gaq query result messages
  • DRILLTHROUGH_HEADER for drillthrough header query result messages
  • DRILLTHROUGH_MESSAGE_WITH_HEADERS_ANSWER for drillthrough query result messages
  • GLOBAL_MESSAGE for global configuration

For instance, the following statement used in prior versions:

- -Dqfs.distribution.netty.message.maxSize=com.quartetfs.biz.pivot.cellset.impl.ScopedAggregatesRetrievalResultTransporter=42m,com.qfs.messenger.message.IInitialDiscoveryMessage=200m

becomes:

+ -DnettyMessageMaxSize=com.quartetfs.biz.pivot.cellset.impl.ScopedAggregatesRetrievalResultTransporter=42m,com.qfs.messenger.message.IInitialDiscoveryMessage=200m
or
+ -DnettyMessageMaxSize=singleGaqResult=42m,initialDiscovery=200m

Miscellaneous

  • ALocationShiftPostProcessor#handleNoUnderlyinMeasure is renamed into ALocationShiftPostProcessor#handleNoUnderlyingMeasure to fix the typo - missing "g".

REST and WS APIs

With the introduction of a new REST call to export query plans and another REST endpoint to forward client traces to a tracing server, the version of ActivePivot REST API has changed from v4 to v5. With the addition of the metadata to the WS updates of the Content Service entries, the WS API has changed from v4 to v5.

As components of the REST and WS API versions remain synchronized, these changes result in the following URL changes:

BeforeAfter
pivot/rest/v4/...pivot/rest/v5/...
pivot/ws/v4/...pivot/ws/v5/...
content/rest/v4/...content/rest/v5/...
content/ws/v4/...content/ws/v5/...

Impacted REST services:

  • ActivePivot services for queries, context configurations, etc
  • Datastore service for queries and updates
  • Content Service
  • Tracing Service

CSV Source

The CSV Source can now accept incomplete lines (where the number of columns is smaller than the expected number of attributes of a record). You can set this in CSVParserConfiguration's full constructor, or through its setAcceptIncompleteLines(boolean). It can finally be set through ParserContext#setAcceptIncompleteLines(boolean) method. This sets this property on its underlying CSVParserConfiguration.

You can also sample the input files to work on a small portion of the input files. The sampling policy is passed as an argument of type ICsvParserConfigPolicy to the constructor of CSVParserConfiguration, or using the method CSVParserConfiguration#setParserPolicy(ICsvParserConfigPolicy).
The Core Product ships basic policies to load the first lines of one or more files, or load a limited number of files. Those are available in ICsvParserConfigPolicy static methods.

Parquet Source

The Parquet Source benefited from the same sampling policies as the CSV Source. They implement IParquetReaderPolicy, that are passed in the constructor of ParquetParser. Like the CSV Source, basic policies are available in IParquetReaderPolicy to load a limited number of lines or files.

Other changes

  • LogWriteException is now a runtime exception. This exception is still thrown when an error occurs during the writing of the Transaction Log.
  • One of the signatures of ITransactionManager.performInTransaction was removed.

Migrate to 5.8

Announcement

JDK 11 Support

ActivePivot 5.8 supports both JDK 8 and JDK 11. The Java version used by Maven when compiling ActivePivot depends on the JAVA_HOME environment variable, or the current java version if the JAVA_HOME environment variable is not set. For more information, see the dedicated page.

ActivePivot

HierarchyUtil

The following methods do not return null but throw an UnknownOlapElementRuntimeException when the element is not found:

public static IHierarchy getHierarchy(IActivePivot pivot, String hierarchyDescription) {...}

public static IHierarchy getHierarchy(IActivePivot pivot, String dimensionName, String hierarchyName) {...}

public static ILevel getLevel(IActivePivot pivot, String levelDescription) {...}

public static ILevel getLevel(final IActivePivot pivot, final String dimensionName, final String hierarchyName, final String levelName) {...}

Filters on analysis hierarchies, PostProcessors and V2

In 5.7 the behavior of aggregate retrievers has been changed: the retrievals with filters on an analysis hierarchy now return null. In 5.7.3 we introduced new versions of post processors (ABasicPostProcessorV2, ABaseDynamicAggregationPostProcessorV2, ADynamicAggregationPostProcessorV2 and AFilteringPostProcessorV2) that automatically remove the filters on analysis hierarchies in the prefetcher of the post processor, and add the filter back while expanding the result on the analysis hierarchies. It was introduced as V2 to avoid changing the behavior of the post processor in a bugfix release.

In 5.8 the post processors V2 have been removed and the core post processors ABasicPostProcessor, ABaseDynamicAggregationPostProcessor, ADynamicAggregationPostProcessor and AFilteringPostProcessor now perform this filter removal if the "analysisLevels" property is used.

Keep in mind that AAdvancedPostProcessor does not handle those filters. If you implement AAdvancedPostProcessor directly, this confluence page explains how to handle them.

In order to solve this problem, ABasicPostProcessor (previously ABaseDynamicPostProcessors) now uses named prefetchers. They will use the constant ABasicPostProcessor.BASIC_POST_PROCESSOR_PREFETCHER (previously ABaseDynamicAggregationPostProcessor.DYNAMIC_AGGREGATION_POST_PROCESSOR_PREFETCHER) as the name of the prefetcher to retrieve. If you define your own prefetcher you need to name it correctly:

IPrefetcher<?> namedPrefetcher = IPrefetcher.name(ABasicPostProcessor.BASIC_POST_PROCESSOR_PREFETCHER, prefetcher);

CubeFilterBuilder

ICubeFilterBuilder.includeMembers([...]) and excludeMembers([...]) now only accept List<String> as members path name argument. The previous signature that accepted List<?> as members has been moved to ICubeFilterBuilder.includeMembersWithConditions([...]) and excludeMembersWithConditions([...]).

The includeMembersWithConditions([...]) and excludeMembersWithConditions([...]) methods should only be used when building the CubeFilter with IConditions.

Range Sharing configuration

Range sharing can no longer be configured with a boolean. You must use an integer or use the withRangeSharing() method on the cube builder, that is, .withProperty("rangeSharing", "false") has been replaced by .withoutRangeSharing().

AStoreStream

Listeners on a continuous query no longer receive an initial view when registering to this query. This was not the case with com.quartetfs.biz.pivot.postprocessing.streams.impl.AStoreStream, and has been fixed. AStoreStream is a selection listener that registers on a datastore selection. This operation can occur in the middle of a transaction. The javadoc of com.quartetfs.biz.pivot.postprocessing.streams.impl.AStoreStream#registerSelection gives some pointers for concrete implementations.

Datastore

Duplicate key handling

Replacement: IDuplicateKeyWithinTransactionListener → IDuplicateKeyHandler

The IDuplicateKeyWithinTransactionListener interface has been replaced by the IDuplicateKeyHandler interface, as described in the Changelog.

With IDuplicateKeyWithinTransactionListener you could only define a custom behavior when two records had the same key fields within the transaction but no record with these key fields existed in the datastore yet. You can still define such a behavior using the IDuplicateKeyHandler.selectDuplicateKeyWithinTransaction function. However, it is now also possible to define a custom behavior when a record in the transaction has the same key fields as a record already in the datastore, by defining the IDuplicateKeyHandler.selectDuplicateKeyInDatastore function.

The default behavior, which is to always update with the latest record, has not changed.

The store description builder has been modified: you should replace .onDuplicateKeyWithinTransaction().logException() with .withDuplicateKeyHandler(DuplicateKeyHandlers.LOG_WITHIN_TRANSACTION)

Selection Builder

When using the datastore selection builder, you can add all fields reachable by references from a given store using .withAllreachableFields(). The only policy used to be that all fields with same name had to be reachable from one another, and that the furthest would be used. The selection builder now walks the field graph using references between stores, and keeps track of the paths leading to the same field. If there are no collisions, nothing happens. If there are, three options are available:

  • resolve conflicts automatically by choosing the furthest field and throw an exception if two fields with the same name are not reachable from one another (example: ref1/field and ref2/field). This is how it used to be in 5.7 and is the default behavior.
  • resolve conflicts automatically by choosing the closest field and throw an exception if two fields with the same name are not reachable from one another (example: ref1/field and ref2/field).
  • resolve conflicts manually. You can use a FieldsCollisionHandler to transform a map of {field name, [possible paths]} into a map of {name, full expression} (the expression is of the form ref1/ref2/fieldName). This allows you to keep one, the other, both (with custom field name), or none.

Test jar

The datastore test utils that were previously packaged in a test-jar have been moved to their own module:

 <groupId>com.activeviam.tech</groupId>
 <artifactId>datastore-test</artifactId>

Removal of LINE storage

The in-line storage type provided by ActivePivot 5.0 as a safe alternative to the new columnar storage has been removed. Only one value remains possible: StorageType.COLUMN.

Changes are required only if your Datastore stores were explicitly configured to use the LINE storage. If so, either replace it with COLUMN as below, or remove the call to configure the storage to use the default version.

new StoreDescriptionBuilder().withStoreName("<store>")
        .withField(...).asKeyField()
        ... // Other configuration calls
-       .withStorage(IStoreDescription.StorageType.LINE)
+       .withStorage(IStoreDescription.StorageType.COLUMN)
        ...
        .build();

Format of partitions sizes

StoreUtils.getPartitionSizes(storeVersion) now returns values of -1 in the array for non-existing partitions (either dropped or never initialized) at the given storeVersion.

JDBCSource

The constructors for IJDBCSource implementations without an appendBatchSize have been removed.

The constructors for IJDBCSource implementations no longer require a driverClassName in the arguments when an IConnectionSupplier is provided.

The SimpleConnectionSupplier implementation of IConnectionSupplier now requires a driverClassName argument.

The appendQueueSize attribute has been removed from all IJDBCSource implementation constructors, and is now a field of the JDBCTopic class.

JSON Web Tokens (JWT)

The RSA key pair defined by the qfs.jwt.key.[public|private] variables in jwt.properties should now be encoded with at least 2048 bits instead of the previous 1024. A new pair can be generated using JwtUtil.

StreamingService

The signature of IStreamingService.updateStreamRanges changed. It now takes an IAxisRanges instead of an array of IRangeProperties.

Spring configuration

With the introduction of the centralized property accessor ActiveViamProperty, the environment properties will be available only if you import the ActiveViamPropertyFromSpringConfig configuration class. For more information, see Properties in ActivePivot.

Memory analysis

With 5.8.0, Memory Analysis services have been improved. Some elements have been migrated to the dedicated application Memory Analysis Cube - DatastoreFeederVisitor, other classes have been renamed to better emphasize that the services are designed for Memory analysis only and not monitoring.

Renamed classes:

  • IMemoryMonitoringService -> IMemoryAnalysisService
  • MemoryMonitoringService -> MemoryAnalysisService
  • MonitoringStatisticSerializerUtil -> MemoryStatisticSerializerUtil

Removed classes:

For the sake of readability, only major classes are mentioned.

  • DatastoreFeederVisitor
  • MemoryMonitoringDatastoreDescription

Previous migration notes

You can find the previous Migration Notes in our old Confluence documentation: