Skip to main content

Changelog 5.11

The 5.11 branch forked the 5.10 branch from version 5.10.2

See the 5.10 changelog for changes before the fork.

5.11.4​

2022-09-26

Added​

  • PIVOT-4365 Add an MdxQueryBouncer limiting the number of concurrently running mdx queries. See What's New? page for configuration details.
  • PIVOT-5027 Empty partitions are automatically dropped at the end of transactions.
  • PIVOT-5077 Add IDistributedActivePivotInstanceDescription.NO_REPLICATION_MEASURES property to disable replication for certain measures on distributed cubes.
  • PIVOT-5631 ActiveUIResourceServerConfig and AdminUIResourceServerConfig are now core classes. You no more need to copy them from the Sandbox to your projects.

Changed​

  • PIVOT-4934 Adjust Logging levels for received statements on the MDX REST endpoint to INFO.
  • PIVOT-5030 Disable RangeSharing for Post-Processors when an Analysis Hierarchy is expressed in the queried location.
  • PIVOT-5304 Improve the performance of the code handling the generation of the locations when analysis hierarchies are expressed.
  • PIVOT-5626 Skip initialization of previously failed post-processors
  • PIVOT-5657 Switch outer reference policy to auto-detection mode for the queries executed from Datastore MBeans.
  • PIVOT-5671 Log the complete APM monitor result's only when detailed logging is enabled.
  • PIVOT-5677 Activiti dependencies upgraded from 5.22.0 to 5.23.0.
  • PIVOT-5728 Improve JIT retrievals completion performance in some applications with a high partition count.

Fixed​

  • PIVOT-5041 Clarify the behavior and error message of Licensing.reload() and its underlying MBean.
  • PIVOT-5365 Fix ActivePivotVersion releasing its reference to the datastore too early. In some cases, a race condition could lead to wrong results when using a KeepLastEpochPolicy and querying an old not marked ActivePivotVersion with a JustInTime provider.
  • PIVOT-5479 Fix ClassCastException when querying the drillthrough headers on a distributed cube.
  • PIVOT-5458 Correctly propagate exception in ExternalDatastoreRetrieval.
  • PIVOT-5502 Fix cached data mutated in DatastoreService.
  • PIVOT-5508 Fix the creation of the ColumnIntegerRowMapping memory statistics for MAC.
  • PIVOT-5515 Fix ArrayIndexOutOfBoundsException in ChunkedBitmap when adding two bitmaps.
  • PIVOT-5539 Fix NullPointerException when discarding transient results from a distributed measure with no value.
  • PIVOT-5502 Fix cached data mutated in DatastoreService.
  • PIVOT-5575 Fix a performance issue impacting queries on cubes having partial providers and a large amount of hierarchies.
  • PIVOT-5639 Fix an issue with nested combine operations with context value dependencies in Copper.
  • PIVOT-5640 Fix an issue where unrelated store partitions could generate unnecessary transient objects. This reduces the memory footprint of small transactions.
  • PIVOT-5661 Fixed potential deadlock in JDBCTask when all JDBCAppendRunnable crashed.
  • PIVOT-5671 Hide sensitive information in logs.
  • PIVOT-5713 Fix ClassCastException during the recording of distributed messages statistics.
  • PIVOT-5759 Fix Deadlock when allocating partition content.
  • PIVOT-5777 Fix wrong parameter order in RemoteContentService.

5.11.3​

2022-05-06

Added​

  • PIVOT-5301 BottomCountFunction and TopCountFunction should accept a string expression as a comparator.
  • PIVOT-5297 Support all arrow TimeStampVector types: TimeStampNanoTZVector, TimeStampMicroTZVector, TimeStampMilliTZVector, TimeStampSecTZVector.

Added​

  • PIVOT-5301 BottomCountFunction and TopCountFunction should accept a string expression as a comparator.

Changed​

  • PIVOT-5202 Improve the Copper doc on map methods to specify that the cells must be written into at each iteration.

Fixed​

  • PIVOT-5318 Fix ClassCastException in the PlanningGraph
  • PIVOT-5333 Increase severity for full reset and pending initial discovery messages to WARNING in distributed clusters.
  • PIVOT-5395 Fix the creation of the HashMapRowMapping memory statistics for MAC.
  • PIVOT-5407 Fix Frequent Value compression not properly handling null values.
  • PIVOT-5428 Fix ArrayIndexOutOfBoundsException when using withBatchSize in parquet source.
  • PIVOT-5443 Fix SimpleCubeDescriptionBuilder not properly setting shared context values.
  • PIVOT-5479 Fix ClassCastException when querying the drillthrough headers on a distributed cube.

5.11.2​

2022-04-01

Added​

  • PIVOT-5232 A UI to browse the Datastore. See AdminUIResourceServerConfig in the Sandbox.
  • PIVOT-5246 Add ChunkSingleBoolean to compress Boolean chunks containing the same value for all positions.

Changed​

  • PIVOT-5232 Breaking Moved JungSchemaPrinter and associated classes to datastore test artifact. spring-boot, netty, google-guava, azure-identity, azure-security-keyvault-keys and azure-storage-blob are updated to the latest version.
  • PIVOT-5239 Improve DirectedGraph.getEdgesTo.
  • PIVOT-5275 Record reading and writing operations now strictly follow the JVM's automatic type widening described in the JVM's spec. The previous behavior was to only allow the operations corresponding to the field type.
  • PIVOT-5322 Renaming of exposed properties related to consul service configuration within APM.
  • PIVOT-5367 Renamed properties intermediateSize and transientSize in QueryResultLimit to intermediateLimit and transientLimit, respectively.
  • PIVOT-5411 Update Spring Boot to 2.6.6 which includes a vulnerability fix.

Removed​

  • PIVOT-5232 BREAKING ContentServerResourceServerConfig was removed. It is replaced by AdminUIResourceServerConfig (See the Sandbox).

Fixed​

  • PIVOT-5098 Fixed an IllegalStateException when performing queries containing cached aggregates for primitive measures and primitive-aggregated measures.
  • PIVOT-5188 Fix potential bug in AnalyticFunctionPostProcessor where partitioning levels were not properly set.
  • PIVOT-5218 Improve performance of the Mdx NonEmpty function
  • PIVOT-5259 Correctly propagate custom aggregation function used for result reduction in distributed post-processors.
  • PIVOT-5280 Using the flexible datastore no longer removes the insertion time and commit time update procedures.
  • PIVOT-5291 Correctly group monitored JMX APM components and fix some variable visibility issues.
  • PIVOT-5308 Clarify the javadoc of Copper.StoreLookup when the specified store has nullable key fields.
  • PIVOT-5309 Correctly set elapsed time in aggregate retrievals.
  • PIVOT-5310 Fix retrieval result size figures when printing the query plan. In some cases, results were discarded before their result size computed.
  • PIVOT-5311 Fix bug where hierarchy visibility was set to false but was still visible.
  • PIVOT-5328 Fix an NullPointerException when having multiple threads discarding the same transient result at the same time.
  • PIVOT-5330 Fix issue with store size and cursor not being printed when using SchemaPrinter.
  • PIVOT-5335 Fix aggregates cache not being updated in Query Cubes after transactions without any impact on the pivot's structure.
  • PIVOT-5342 Fix bug where we could not query joined store's lookup measure on grand total.
  • PIVOT-5396 The reversed comparators did not return their plugin key.

5.11.1​

2022-01-12

Added​

  • PIVOT-5170 Add result size reporting for distributed query plan print.

Changed​

  • PIVOT-5009 Prevent users from creating Copper join hierarchies based on a store field included in the mapping.
  • PIVOT-5238 Parallelize ActivePivotManagerBuilder's postProcess to reduce cubes creation time.

Fixed​

  • PIVOT-5095 Fix real-time update on query with epoch hierarchy. Previously, real-time updates on such queries resulted in branch disappearing from the updated result. See Real-Time What-If Queries section.
  • PIVOT-5189 Fix IIterableAggregatesRetrievalResult#transferValues that could copy the wrong aggregates.
  • PIVOT-5195 Fix bug in the revert to version. Some records were not reverted in some edge cases.
  • PIVOT-5208 Enforce PostProcessor initialization when requesting the dataType of a post-processed measure.
  • PIVOT-5210 Fix a bug in lead, lag, first, last and window Copper functionalities when relying on a Context Value measure.
  • PIVOT-5224 License is now redacted when logged.

5.11.0​

2021-11-26

Added​

  • PIVOT-4449 Added a detailed reporting of CSV parsing errors. Use the CSV_PARSING_REPORT_ENABLED property to enable it, and CSV_PARSING_REPORT_MAX_ANOMALIES to limit its size.
  • PIVOT-4639 Add parameter to auto-detect outer references in DatastoreService.
  • PIVOT-4642 Integrate APM (Application Performance Monitoring) library into ActivePivot.
  • PIVOT-4653 ActivePivot's custom data structures following com.qfs.chunk.IArray and com.qfs.chunk.IWritableArray can now spawn a cursor. This cursor can be moved up and down the array and perform read/write operations at its current position.
  • PIVOT-4655 Add a data export service to download or export MDX queries.
  • PIVOT-4676 The query plan now includes aggregation procedures time and external datastore retrievals for PrimitiveAnalysisAggregationRetrieval. It also includes the name of the used partial provider for PartialPrimitiveAggregatesRetrieval.
  • PIVOT-4677 Integrate the Excel Add-in library into ActivePivot.
  • PIVOT-4708 Added virtual hierarchies experimental feature. Those are hierarchies whose members are not materialized. A hierarchy can be defined as virtual using the hierarchy builder API .virtual().
  • PIVOT-4715 Added an interface ICanUseOtherReferenceOrBuild in the selection fluent builders for ease of use with Spring Beans.
  • PIVOT-4733 Allow to override measure aggregator xmla property with CopperMeasure#withXmlaAggregator(int).
  • PIVOT-4804 Throw an exception when creating measures names with commas.
  • PIVOT-4806 License path is now printed in the log.
  • PIVOT-4815 Ability to name (give a alias) when selecting field in a RecordQuery or a GetByKeyQuery. Before the name was the path of the field.
  • PIVOT-4824 Include epochId of the max and latest continuous query statistics.
  • PIVOT-4847 Add mark() and unmark() on IDatastoreVersion.
  • PIVOT-4913 The JDBC Source will now log the exceptions that make its workers exit the parallel processing of records.
  • PIVOT-5037 The Copper tester now supports hasValues() in every case and a new method withDelta() to set the allowed delta between the values given by the user in hasValues() and the actual values returned by the measures.
  • PIVOT-5059 Partial providers can now have a name using the fluent builder API PartialBuilder#withName(..).
  • PIVOT-5060 Support encoding and decoding NaN and +/-Infinity in JSON.
  • PIVOT-5069 Add the currently written record in case of type error in IRecordFormat#newRecord().
  • PIVOT-5118 Add a toString() to CloudCsvDataProvider() for better error message.
  • PIVOT-5170 Log query results size by default when printing the query plan. Note that DistributedAggregatesRetrieval doesn't support such feature yet.

Changed​

  • PIVOT-3978 Partitioning fields are now automatically dictionarized.
  • PIVOT-4315 The Copper Typing API is now simpler, only allowing to call withType after operations like plus, minus, etc.
  • PIVOT-4498 The ActiveViamPropertyFromSpringConfig.resolve() bean is now named ActiveViamPropertyFromSpringConfig.SPRING_PROPERTIES_RESOLVER.
  • PIVOT-4484 Deprecate usage of OutputStream in TableBuilder and add usage of StringBuilder
  • PIVOT-4653 IVector and IBlock no longer extend IWritableArray.
  • PIVOT-4653 IPostProcessor abstract implementations are given a version 2, such as ABasicPostProcessorV2, ADynamicAggregationPostProcessorV2 and AAdvancedPostProcessorV2. See the migration notes for details.
  • PIVOT-4747 ReferencedField was replaced by ReachableField.
  • PIVOT-4781 StreamSnapshotService#snapShotStreams(String) is now compatible with StreamSnapshotService#loadStreams(String). Now, both operation ignore STOPPED streams. Use StreamSnapshotService#snapShotAllStreams(String) to snapshot all streams including the stopped ones.
  • PIVOT-4789 Provide possible solutions in error message when we failed to initialize the SaferNative class in LinuxPlatform.
  • PIVOT-4802 Reduce the memory consumption of ModifiedLocation, primarily used in cumulated aggregates and in shifted measures.
  • PIVOT-4808 Report error if the ActivePivotSchema fails to attach.
  • PIVOT-4813 Update ActiveMonitor REST API documentation to correctly illustrate how a new monitor can be added.
  • PIVOT-4820 Change constructors of KeepLastEpochPolicy and FlexibleDatastoreEpochPolicy to use Duration instead of long.
  • PIVOT-4833 AVectorAggregationFunction.aggregate method will now receive a vector allocator.
  • PIVOT-4834 Chunks reading and writing operations now strictly follow the JVM's automatic type widening described in the JVM's spec. The previous behavior was to only allow the operations corresponding to the chunk type.
  • PIVOT-4893 Vectors reading and writing operations now strictly follow the JVM's automatic type widening described in the JVM's spec. The previous behavior was to only allow the operations corresponding to the vector type.
  • PIVOT-4912 Add default implementation of CancellableCountedCompleter#afterCompute(Throwable) calling CancellableCountedCompleter#tryComplete().
  • PIVOT-4947 Return a lighter ActivePivot description in SOAP webservices called SoapActivePivotDiscovery.
  • PIVOT-4959 Reorganize the IVectorBinding API. Some methods have seen their two IVector parameters inverted, but they now all carry the less confusing names of left and right (rather than source and destination). The left vector will always receive the result of the vector operation.
  • PIVOT-4973 IReadableDatastore now implements IBranchedMultiVersion.
  • PIVOT-5003 Reorganize AMultiVersionAnalysisHierarchy and AAnalysisHierarchy to clearly state the methods to implement. The notable changes is that now, users must override AMultiVersionAnalysisHierarchy#processIntrospectedMember to complete paths of introspected members, instead of overriding #contributeMember directly. This change removed a lot of internal methods from the interface IAnalysisHierarchy.
  • PIVOT-5003 Introduce AAnalysisHierarchyV2. This class will replace AAnalysisHierarchy in ActivePivot 6.1.
  • PIVOT-5003 Correct typo by renaming IAnalysisHierarchyInfo#getUseIntrospection into IAnalysisHierarchyInfo#getUseIntrospection.
  • PIVOT-5032 Upgrade org.apache.hadoop to 3.3.1.
  • PIVOT-5021 Upgrade Spring Boot version to 2.5.4 and update dependencies.
  • PIVOT-5024 Use ChunkedIntegerArray instead of IntegerArray in AMultiVersionUniqueRecordIndex so that we can now put more than 300M records per partition.
  • PIVOT-5058 Implements the correct equals() and hashcode() for StoreDescription and VectorFieldDescription.
  • PIVOT-5123 Partitioning modulo 1 now throws an exception.
  • PIVOT-5129 Renamed AnalysigAggregationProcedureFactory to AnalysisAggregationProcedureFactory.
  • PIVOT-5168 Bump Spring version to 2.6
  • PIVOT-5168 Bump Jackson version to 2.13.0
  • PIVOT-5168 Bump Apache Arrow version to 6.0.1
  • PIVOT-5168 Bump Apache CXF version to 3.4.5

Removed​

  • APS-12776 Remove IEpochManager.mark(...) and IEpochManager.unmark(...). One should call them on the underlying IEpochHistory.
  • PIVOT-4244 Remove the XML descriptions of the cubes.
  • PIVOT-4747 ISelectionField was removed. Use SelectionField instead.
  • PIVOT-4748 StreamUtil.negate has been removed; we use Predicate.not instead
  • PIVOT-4776 IMultiMap was removed. Use a Map whose values are collections instead.
  • PIVOT-4986 Release more resources after dropping a partition (dictionaries of indexes that are not used by a reference can now be released every time).
  • PIVOT-5011 Remove the provided strings from exception messages in RestContentServer and JsonQueryService.
  • PIVOT-5029 IAggregatedMeasureAware was removed.
  • PIVOT-5084 Removed an invalid warning that was triggered when building Copper Hierarchies backed by a store. Said warning was stating these hierarchies shouldn't be factless.
  • PIVOT-5094 IMultiVersionStorePartition#drop(IDroppedPartitionInformation) does not take any argument anymore.
  • PIVOT-5117 Remove TransactionWrapper.

Fixed​

  • PIVOT-4487 Fix Copper post-processor bug where if the only underlying measures of a calculated measure were null or constant measures, the post-processor wouldn't be evaluated at all.
  • PIVOT-4735 Fix continuous handler registration issue. The issue happened in distribution when some underlying measures have not yet appeared thus causing a duplication in handler key of the parent measure.
  • PIVOT-4780 Fix unnecessary query plan logging when queryPlanSummary is enabled.
  • PIVOT-4800 Enhance query plan timing reports. This particularly fixes two man issues:
    • JITPrimitiveAggregatesRetrieval Partitioned retrievals for non-existing partitions used to display the largest execution time of their siblings. Now, only partitioned retrievals that performed some work at least and whose result is empty will be reported in the timing section, along with the other retrievals with result
    • In some cases, for a group of related retrievals (siblings), timing were not printed if one of the sibling has no result. This edge case has been fixed
  • PIVOT-4811 Fix ClassCastException in SingleValueFunction when working with arrays.
  • PIVOT-4814 Fix an issue where exporting the memory statistic of a chunk containing non-direct vector blocks would throw a NPE.
  • PIVOT-4816 Fix an issue occurring when merging two AggregateStores. This caused the merged result to include null values instead of the correct value for some measures.
  • PIVOT-4818 Fix an issue leading to incorrect transitions between IntegerArrayVersion.
  • PIVOT-4819 Fix UnsupportedOperationException in ActivePivotStatistics. The issue happened when computing transaction fact statistics on an incompatible type of transaction.
  • PIVOT-4862 Fix an export error in the IMemoryStatistic generation for TombStone classes.
  • PIVOT-4866 Fix NullPointerException in FilterTranslator. The issue could happen when translating a wildcard, e.g. null, to the data cube.
  • PIVOT-4881 Fix an issue where the resizing of IntegerArray and LongArray generated non-transient chunks regardless of its isTransient parameter.
  • PIVOT-4901 Prevent aggregates computation (i.e. GetAggregatesQuery) when mdx CELL PROPERTIES is equal to CELL_ORDINAL.
  • PIVOT-4923 Fix hiding subtotals issue that could not hide all the measures.
  • PIVOT-4935 Fix NPE when accessing an uninitialized datastore via MBean.
  • PIVOT-4944 Fix the cleaning method of the prefetch scope location when executing more than one aggregation procedure in a single PrimitiveAnalysisAggregationRetrieval.
  • PIVOT-4946 Fix an issue with Copper storeLookup that could fail to find its underlying measures, in the case where these measures were previously re-optimized.
  • PIVOT-4949 Fix thread safety for SharedDelegateDictionary#mapFromCell.
  • PIVOT-4951 Fix an edge case issue with IChunkInteger sparse compression.
  • PIVOT-4956 Fix GenericVectorBinding#applyAs.. operations for non-commutative operators.
  • PIVOT-4958 Fix the update of the ContentServer entries for branches containing characters that are illegal for FileSystem-specific paths.
  • PIVOT-4964 Fix an ArrayIndexOutOfBoundException happening when compressing chunks with freed rows as FrequentValueChunk.
  • PIVOT-4972 Fix branches not visible sometimes with continuous queries. The ISecurityDetails was not always propagated when registering queries.
  • PIVOT-4979 Fix missing fields issue in the analysis aggregation procedures when a cube filter is done on a Copper join.
  • PIVOT-4992 Fix dependency issue in content-server-spring.
  • PIVOT-4998 Reduce time spent at startup optimizing Copper measures
  • PIVOT-5010 Properly set the level type of a slicing Copper hierarchy
  • PIVOT-5016 Fix ArrayIndexOutOfBoundsException in the SetExpander of the Mdx engine.
  • PIVOT-5038 Fixed errors in usage counting caused by decrementing the usage too many times.
  • PIVOT-5050 Fix distributed post-processors initialization. This issue happened when chaining at least three post-processors where the bottom one in the chain is not initialized and caused missing dependencies propagation in some cases.
  • PIVOT-5062 Fix NullPointerException in PointListRetrievalResult in the case of a polymorphic distributed setup.
  • PIVOT-5068 Fix NullPointerException in the AggregatesResultCopier used by postprocessors.
  • PIVOT-5074 Fix SubCubeProperties#removeRestriction(dimension, hierarchy). The issue happened in distributed polymorphic setup where the same dimension from two data nodes includes different hierarchies and one of the dimension hierarchies cardinality is greater than 1.
  • PIVOT-5078 The Parquet Source correctly parses boolean vectors.
  • PIVOT-5097 Fix NullPointerException when registering a GetAggregatesQuery build using a location Collections not supporting null elements.
  • PIVOT-5115 Fix IllegalArgumentException during query plan logging when executionTimingPrint is enabled .
  • PIVOT-5119 Fix rollback process of store transactions.
  • PIVOT-5128 Fix a memory leak that could happen upon query failure/cancellation if an AggregatesCache was used.
  • PIVOT-5137 Fix LevelIdentifier.fromDescription and HierarchyIdentifier.fromDescription. They did not handle correctly the escaping of @.
  • PIVOT-5151 Fix QfsStrings.escape with $ as escape symbol.
  • PIVOT-5156 Fixed missing query plan retrievals. The issue happened when underlying retrievals with different partitioning were merged. Note that this does not affect query results nor the query plan export feature, but only the query plan printing.
  • PIVOT-5162 Fix potential NPE in PrimitiveAnalysisAggregationRetrieval when joining results.