Skip to main content

Changelog 5.10

5.10.2​

2021-04-30

Added​

  • PIVOT-4486 Add the possibility to apply query cube filter on the prefetch of Copper analytic functions using the API CopperAnalyticMeasure#withFilter(AnalyticFilter).
  • PIVOT-4713 Add the method IDistributedHierarchyManager#waitForAllMessages allowing to wait for all the recorded messages to be processed.

Changed​

  • PIVOT-4300 Post-processor dependency graph cycle detection is now enabled by default.
  • PIVOT-4780 Change MBeans in QueryPlanMonitoring to print execution plan and timing full or as summary, e.g. QueryPlanMonitoring#enableExecutionTimingPrintFull() or QueryPlanMonitoring#enableExecutionTimingPrintAsSummary().

Fixed​

  • APS-13016 Better handle network errors which were causing issues with license checks.
  • PIVOT-4717 Fix UDAF error that could occur when totally emptying the underlying store.
  • PIVOT-4725 Some datastore versions were not unmarked after stopping the cubes.
  • PIVOT-4728 Measure names with several parentheses were not correctly supported by the Formula PostProcessor.
  • PIVOT-4730 Prevent Real-Time statistics errors from crashing continuous queries update. The issue happened when statistics computations tried to fetch retrievals that had been dismissed by the query engine (usually because of a non-existent location).
  • PIVOT-4739 Fix publishing of WindowReduceMeasure (description equality).
  • PIVOT-4749 Clarify the documentation of the cube/export/mdx/download REST service endpoint.
  • PIVOT-4760 Fix an issue where IStoreToParquetMapping would map fields from the datastore to the wrong fields in the parquet schema.
  • PIVOT-4766 Fix Frequent Chunk Compression bug where nested FrequentNullDoubleChunk could be created.
  • PIVOT-4771 Fix NPE cause by CopperPostProcessor#withInheritedContinuousQueryHandlers. The issue happened while inheriting the handlers of a post-processor that hasn't any.
  • PIVOT-4772 Fix a query issue where wildcards on slicing analysis hierarchies based on isolated store fields would conflict with Copper bucketing hierarchies.
  • PIVOT-4775 Fix an issue with AAdvancedPostProcessor performing an eager modification of the prefetched cube filter.
  • PIVOT-4775 Fix an issue with the query engine where some retrievals handing analysis hierarchies, were wrongly declaring their dependencies.
  • PIVOT-4791 All datastore versions are now synchronously unmarked when stopping the pivot.

5.10.1​

2021-03-26

Added​

  • PIVOT-4473 Add loggers for CREATE, UPDATE and DROP MDX statements
  • PIVOT-4638 Add the maskAtOrBelow() method to the CopperMeasureToAggregateAbove interface, allowing to define levels below which a measure will not return aggregated values when queried. The behavior is the complementary of the doNotAggregateAbove() method.
  • PIVOT-4702 Add support for DATEPART function in MDX with the simple signature DatePart(interval, date).

Changed​

  • PIVOT-4705 Make Records#getGlobalDefaultValue use int instead of ILiteralType.
  • PIVOT-4707 PointIndexedRecordBlock no longer implements IChunkSet.
  • PIVOT-4707 Renamed ImmutableRecordList in ImmutableRecordBlock to match the pattern established in sibling classes.

Deprecated​

  • PIVOT-4705 Deprecate SchemaHelper#undictionarize in favour of Records#undictionarize.
  • PIVOT-4724 Deprecate com.qfs.pivot.impl.ActivePivotSchemaExtendedPlugin in favour of com.quartetfs.biz.pivot.impl.ActivePivotSchemaExtendedPlugin located in the conventional package.

Removed​

  • PIVOT-4707 Removed PointIndexedRecordBlock destroyer methods.
  • PIVOT-4707 Removed IAggregateSource interface.
  • PIVOT-4707 Removed a constructor in AWritableFixedSizeRecordBlock.
  • PIVOT-4707 Removed transferFrom(IArrayReader source, int sourceIndex, IWritableByteRecord record) from class ARecordFieldHandler.
  • PIVOT-4718 Drop legacy methods from IByteRecordFormat.

Fixed​

  • PIVOT-3790 Fix the partitioning levels of the legacy post-processor NeighborValuePostProcessor.
  • PIVOT-4177 Prevent two Store Lookups from colliding in the query cache if they ask for different fields on the same store.
  • PIVOT-4344 Fix SingleValueFunction on Arrays type.
  • PIVOT-4364 Fix Real-Time issues with Copper. This includes the following corrections:
    • In addition to the CommitIsolatedStoreHandler handler, join measures now include the StoreMeasureHandler in their handlers' list
    • Copper#newPostProcessor(..) allows now to propagate the underlying measures handlers using CopperPostProcessor#withInheritedContinuousQueryHandlers()
    • Delegate continuous handler initialization to post-processors
  • PIVOT-4452 Fix an issue where exported memory statistics could not be read by MAC.
  • PIVOT-4690 Restore allocators that where removed in the previous release: HeapBufferChunkAllocator, DirectBufferChunkAllocator and DirectChunkAllocator.
  • PIVOT-4691 Fix possible casting exception when using CopperMeasure.withType(ILiteralType.BOOLEAN) with an aggregation function returning Boolean.
  • PIVOT-4710 Copper.storeLookup(...) (i.e. StoreLookupMeasure) now handles real time updates.
  • PIVOT-4719 Fix lazy initialization and double check idiom in RecordFormat#getFieldIndex(String).

Security​

  • APS-12594 BREAKING The XML parsers used throughout ActivePivot (JAXB, XMLA Servlet...) are now given a restrictive configuration to better prevent XXE injection attacks:
    • Automatic DDoS prevention. This may limit XML constructs, such as deep recursive XML elements (XML entities calling more XML entities, and so on).
    • Disables !DOCTYPE annotation to prevent XXE injections through this annotation.
    • Disables the XInclude annotation to prevent construction of sub-documents that could be retrieved from a distant server.
    • Disables the use of external Document Type Definitions, for the same reason.

5.10.0​

2021-02-26

Added​

  • APS-12558 Add new MdxContext parameters that can override the visibility of hierarchies and dimensions, forcing them to be shown or hidden regardless of the cube's configuration.
  • APS-12866 Rename NotificationMember#AWAIT_NOTIFICATIONS to IDataClusterDefinition#AWAIT_NOTIFICATIONS_TIME. The property can now be declared in data cluster definitions using ICanBuildDataClusterDescriptionBuilder#withProperty.
  • APS-12879 Expressions like Copper.sum("pnl").publish(context) have to be replaced with Copper.sum("pnl").withName("pnl.SUM").publish(context).
  • APS-12908 Add a selectDistinct() method to DatastoreQueryHelper that accepts a condition.
  • PIVOT-3352 Add fencing on AUniqueRecordIndexTransaction methods.
  • PIVOT-3978 Changed when the 'force discard' was called in KeepLastEpochPolicy.
  • PIVOT-3978 IPrimaryRecordIndexVersion#getKeyFields() moved to IPrimaryRecordIndex#getFields.
  • PIVOT-3978 Rename ARemoveUnknowKeyListener to ARemoveUnknownKeyListener to fix typo.
  • PIVOT-4177 Add the possibility to define a Store Lookup in Copper. See Copper.storeLookup(storeName).
  • PIVOT-4180 Properly shutdown DataNodes in a distributed cluster. When a data node is being shutdown, it will send a GoodbyeMessage to the query node which will immediately trigger an asynchronous task responsible of removing all of its subsequent contributions.
  • PIVOT-4220 Add on-the-fly discard of transient results during query execution.
  • PIVOT-4280 Support copper join on any level. It was not possible to do a join on a bucketing level for instance.
  • PIVOT-4356 Join Hierarchies with multiple levels can now be published with Copper, using the unified API Copper.newHierarchy(...).
  • PIVOT-4357 User-defined aggregate functions can now perform dynamic aggregation on measures.
  • PIVOT-4409 Allow to define an Analysis hierarchy on the base store (withStoreName method)
  • PIVOT-4417 Add a new compression for chunks, storing only values different from a frequent value.
  • PIVOT-4438 Update MDX engine error messages to use hierarchy unique names instead of simple names.
  • PIVOT-4476 Add the possibility to specify the fields of the records given to the StoreTransactionBuilder in Copper testing with the method withFields().
  • PIVOT-4480 Add a JMX operation in the store monitoring MBean that produces a general chunk memory usage report. MAC is still recommended for detailed memory analysis.
  • PIVOT-4490 Add health events to the JDBC source notifying the success or the failure of a JDBC source fetch.
  • PIVOT-4491 Added support for primitive aggregated measures in the Aggregates Cache.
  • PIVOT-4546 All licenses must have an end date.
  • PIVOT-4549 Add truncate method to ITransactionManager for deleting all records from a store.
  • PIVOT-4566 Add a fluent builder withSharedContextValues(values...) allowing the definition of multiple shared context values at once avoiding repetitive calls to withSharedContextValue(value).
  • PIVOT-4569 Add Copper.timestamp() to access timestamp measure.
  • PIVOT-4589 Add support in the parquet source for filtering parsed files using a java.util.regex.Pattern, through IParquetParserBuilder.withPattern(java.util.regex.Pattern). These health events can be handled using the interface IJdbcSourceHealthEventHandler.
  • PIVOT-4599 Add IClearableBitmap interface for non-incremental add and clear operations.
  • PIVOT-4600 Add MultiVersionLongArray.

Changed​

  • APS-12788 AxisHierarchy now reports that a field used to create a level should be dictionarized, instead of stating that this operation is only available in unit-tests.
  • APS-12918 Add more parser-policy-related interruption checks to the CSV source's ParserTasks. This can considerably improve performance when using the sampling parser policy.
  • APS-12932 Bitmap operations now catch CancellationException early to prevent logging bitmaps that are valid.
  • APS-12943 Change the default behavior of QueriesResultLimit context value from #withoutLimit() to #defaultLimit(). Now, we also allow -1 value for a limit to represent no-limit on QueriesResultLimit context value.
  • APS-12946 The DistributedEpochManager added in PIVOT-4501 is visible in the Manager MBean under the corresponding distributed cube's MBean.
  • APS-12975 Reduce DiscardOnGcTask timeout duration from one hour to one minute. In very rare cases, this will let a version be discarded more rapidly after being released.
  • PIVOT-3914 Change AAdvancedPostProcessor.computeOutputType() to fall back to the return type of its evaluator, if it has one.
  • PIVOT-4191 Rename methods in QueriesResultLimit.
  • PIVOT-4356 Bucketing Hierarchies must now be published using the unified API Copper.newHierarchy(...).fromValues(...).
  • PIVOT-4375 The Azure Cloud Source has been migrated from Azure Blob Storage SDK v8 to v12. Please refer to the migration notes to see what has changed.
  • PIVOT-4402 Remove static getters from JoinMeasureDescription.
  • PIVOT-4410 BREAKING: Changes were made on the hierarchy builder. hierarchyBuilder.factless().withStoreName("myStore") was replaced by hierarchyBuilder.fromStore("myStore").
  • PIVOT-4464 Remove ActiveMonitor from the sandbox projects.
  • PIVOT-4501 Enhance version management in a distributed setup. Versions are now handled in the Query Cube using a dedicated DistributedEpochManager that keeps track of versions history and trigger discard tasks on GC.
  • PIVOT-4502 Optimize number of operations in chunk scan methods by computing '~epoch' once before iterating over the rows.
  • PIVOT-4509 Register standard Drillthrough calculators through an extended plugin.
  • PIVOT-4522 Change the signature to create a UDAF to allow allocating vectors off-heap and lazy cloning. One should now call IWritableBuffer.allocateVector instead of IVector.cloneOnHeap(), and IWritableBuffer.readWritableVector(int) instead of IWritableArray.readVector(int).
  • PIVOT-4558 Change the behavior of IHierarchy#isEmpty(). Now, the method checks whether the corresponding hierarchy is empty or not instead of checking for leaf members. This enhances the performance of queries with high cardinality hierarchies.
  • PIVOT-4564 Levels from hidden dimensions and hierarchies are now omitted in XMLA discoveries. This caused an incompatibility with some XMLA clients such as ADOMD.NET.
  • PIVOT-4579 The severity of the log message notifying that a property is missing has been changed from WARNING to INFO.
  • PIVOT-4599 Rename FastLogical to FastLogicalQFSBitmap.
  • PIVOT-4608 Lift limitation on Datastore queries with respect to the combination of indexes. Previously, the Datastore could combine up to 3 indexes when looking for range patterns like a in (...) and b in (...). Now, the maximal limit is configured by ActiveViamProperty.MAX_RANGE_OPERANDS_WITH_MULTIPLE_VALUES and set by default to 10.
  • PIVOT-4620 Fix issue with Hibernate where a high number of input parameters causes server exceptions.
  • PIVOT-4622 Fix ActiveMonitor PointValue commit logic. This adds a consistency check when committing an update on an existing point value. Now, we make sure that the "from" point value and its active value in the database have the same key. Otherwise, the commit is not allowed and an exception is thrown.
  • PIVOT-4626 Fix a bug in the parquet source where an invalid index exception would be thrown if a parquet vector field was not mapped to any field in the fed datastore.

Deprecated​

  • PIVOT-4244 Our web services exposed with SOAP are deprecated. Your are encourage to start using the REST services.
  • PIVOT-4244 XML representation of the cube configuration and our DTOs.
  • PIVOT-4679 Ability to filter the facts sent to a cube (ActivePivotInstanceDescription.setFilterDescription).

Removed​

  • APS-12686 Remove JGroupsMessenger.
  • PIVOT-3978 Remove the Transaction Log.
  • PIVOT-3978 Remove legacy TransactionWrapper.
  • PIVOT-4428 Remove following methods:
    • IChunk.findRowsInTransactionEqualTo(Object, IIntIterator, int, IRowMapping, int, IBitmap)
    • IChunkSet.findRowsInTransactionEqualTo(int, Object, IIntIterator, int, int, IBitmap)
    • IChunk.findRowsInTransactionInSet(Set, IIntIterator, int, IRowMapping, int, IBitmap)
    • IChunkSet.findRowsInTransactionInSet(int, Set, IIntIterator, int, int, IBitmap) as duplicates of their counterpart outside of a transaction (findRowsEqualTo, ...)
  • PIVOT-4560 Remove duplicate javax.xml.stream dependency in serialization-xstream (this dependency is included in the JRE).
  • PIVOT-4615 Remove heavy-to-compute logging in ImpactComputationTask.
  • PIVOT-4669 Remove deprecated MapAggregatesProvider.

Fixed​

  • APS-12090 Fix issue with branch permission manager preventing to see new branches on real time Mdx queries.
  • APS-12909 Fix a memory leak within the catch up listener's mechanism.
  • APS-12911 Fix postprocessors auto-partitioning in the the planning phase.
  • APS-12912 Fix a bug where spring boot would occasionally fail with message BeanCurrentlyInCreationException: Error creating bean with name 'resourceMapping': Requested bean is currently in creation. when importing multiple configuration classes inheriting ASpringResourceServerConfig.
  • APS-12915 Correctly process Cube Filters in a distributed application.
  • APS-12931 Fix an erroneous check that would throw when trying to define user-defined aggregate functions in partial aggregate providers.
  • APS-12937 The FullRefreshEvent will now include the set of all impacted measures. This allows to distinguish between measures not impacted by the real-time event and measures with null update.
  • APS-12962 Fix bug preventing measure publication when using withType(...) on a calculated measure, and reusing this measure in another calculated measure.
  • APS-12965 Fix Copper Join producing incorrect aggregation results when using a Join Measure based on the key fields of the joined Store.
  • APS-12966 Fix an issue where formatters would not be taken into account for UDAF measures.
  • APS-12985 Correct overflow error in dictionaries occurring when inserted more than 2 ^ 30 values in a field dictionary.
  • PIVOT-4241 Fix the reserved memory tracking of the SlabMemoryAllocator when the cache is evicted.
  • PIVOT-4243 Fix the reserved memory tracking of the INativeMemoryAllocator classes to track the reserved memory and not only the used memory. The tracking is done at the lowest level for more accuracy.
  • PIVOT-4370 The Parquet Source now correctly handles empty strings when writing into a store field that should not support empty strings, replacing them with the store field's default value.
  • PIVOT-4388 Changed the return type of the Store Builder method withVectorField when chained after a dictionarized or indexed vector field. It gives access to a much fuller API to configure this second vector field, or finish configuring the store with methods like withoutKey().
  • PIVOT-4427 Fix ChunkedBitmap#equals implementation.
  • PIVOT-4436 Fix ClassCastException occurring when a join measure was defined on a dictionarized field.
  • PIVOT-4453 ActivePivot now correctly unregister all its datastore continuous selections when stopped.
  • PIVOT-4455 Correctly compute vector average, ignoring null vectors instead of counting them as actual values.
  • PIVOT-4456 PIVOT-4457 Correct Datastore conditions testing for equality or greater-than/less-than on nullable fields. Previously, this could:
    • throw an exception for a non-dictionarized field with long series of null values;
    • return incorrect rows from the store when looking for the primitive value 0 on a non-dictionarized field;
    • return rows with null for greater-than/less-than conditions.
  • PIVOT-4460 Fix a bug where MemoryAnalysisService would not produce report files for query cubes in distributed application.
  • PIVOT-4461 Fix NullPointerException during Datastore queries looking for null in store fields after compression. This only happened in the case of offset compression, for int and long values or dictionarized fields.
  • PIVOT-4463 Fix IllegalStateException (No free or removed slots available) in MergingStorePartitionUpdate.
  • PIVOT-4500 Fix filter translation edge case in distribution. This caused AND filtering operation on the same hierarchy to be wrongfully translated into an OR operation.
  • PIVOT-4503 Fix a bug where the batched parquet parser could throw an end-of-file exception when reading a vector column.
  • PIVOT-4503 Fix a bug where the batched parquet parser would not properly read row group and page metadata, potentially throwing an end-of-file exception.
  • PIVOT-4507 Fix issue with mdx drillthrough query involving a member whose name contains brackets.
  • PIVOT-4516 UDAF used in a aggregate provider always stored its vectors on heap.
  • PIVOT-4517 Fix a bug where an ArrayIndexOutOfBoundsException would be thrown when generating the memory statistic of an empty target store referenced by a non-empty owner store.
  • PIVOT-4525 CopperMeasure#parentValueOn(CopperHierarchy) now properly handles an analysis hierarchy being passed as an argument.
  • PIVOT-4527 Fix XMLA discovery of UDAF measures.
  • PIVOT-4532 Fix the computation of retrieval results size in the query plan. The total result size for partitioned retrievals were not accumulated together correctly and an empty result for a partition was causing a NPE.
  • PIVOT-4533 Fix a bug in Copper where an UnsupportedOperationException would be thrown when subtracting a float, long, or int vector to a null vector aggregate.
  • PIVOT-4534 Prevent PrimitiveAnalysisAggregationRetrieval from inheriting not required measures from its underlying retrieval. This prevents mismatch binding when a parent PrimitiveAnalysisAggregationRetrieval depends on a child PrimitiveAnalysisAggregationRetrieval including additional join measures. It also improves the query times and memory usage for such cases.
  • PIVOT-4542 Improve error message when QueriesResultLimits are exceeded.
  • PIVOT-4550 Fix a Copper bug where measures could not be published.
  • PIVOT-4563 Fix a bug where the Left and Right MDX functions would throw a NullPointerException for null values.
  • PIVOT-4565 Fix a bug where enforcing a parallel computation of a Copper join could make a query timeout.
  • PIVOT-4572 Fix ClassCastException during the execution of MDX queries relying on previously cached aggregates. The issue is due to a bad partitioning of a post-processor depending on a cached node and a non-partitioned post-processor node. Now, cached nodes forbid all partitioning levels and bubble them up to their respective parents.
  • PIVOT-4586 Fix NullPointerException in DrillthroughResultDTO.
  • PIVOT-4590 Fix ArrayIndexOutOfBoundsException occurring when merging two retrieval results. Particularly, the issue was due to the fact that some post-processors ignore null values and don't write them into the final result. As a consequence, the capacity of the underlying post-processor column won't be aligned with the retrievals points, causing such exception when merging the actual result with another one.
  • PIVOT-4592 Fix Copper errors when reusing code to build complex measures.
  • PIVOT-4593 Fix a bug where the batched parquet parser would read incorrect values for all subsequent vectors if a column contained a null or empty vector.
  • PIVOT-4633 Fix ClassCastException in DataCubeAdapter.
  • PIVOT-4633 BREAKING: The method Location#getArrayLocation() has been removed. Please use the method ILocation#getCoordinate(hierarchyIdx, levelIndex) instead.
  • PIVOT-4649 Fix issue with DrillthroughQuery headers incorrectly returning the type of a column.
  • PIVOT-4671 The Query cube will now always reject messages and contributions from unknown applications.
  • PIVOT-4678 Fix a bug where similar Copper measures using different constants were confused with each other.