> ## Documentation Index
> Fetch the complete documentation index at: https://docs.activeviam.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Changelog 5.10

## 5.10.2

2021-04-30

### Added

* PIVOT-4486 Add the possibility to apply query cube filter on the prefetch of Copper analytic functions using the API `CopperAnalyticMeasure#withFilter(AnalyticFilter)`.
* PIVOT-4713 Add the method `IDistributedHierarchyManager#waitForAllMessages` allowing to wait for all the recorded messages to be processed.

### Changed

* PIVOT-4300 Post-processor dependency graph cycle detection is now enabled by default.
* PIVOT-4780 Change MBeans in `QueryPlanMonitoring` to print execution plan and timing full or as summary, e.g. `QueryPlanMonitoring#enableExecutionTimingPrintFull()` or `QueryPlanMonitoring#enableExecutionTimingPrintAsSummary()`.

### Fixed

* APS-13016 Better handle network errors which were causing issues with license checks.
* PIVOT-4717 Fix UDAF error that could occur when totally emptying the underlying store.
* PIVOT-4725 Some datastore versions were not unmarked after stopping the cubes.
* PIVOT-4728 Measure names with several parentheses were not correctly supported by the Formula PostProcessor.
* PIVOT-4730 Prevent Real-Time statistics errors from crashing continuous queries update. The issue happened when statistics computations tried to fetch retrievals that had been dismissed by the query engine (usually because of a non-existent location).
* PIVOT-4739 Fix publishing of WindowReduceMeasure (description equality).
* PIVOT-4749 Clarify the documentation of the `cube/export/mdx/download` REST service endpoint.
* PIVOT-4760 Fix an issue where `IStoreToParquetMapping` would map fields from the datastore to the wrong fields in the parquet schema.
* PIVOT-4766 Fix Frequent Chunk Compression bug where nested `FrequentNullDoubleChunk` could be created.
* PIVOT-4771 Fix NPE cause by `CopperPostProcessor#withInheritedContinuousQueryHandlers`. The issue happened while inheriting the handlers of a post-processor that hasn't any.
* PIVOT-4772 Fix a query issue where wildcards on slicing analysis hierarchies based on isolated store fields would conflict with Copper bucketing hierarchies.
* PIVOT-4775 Fix an issue with `AAdvancedPostProcessor` performing an eager modification of the prefetched cube filter.
* PIVOT-4775 Fix an issue with the query engine where some retrievals handing analysis hierarchies, were wrongly declaring their dependencies.
* PIVOT-4791 All datastore versions are now synchronously unmarked when stopping the pivot.

## 5.10.1

2021-03-26

### Added

* PIVOT-4473 Add loggers for CREATE, UPDATE and DROP MDX statements
* PIVOT-4638 Add the `maskAtOrBelow()` method to the `CopperMeasureToAggregateAbove` interface, allowing to define levels below which a measure will not return aggregated values when queried. The behavior is the complementary of the `doNotAggregateAbove()` method.
* PIVOT-4702 Add support for DATEPART function in MDX with the simple signature DatePart(interval, date).

### Changed

* PIVOT-4705 Make `Records#getGlobalDefaultValue` use `int` instead of `ILiteralType`.
* PIVOT-4707 `PointIndexedRecordBlock` no longer implements `IChunkSet`.
* PIVOT-4707 Renamed `ImmutableRecordList` in `ImmutableRecordBlock` to match the pattern established in sibling classes.

### Deprecated

* PIVOT-4705 Deprecate `SchemaHelper#undictionarize` in favour of `Records#undictionarize`.
* PIVOT-4724 Deprecate `com.qfs.pivot.impl.ActivePivotSchemaExtendedPlugin` in favour of `com.quartetfs.biz.pivot.impl.ActivePivotSchemaExtendedPlugin` located in the conventional package.

### Removed

* PIVOT-4707 Removed `PointIndexedRecordBlock` destroyer methods.
* PIVOT-4707 Removed `IAggregateSource` interface.
* PIVOT-4707 Removed a constructor in `AWritableFixedSizeRecordBlock`.
* PIVOT-4707 Removed `transferFrom(IArrayReader source, int sourceIndex, IWritableByteRecord record)` from class `ARecordFieldHandler`.
* PIVOT-4718 Drop legacy methods from `IByteRecordFormat`.

### Fixed

* PIVOT-3790 Fix the partitioning levels of the legacy post-processor `NeighborValuePostProcessor`.
* PIVOT-4177 Prevent two Store Lookups from colliding in the query cache if they ask for different fields on the same store.
* PIVOT-4344 Fix SingleValueFunction on Arrays type.
* PIVOT-4364 Fix Real-Time issues with Copper. This includes the following corrections:
  * In addition to the `CommitIsolatedStoreHandler` handler, join measures now include the `StoreMeasureHandler` in their handlers' list
  * `Copper#newPostProcessor(..)` allows now to propagate the underlying measures handlers using `CopperPostProcessor#withInheritedContinuousQueryHandlers()`
  * Delegate continuous handler initialization to post-processors
* PIVOT-4452 Fix an issue where exported memory statistics could not be read by MAC.
* PIVOT-4690 Restore allocators that where removed in the previous release: `HeapBufferChunkAllocator`, `DirectBufferChunkAllocator` and `DirectChunkAllocator`.
* PIVOT-4691 Fix possible casting exception when using `CopperMeasure.withType(ILiteralType.BOOLEAN)` with an aggregation function returning `Boolean`.
* PIVOT-4710 `Copper.storeLookup(...)` (i.e. `StoreLookupMeasure`) now handles real time updates.
* PIVOT-4719 Fix lazy initialization and double check idiom in `RecordFormat#getFieldIndex(String)`.

### Security

* APS-12594 **BREAKING** The XML parsers used throughout ActivePivot
  (JAXB, XMLA Servlet...) are now given a restrictive configuration to better prevent XXE injection attacks:
  * Automatic DDoS prevention. This may limit XML constructs, such as deep recursive XML elements (XML entities calling more XML entities, and so on).
  * Disables `!DOCTYPE` annotation to prevent XXE injections through this annotation.
  * Disables the `XInclude` annotation to prevent construction of sub-documents that could be retrieved from a distant server.
  * Disables the use of external Document Type Definitions, for the same reason.

## 5.10.0

2021-02-26

### Added

* APS-12558 Add new `MdxContext` parameters that can override the visibility of hierarchies and dimensions, forcing them to be shown or hidden regardless of the cube's configuration.
* APS-12866 Rename `NotificationMember#AWAIT_NOTIFICATIONS` to `IDataClusterDefinition#AWAIT_NOTIFICATIONS_TIME`. The property can now be declared in data cluster definitions using `ICanBuildDataClusterDescriptionBuilder#withProperty`.
* APS-12879 Expressions like `Copper.sum("pnl").publish(context)` have to be replaced with `Copper.sum("pnl").withName("pnl.SUM").publish(context)`.
* APS-12908 Add a `selectDistinct()` method to `DatastoreQueryHelper` that accepts a condition.
* PIVOT-3352 Add fencing on `AUniqueRecordIndexTransaction` methods.
* PIVOT-3978 Changed when the 'force discard' was called in `KeepLastEpochPolicy`.
* PIVOT-3978 `IPrimaryRecordIndexVersion#getKeyFields()` moved to `IPrimaryRecordIndex#getFields`.
* PIVOT-3978 Rename `ARemoveUnknowKeyListener` to `ARemoveUnknownKeyListener` to fix typo.
* PIVOT-4177 Add the possibility to define a Store Lookup in Copper. See `Copper.storeLookup(storeName)`.
* PIVOT-4180 Properly shutdown `DataNodes` in a distributed cluster. When a data node is being shutdown, it will send a `GoodbyeMessage` to the query node which will immediately trigger an asynchronous task responsible of removing all of its subsequent contributions.
* PIVOT-4220 Add on-the-fly discard of transient results during query execution.
* PIVOT-4280 Support copper join on any level. It was not possible to do a join on a bucketing level for instance.
* PIVOT-4356 Join Hierarchies with multiple levels can now be published with Copper, using the unified API `Copper.newHierarchy(...)`.
* PIVOT-4357 User-defined aggregate functions can now perform dynamic aggregation on measures.
* PIVOT-4409 Allow to define an Analysis hierarchy on the base store (`withStoreName` method)
* PIVOT-4417 Add a new compression for chunks, storing only values different from a frequent value.
* PIVOT-4438 Update MDX engine error messages to use hierarchy unique names instead of simple names.
* PIVOT-4476 Add the possibility to specify the fields of the records given to the `StoreTransactionBuilder` in Copper testing with the method `withFields()`.
* PIVOT-4480 Add a JMX operation in the store monitoring MBean that produces
  a general chunk memory usage report. MAC is still recommended for detailed memory analysis.
* PIVOT-4490 Add health events to the JDBC source notifying the success or the failure of a JDBC source fetch.
* PIVOT-4491 Added support for primitive aggregated measures in the Aggregates Cache.
* PIVOT-4546 All licenses must have an end date.
* PIVOT-4549 Add `truncate` method to `ITransactionManager` for deleting all records from a store.
* PIVOT-4566 Add a fluent builder `withSharedContextValues(values...)` allowing the definition of multiple shared context values at once avoiding repetitive calls to `withSharedContextValue(value)`.
* PIVOT-4569 Add `Copper.timestamp()` to access timestamp measure.
* PIVOT-4589 Add support in the parquet source for filtering parsed files using a `java.util.regex.Pattern`, through `IParquetParserBuilder.withPattern(java.util.regex.Pattern)`. These health events can be handled using the interface `IJdbcSourceHealthEventHandler`.
* PIVOT-4599 Add `IClearableBitmap` interface for non-incremental add and clear operations.
* PIVOT-4600 Add `MultiVersionLongArray`.

### Changed

* APS-12788 `AxisHierarchy` now reports that a field used to create a level should be dictionarized, instead of stating that this operation is only available in unit-tests.
* APS-12918 Add more parser-policy-related interruption checks to the CSV source's `ParserTask`s. This can considerably improve performance when using the sampling parser policy.
* APS-12932 Bitmap operations now catch `CancellationException` early to prevent logging bitmaps that are valid.
* APS-12943 Change the default behavior of `QueriesResultLimit` context value from `#withoutLimit()` to `#defaultLimit()`. Now, we also allow -1 value for a limit to represent no-limit on `QueriesResultLimit` context value.
* APS-12946 The `DistributedEpochManager` added in PIVOT-4501 is visible in the Manager MBean under the corresponding distributed cube's MBean.
* APS-12975 Reduce `DiscardOnGcTask` timeout duration from one hour to one minute. In very rare cases, this will let a version be discarded more rapidly after being released.
* PIVOT-3914 Change `AAdvancedPostProcessor.computeOutputType()` to fall back to the return type of its evaluator, if it has one.
* PIVOT-4191 Rename methods in `QueriesResultLimit`.
* PIVOT-4356 Bucketing Hierarchies must now be published using the unified API `Copper.newHierarchy(...).fromValues(...)`.
* PIVOT-4375 The Azure Cloud Source has been migrated from Azure Blob Storage SDK v8 to v12. Please refer to the migration notes to see what has changed.
* PIVOT-4402 Remove static getters from `JoinMeasureDescription`.
* PIVOT-4410 **BREAKING**: Changes were made on the hierarchy builder. `hierarchyBuilder.factless().withStoreName("myStore")` was replaced by `hierarchyBuilder.fromStore("myStore")`.
* PIVOT-4464 Remove ActiveMonitor from the sandbox projects.
* PIVOT-4501 Enhance version management in a distributed setup. Versions are now handled in the Query Cube using a dedicated `DistributedEpochManager` that keeps track of versions history and trigger discard tasks on GC.
* PIVOT-4502 Optimize number of operations in chunk scan methods by computing '\~epoch' once before iterating over the rows.
* PIVOT-4509 Register standard Drillthrough calculators through an extended plugin.
* PIVOT-4522 Change the signature to create a UDAF to allow allocating vectors off-heap and lazy cloning. One should now call `IWritableBuffer.allocateVector` instead of `IVector.cloneOnHeap()`, and `IWritableBuffer.readWritableVector(int)` instead of `IWritableArray.readVector(int)`.
* PIVOT-4558 Change the behavior of `IHierarchy#isEmpty()`. Now, the method checks whether the corresponding hierarchy is empty or not instead of checking for leaf members. This enhances the performance of queries with high cardinality hierarchies.
* PIVOT-4564 Levels from hidden dimensions and hierarchies are now omitted in XMLA discoveries. This caused an incompatibility with some XMLA clients such as ADOMD.NET.
* PIVOT-4579 The severity of the log message notifying that a property is missing has been changed from WARNING to INFO.
* PIVOT-4599 Rename `FastLogical` to `FastLogicalQFSBitmap`.
* PIVOT-4608 Lift limitation on Datastore queries with respect to the combination of indexes. Previously, the Datastore could combine up to 3 indexes when looking for range patterns like `a in (...) and b in (...)`. Now, the maximal limit is configured by `ActiveViamProperty.MAX_RANGE_OPERANDS_WITH_MULTIPLE_VALUES` and set by default to 10.
* PIVOT-4620 Fix issue with Hibernate where a high number of input parameters causes server exceptions.
* PIVOT-4622 Fix ActiveMonitor PointValue commit logic. This adds a consistency check when committing an update on an existing point value. Now, we make sure that the "from" point value and its active value in the database have the same key. Otherwise, the commit is not allowed and an exception is thrown.
* PIVOT-4626 Fix a bug in the parquet source where an invalid index exception would be thrown if a parquet vector field was not mapped to any field in the fed datastore.

### Deprecated

* PIVOT-4244 Our web services exposed with SOAP are deprecated. Your are encourage to start using the REST services.
* PIVOT-4244 XML representation of the cube configuration and our DTOs.
* PIVOT-4679 Ability to filter the facts sent to a cube (`ActivePivotInstanceDescription.setFilterDescription`).

### Removed

* APS-12686 Remove `JGroupsMessenger`.
* PIVOT-3978 Remove the Transaction Log.
* PIVOT-3978 Remove legacy `TransactionWrapper`.
* PIVOT-4428 Remove following methods:
  * `IChunk.findRowsInTransactionEqualTo(Object, IIntIterator, int, IRowMapping, int, IBitmap)`
  * `IChunkSet.findRowsInTransactionEqualTo(int, Object, IIntIterator, int, int, IBitmap)`
  * `IChunk.findRowsInTransactionInSet(Set, IIntIterator, int, IRowMapping, int, IBitmap)`
  * `IChunkSet.findRowsInTransactionInSet(int, Set, IIntIterator, int, int, IBitmap)`
    as duplicates of their counterpart outside of a transaction (`findRowsEqualTo`, ...)
* PIVOT-4560 Remove duplicate `javax.xml.stream` dependency in `serialization-xstream` (this dependency is included in the JRE).
* PIVOT-4615 Remove heavy-to-compute logging in `ImpactComputationTask`.
* PIVOT-4669 Remove deprecated `MapAggregatesProvider`.

### Fixed

* APS-12090 Fix issue with branch permission manager preventing to see new branches on real time Mdx queries.
* APS-12909 Fix a **memory leak** within the catch up listener's mechanism.
* APS-12911 Fix postprocessors auto-partitioning in the the planning phase.
* APS-12912 Fix a bug where spring boot would occasionally fail with message `BeanCurrentlyInCreationException: Error creating bean with name 'resourceMapping': Requested bean is currently in creation.` when importing multiple configuration classes inheriting `ASpringResourceServerConfig`.
* APS-12915 Correctly process Cube Filters in a distributed application.
* APS-12931 Fix an erroneous check that would throw when trying to define user-defined aggregate functions in partial aggregate providers.
* APS-12937 The `FullRefreshEvent` will now include the set of all impacted measures. This allows to distinguish between measures not impacted by the real-time event and measures with `null` update.
* APS-12962 Fix bug preventing measure publication when using `withType(...)` on a calculated measure, and reusing this measure in another calculated measure.
* APS-12965 Fix Copper Join producing incorrect aggregation results when using a Join Measure based on the key fields of the joined Store.
* APS-12966 Fix an issue where formatters would not be taken into account for UDAF measures.
* APS-12985 Correct overflow error in dictionaries occurring when inserted more than `2 ^ 30` values in a field dictionary.
* PIVOT-4241 Fix the reserved memory tracking of the `SlabMemoryAllocator` when the cache is evicted.
* PIVOT-4243 Fix the reserved memory tracking of the `INativeMemoryAllocator` classes to track the reserved memory and not only the used memory. The tracking is done at the lowest level for more accuracy.
* PIVOT-4370 The Parquet Source now correctly handles empty strings when writing into a store field that should not support empty strings, replacing them with the store field's default value.
* PIVOT-4388 Changed the return type of the Store Builder method `withVectorField` when chained after a dictionarized or indexed vector field. It gives access to a much fuller API to configure this second vector field, or finish configuring the store with methods like `withoutKey()`.
* PIVOT-4427 Fix `ChunkedBitmap#equals` implementation.
* PIVOT-4436 Fix `ClassCastException` occurring when a join measure was defined on a dictionarized field.
* PIVOT-4453 ActivePivot now correctly unregister all its datastore continuous selections when stopped.
* PIVOT-4455 Correctly compute vector average, ignoring null vectors instead of counting them as actual values.
* PIVOT-4456 PIVOT-4457 Correct Datastore conditions testing for equality or greater-than/less-than on nullable fields. Previously, this could:
  * throw an exception for a non-dictionarized field with long series of `null` values;
  * return incorrect rows from the store when looking for the primitive value 0 on a non-dictionarized field;
  * return rows with `null` for greater-than/less-than conditions.
* PIVOT-4460 Fix a bug where `MemoryAnalysisService` would not produce report files for query cubes in distributed application.
* PIVOT-4461 Fix `NullPointerException` during Datastore queries looking for `null` in store fields after compression. This only happened in the case of offset compression, for int and long values or dictionarized fields.
* PIVOT-4463 Fix `IllegalStateException`  (No free or removed slots available) in `MergingStorePartitionUpdate`.
* PIVOT-4500 Fix filter translation edge case in distribution. This caused `AND` filtering operation on the same hierarchy to be wrongfully translated into an `OR` operation.
* PIVOT-4503 Fix a bug where the batched parquet parser could throw an end-of-file exception when reading a vector column.
* PIVOT-4503 Fix a bug where the batched parquet parser would not properly read row group and page metadata, potentially throwing an end-of-file exception.
* PIVOT-4507 Fix issue with mdx drillthrough query involving a member whose name contains brackets.
* PIVOT-4516 UDAF used in a aggregate provider always stored its vectors on heap.
* PIVOT-4517 Fix a bug where an `ArrayIndexOutOfBoundsException` would be thrown when generating the memory statistic of an empty target store referenced by a non-empty owner store.
* PIVOT-4525 `CopperMeasure#parentValueOn(CopperHierarchy)` now properly handles an analysis hierarchy being passed as an argument.
* PIVOT-4527 Fix XMLA discovery of UDAF measures.
* PIVOT-4532 Fix the computation of retrieval results size in the query plan. The total result size for partitioned retrievals were not accumulated together correctly and an empty result for a partition was causing a NPE.
* PIVOT-4533 Fix a bug in Copper where an `UnsupportedOperationException` would be thrown when subtracting a float, long, or int vector to a null vector aggregate.
* PIVOT-4534 Prevent `PrimitiveAnalysisAggregationRetrieval` from inheriting not required measures from its underlying retrieval. This prevents mismatch binding when a parent `PrimitiveAnalysisAggregationRetrieval` depends on a child `PrimitiveAnalysisAggregationRetrieval` including additional join measures. It also improves the query times and memory usage for such cases.
* PIVOT-4542 Improve error message when `QueriesResultLimit`s are exceeded.
* PIVOT-4550 Fix a Copper bug where measures could not be published.
* PIVOT-4563 Fix a bug where the `Left` and `Right` MDX functions would throw a
  `NullPointerException` for `null` values.
* PIVOT-4565 Fix a bug where enforcing a parallel computation of a Copper join could make a query timeout.
* PIVOT-4572 Fix `ClassCastException` during the execution of MDX queries relying on previously cached aggregates. The issue is due to a bad partitioning of a post-processor depending on a cached node and a non-partitioned post-processor node. Now, cached nodes forbid all partitioning levels and bubble them up to their respective parents.
* PIVOT-4586 Fix `NullPointerException` in DrillthroughResultDTO.
* PIVOT-4590 Fix `ArrayIndexOutOfBoundsException` occurring when merging two retrieval results. Particularly, the issue was due to the fact that some post-processors ignore `null` values and don't write them into the final result. As a consequence, the capacity of the underlying post-processor column won't be aligned with the retrievals points, causing such exception when merging the actual result with another one.
* PIVOT-4592 Fix Copper errors when reusing code to build complex measures.
* PIVOT-4593 Fix a bug where the batched parquet parser would read incorrect values for all subsequent vectors if a column contained a `null` or empty vector.
* PIVOT-4633 Fix `ClassCastException` in DataCubeAdapter.
* PIVOT-4633 **BREAKING**: The method `Location#getArrayLocation()` has been removed. Please use the method `ILocation#getCoordinate(hierarchyIdx, levelIndex)` instead.
* PIVOT-4649 Fix issue with DrillthroughQuery headers incorrectly returning the type of a column.
* PIVOT-4671 The Query cube will now always reject messages and contributions from unknown applications.
* PIVOT-4678 Fix a bug where similar Copper measures using different constants were confused with each other.