Changelog 5.11
The 5.11 branch forked the 5.10 branch from version 5.10.2
See the 5.10 changelog for changes before the fork.
5.11.4
2022-09-26
Added
- PIVOT-4365 Add an
MdxQueryBouncer
limiting the number of concurrently running mdx queries. See What's New? page for configuration details. - PIVOT-5027 Empty partitions are automatically dropped at the end of transactions.
- PIVOT-5077 Add
IDistributedActivePivotInstanceDescription.NO_REPLICATION_MEASURES
property to disable replication for certain measures on distributed cubes. - PIVOT-5631
ActiveUIResourceServerConfig
andAdminUIResourceServerConfig
are now core classes. You no more need to copy them from the Sandbox to your projects.
Changed
- PIVOT-4934 Adjust Logging levels for received statements on the MDX REST endpoint to INFO.
- PIVOT-5030 Disable RangeSharing for Post-Processors when an Analysis Hierarchy is expressed in the queried location.
- PIVOT-5304 Improve the performance of the code handling the generation of the locations when analysis hierarchies are expressed.
- PIVOT-5626 Skip initialization of previously failed post-processors
- PIVOT-5657 Switch outer reference policy to auto-detection mode for the queries executed from Datastore MBeans.
- PIVOT-5671 Log the complete APM monitor result's only when detailed logging is enabled.
- PIVOT-5677 Activiti dependencies upgraded from 5.22.0 to 5.23.0.
- PIVOT-5728 Improve JIT retrievals completion performance in some applications with a high partition count.
Fixed
- PIVOT-5041 Clarify the behavior and error message of
Licensing.reload()
and its underlying MBean. - PIVOT-5365 Fix
ActivePivotVersion
releasing its reference to the datastore too early. In some cases, a race condition could lead to wrong results when using aKeepLastEpochPolicy
and querying an old not markedActivePivotVersion
with a JustInTime provider. - PIVOT-5479 Fix
ClassCastException
when querying the drillthrough headers on a distributed cube. - PIVOT-5458 Correctly propagate exception in
ExternalDatastoreRetrieval
. - PIVOT-5502 Fix cached data mutated in DatastoreService.
- PIVOT-5508 Fix the creation of the
ColumnIntegerRowMapping
memory statistics for MAC. - PIVOT-5515 Fix
ArrayIndexOutOfBoundsException
inChunkedBitmap
when adding two bitmaps. - PIVOT-5539 Fix
NullPointerException
when discarding transient results from a distributed measure with no value. - PIVOT-5502 Fix cached data mutated in DatastoreService.
- PIVOT-5575 Fix a performance issue impacting queries on cubes having partial providers and a large amount of hierarchies.
- PIVOT-5639 Fix an issue with nested combine operations with context value dependencies in Copper.
- PIVOT-5640 Fix an issue where unrelated store partitions could generate unnecessary transient objects. This reduces the memory footprint of small transactions.
- PIVOT-5661 Fixed potential deadlock in JDBCTask when all JDBCAppendRunnable crashed.
- PIVOT-5671 Hide sensitive information in logs.
- PIVOT-5713 Fix
ClassCastException
during the recording of distributed messages statistics. - PIVOT-5759 Fix Deadlock when allocating partition content.
- PIVOT-5777 Fix wrong parameter order in
RemoteContentService
.
5.11.3
2022-05-06
Added
- PIVOT-5301
BottomCountFunction
andTopCountFunction
should accept a string expression as a comparator. - PIVOT-5297 Support all arrow
TimeStampVector
types:TimeStampNanoTZVector
,TimeStampMicroTZVector
,TimeStampMilliTZVector
,TimeStampSecTZVector
.
Added
- PIVOT-5301
BottomCountFunction
andTopCountFunction
should accept a string expression as a comparator.
Changed
- PIVOT-5202 Improve the Copper doc on map methods to specify that the cells must be written into at each iteration.
Fixed
- PIVOT-5318 Fix ClassCastException in the PlanningGraph
- PIVOT-5333 Increase severity for full reset and pending initial discovery messages to
WARNING
in distributed clusters. - PIVOT-5395 Fix the creation of the
HashMapRowMapping
memory statistics for MAC. - PIVOT-5407 Fix Frequent Value compression not properly handling null values.
- PIVOT-5428 Fix
ArrayIndexOutOfBoundsException
when usingwithBatchSize
in parquet source. - PIVOT-5443 Fix
SimpleCubeDescriptionBuilder
not properly setting shared context values. - PIVOT-5479 Fix
ClassCastException
when querying the drillthrough headers on a distributed cube.
5.11.2
2022-04-01
Added
- PIVOT-5232 A UI to browse the Datastore. See
AdminUIResourceServerConfig
in the Sandbox. - PIVOT-5246 Add
ChunkSingleBoolean
to compress Boolean chunks containing the same value for all positions.
Changed
- PIVOT-5232 Breaking Moved
JungSchemaPrinter
and associated classes to datastore test artifact.spring-boot
,netty
,google-guava
,azure-identity
,azure-security-keyvault-keys
andazure-storage-blob
are updated to the latest version. - PIVOT-5239 Improve
DirectedGraph.getEdgesTo
. - PIVOT-5275
Record
reading and writing operations now strictly follow the JVM's automatic type widening described in the JVM's spec. The previous behavior was to only allow the operations corresponding to the field type. - PIVOT-5322 Renaming of exposed properties related to consul service configuration within APM.
- PIVOT-5367 Renamed properties
intermediateSize
andtransientSize
inQueryResultLimit
tointermediateLimit
andtransientLimit
, respectively. - PIVOT-5411 Update Spring Boot to 2.6.6 which includes a vulnerability fix.
Removed
- PIVOT-5232 BREAKING
ContentServerResourceServerConfig
was removed. It is replaced byAdminUIResourceServerConfig
(See the Sandbox).
Fixed
- PIVOT-5098 Fixed an
IllegalStateException
when performing queries containing cached aggregates for primitive measures and primitive-aggregated measures. - PIVOT-5188 Fix potential bug in AnalyticFunctionPostProcessor where partitioning levels were not properly set.
- PIVOT-5218 Improve performance of the Mdx NonEmpty function
- PIVOT-5259 Correctly propagate custom aggregation function used for result reduction in distributed post-processors.
- PIVOT-5280 Using the flexible datastore no longer removes the insertion time and commit time update procedures.
- PIVOT-5291 Correctly group monitored JMX APM components and fix some variable visibility issues.
- PIVOT-5308 Clarify the javadoc of
Copper.StoreLookup
when the specified store has nullable key fields. - PIVOT-5309 Correctly set elapsed time in aggregate retrievals.
- PIVOT-5310 Fix retrieval result size figures when printing the query plan. In some cases, results were discarded before their result size computed.
- PIVOT-5311 Fix bug where hierarchy visibility was set to false but was still visible.
- PIVOT-5328 Fix an
NullPointerException
when having multiple threads discarding the same transient result at the same time. - PIVOT-5330 Fix issue with store size and cursor not being printed when using
SchemaPrinter
. - PIVOT-5335 Fix aggregates cache not being updated in Query Cubes after transactions without any impact on the pivot's structure.
- PIVOT-5342 Fix bug where we could not query joined store's lookup measure on grand total.
- PIVOT-5396 The reversed comparators did not return their plugin key.
5.11.1
2022-01-12
Added
- PIVOT-5170 Add result size reporting for distributed query plan print.
Changed
- PIVOT-5009 Prevent users from creating Copper join hierarchies based on a store field included in the mapping.
- PIVOT-5238 Parallelize
ActivePivotManagerBuilder
'spostProcess
to reduce cubes creation time.
Fixed
- PIVOT-5095 Fix real-time update on query with epoch hierarchy. Previously, real-time updates on such queries resulted in branch disappearing from the updated result. See Real-Time What-If Queries section.
- PIVOT-5189 Fix
IIterableAggregatesRetrievalResult#transferValues
that could copy the wrong aggregates. - PIVOT-5195 Fix bug in the revert to version. Some records were not reverted in some edge cases.
- PIVOT-5208 Enforce
PostProcessor
initialization when requesting the dataType of a post-processed measure. - PIVOT-5210 Fix a bug in lead, lag, first, last and window Copper functionalities when relying on a Context Value measure.
- PIVOT-5224 License is now redacted when logged.
5.11.0
2021-11-26
Added
- PIVOT-4449 Added a detailed reporting of CSV parsing errors. Use the
CSV_PARSING_REPORT_ENABLED
property to enable it, andCSV_PARSING_REPORT_MAX_ANOMALIES
to limit its size. - PIVOT-4639 Add parameter to auto-detect outer references in
DatastoreService
. - PIVOT-4642 Integrate APM (Application Performance Monitoring) library into ActivePivot.
- PIVOT-4653 ActivePivot's custom data structures following
com.qfs.chunk.IArray
andcom.qfs.chunk.IWritableArray
can now spawn a cursor. This cursor can be moved up and down the array and perform read/write operations at its current position. - PIVOT-4655 Add a data export service to download or export MDX queries.
- PIVOT-4676 The query plan now includes aggregation procedures time and external datastore retrievals for
PrimitiveAnalysisAggregationRetrieval
. It also includes the name of the used partial provider forPartialPrimitiveAggregatesRetrieval
. - PIVOT-4677 Integrate the Excel Add-in library into ActivePivot.
- PIVOT-4708 Added virtual hierarchies experimental feature. Those are hierarchies whose members are not materialized. A hierarchy can be defined as virtual using the hierarchy builder API
.virtual()
. - PIVOT-4715 Added an interface
ICanUseOtherReferenceOrBuild
in the selection fluent builders for ease of use with Spring Beans. - PIVOT-4733 Allow to override measure aggregator xmla property with
CopperMeasure#withXmlaAggregator(int)
. - PIVOT-4804 Throw an exception when creating measures names with commas.
- PIVOT-4806 License path is now printed in the log.
- PIVOT-4815 Ability to name (give a alias) when selecting field in a
RecordQuery
or aGetByKeyQuery
. Before the name was the path of the field. - PIVOT-4824 Include epochId of the max and latest continuous query statistics.
- PIVOT-4847 Add
mark()
andunmark()
onIDatastoreVersion
. - PIVOT-4913 The JDBC Source will now log the exceptions that make its workers exit the parallel processing of records.
- PIVOT-5037 The Copper tester now supports
hasValues()
in every case and a new methodwithDelta()
to set the allowed delta between the values given by the user inhasValues()
and the actual values returned by the measures. - PIVOT-5059 Partial providers can now have a name using the fluent builder API
PartialBuilder#withName(..)
. - PIVOT-5060 Support encoding and decoding
NaN
and+/-Infinity
in JSON. - PIVOT-5069 Add the currently written record in case of type error in
IRecordFormat#newRecord()
. - PIVOT-5118 Add a toString() to
CloudCsvDataProvider()
for better error message. - PIVOT-5170 Log query results size by default when printing the query plan. Note that
DistributedAggregatesRetrieval
doesn't support such feature yet.
Changed
- PIVOT-3978 Partitioning fields are now automatically dictionarized.
- PIVOT-4315 The Copper Typing API is now simpler, only allowing to call
withType
after operations likeplus
,minus
, etc. - PIVOT-4498 The
ActiveViamPropertyFromSpringConfig.resolve()
bean is now namedActiveViamPropertyFromSpringConfig.SPRING_PROPERTIES_RESOLVER
. - PIVOT-4484 Deprecate usage of
OutputStream
inTableBuilder
and add usage ofStringBuilder
- PIVOT-4653
IVector
andIBlock
no longer extendIWritableArray
. - PIVOT-4653
IPostProcessor
abstract implementations are given a version 2, such asABasicPostProcessorV2
,ADynamicAggregationPostProcessorV2
andAAdvancedPostProcessorV2
. See the migration notes for details. - PIVOT-4747
ReferencedField
was replaced byReachableField
. - PIVOT-4781
StreamSnapshotService#snapShotStreams(String)
is now compatible withStreamSnapshotService#loadStreams(String)
. Now, both operation ignoreSTOPPED
streams. UseStreamSnapshotService#snapShotAllStreams(String)
to snapshot all streams including the stopped ones. - PIVOT-4789 Provide possible solutions in error message when we failed to initialize the
SaferNative
class inLinuxPlatform
. - PIVOT-4802 Reduce the memory consumption of
ModifiedLocation
, primarily used in cumulated aggregates and in shifted measures. - PIVOT-4808 Report error if the ActivePivotSchema fails to attach.
- PIVOT-4813 Update ActiveMonitor REST API documentation to correctly illustrate how a new monitor can be added.
- PIVOT-4820 Change constructors of
KeepLastEpochPolicy
andFlexibleDatastoreEpochPolicy
to useDuration
instead oflong
. - PIVOT-4833
AVectorAggregationFunction.aggregate
method will now receive a vector allocator. - PIVOT-4834 Chunks reading and writing operations now strictly follow the JVM's automatic type widening described in the JVM's spec. The previous behavior was to only allow the operations corresponding to the chunk type.
- PIVOT-4893 Vectors reading and writing operations now strictly follow the JVM's automatic type widening described in the JVM's spec. The previous behavior was to only allow the operations corresponding to the vector type.
- PIVOT-4912 Add default implementation of
CancellableCountedCompleter#afterCompute(Throwable)
callingCancellableCountedCompleter#tryComplete()
. - PIVOT-4947 Return a lighter ActivePivot description in SOAP webservices called
SoapActivePivotDiscovery
. - PIVOT-4959 Reorganize the
IVectorBinding
API. Some methods have seen their twoIVector
parameters inverted, but they now all carry the less confusing names ofleft
andright
(rather than source and destination). Theleft
vector will always receive the result of the vector operation. - PIVOT-4973
IReadableDatastore
now implementsIBranchedMultiVersion
. - PIVOT-5003 Reorganize
AMultiVersionAnalysisHierarchy
andAAnalysisHierarchy
to clearly state the methods to implement. The notable changes is that now, users must overrideAMultiVersionAnalysisHierarchy#processIntrospectedMember
to complete paths of introspected members, instead of overriding#contributeMember
directly. This change removed a lot of internal methods from the interfaceIAnalysisHierarchy
. - PIVOT-5003 Introduce
AAnalysisHierarchyV2
. This class will replaceAAnalysisHierarchy
in ActivePivot 6.1. - PIVOT-5003 Correct typo by renaming
IAnalysisHierarchyInfo#getUseIntrospection
intoIAnalysisHierarchyInfo#getUseIntrospection
. - PIVOT-5032 Upgrade
org.apache.hadoop
to 3.3.1. - PIVOT-5021 Upgrade Spring Boot version to 2.5.4 and update dependencies.
- PIVOT-5024 Use
ChunkedIntegerArray
instead ofIntegerArray
inAMultiVersionUniqueRecordIndex
so that we can now put more than 300M records per partition. - PIVOT-5058 Implements the correct
equals()
andhashcode()
forStoreDescription
andVectorFieldDescription
. - PIVOT-5123 Partitioning modulo 1 now throws an exception.
- PIVOT-5129 Renamed
AnalysigAggregationProcedureFactory
toAnalysisAggregationProcedureFactory
. - PIVOT-5168 Bump Spring version to 2.6
- PIVOT-5168 Bump Jackson version to 2.13.0
- PIVOT-5168 Bump Apache Arrow version to 6.0.1
- PIVOT-5168 Bump Apache CXF version to 3.4.5
Removed
- APS-12776 Remove
IEpochManager.mark(...)
andIEpochManager.unmark(...)
. One should call them on the underlyingIEpochHistory
. - PIVOT-4244 Remove the XML descriptions of the cubes.
- PIVOT-4747
ISelectionField
was removed. UseSelectionField
instead. - PIVOT-4748
StreamUtil.negate
has been removed; we usePredicate.not
instead - PIVOT-4776
IMultiMap
was removed. Use aMap
whose values are collections instead. - PIVOT-4986 Release more resources after dropping a partition (dictionaries of indexes that are not used by a reference can now be released every time).
- PIVOT-5011 Remove the provided strings from exception messages in
RestContentServer
andJsonQueryService
. - PIVOT-5029
IAggregatedMeasureAware
was removed. - PIVOT-5084 Removed an invalid warning that was triggered when building Copper Hierarchies backed by a store. Said warning was stating these hierarchies shouldn't be factless.
- PIVOT-5094
IMultiVersionStorePartition#drop(IDroppedPartitionInformation)
does not take any argument anymore. - PIVOT-5117 Remove
TransactionWrapper
.
Fixed
- PIVOT-4487 Fix Copper post-processor bug where if the only underlying measures of a calculated measure were null or constant measures, the post-processor wouldn't be evaluated at all.
- PIVOT-4735 Fix continuous handler registration issue. The issue happened in distribution when some underlying measures have not yet appeared thus causing a duplication in handler key of the parent measure.
- PIVOT-4780 Fix unnecessary query plan logging when
queryPlanSummary
is enabled. - PIVOT-4800 Enhance query plan timing reports. This particularly fixes two man issues:
JITPrimitiveAggregatesRetrieval
Partitioned retrievals for non-existing partitions used to display the largest execution time of their siblings. Now, only partitioned retrievals that performed some work at least and whose result is empty will be reported in the timing section, along with the other retrievals with result- In some cases, for a group of related retrievals (siblings), timing were not printed if one of the sibling has no result. This edge case has been fixed
- PIVOT-4811 Fix
ClassCastException
inSingleValueFunction
when working with arrays. - PIVOT-4814 Fix an issue where exporting the memory statistic of a chunk containing non-direct vector blocks would throw a NPE.
- PIVOT-4816 Fix an issue occurring when merging two
AggregateStore
s. This caused the merged result to includenull
values instead of the correct value for some measures. - PIVOT-4818 Fix an issue leading to incorrect transitions between
IntegerArrayVersion
. - PIVOT-4819 Fix
UnsupportedOperationException
inActivePivotStatistics
. The issue happened when computing transaction fact statistics on an incompatible type of transaction. - PIVOT-4862 Fix an export error in the
IMemoryStatistic
generation for TombStone classes. - PIVOT-4866 Fix
NullPointerException
inFilterTranslator
. The issue could happen when translating a wildcard, e.g.null
, to the data cube. - PIVOT-4881 Fix an issue where the resizing of
IntegerArray
andLongArray
generated non-transient chunks regardless of itsisTransient
parameter. - PIVOT-4901 Prevent aggregates computation (i.e. GetAggregatesQuery) when mdx
CELL PROPERTIES
is equal toCELL_ORDINAL
. - PIVOT-4923 Fix hiding subtotals issue that could not hide all the measures.
- PIVOT-4935 Fix NPE when accessing an uninitialized datastore via MBean.
- PIVOT-4944 Fix the cleaning method of the prefetch scope location when executing more than one aggregation procedure in a single
PrimitiveAnalysisAggregationRetrieval
. - PIVOT-4946 Fix an issue with Copper
storeLookup
that could fail to find its underlying measures, in the case where these measures were previously re-optimized. - PIVOT-4949 Fix thread safety for
SharedDelegateDictionary#mapFromCell
. - PIVOT-4951 Fix an edge case issue with
IChunkInteger
sparse compression. - PIVOT-4956 Fix
GenericVectorBinding#applyAs..
operations for non-commutative operators. - PIVOT-4958 Fix the update of the ContentServer entries for branches containing characters that are illegal for FileSystem-specific paths.
- PIVOT-4964 Fix an
ArrayIndexOutOfBoundException
happening when compressing chunks with freed rows asFrequentValueChunk
. - PIVOT-4972 Fix branches not visible sometimes with continuous queries. The
ISecurityDetails
was not always propagated when registering queries. - PIVOT-4979 Fix missing fields issue in the analysis aggregation procedures when a cube filter is done on a Copper join.
- PIVOT-4992 Fix dependency issue in content-server-spring.
- PIVOT-4998 Reduce time spent at startup optimizing Copper measures
- PIVOT-5010 Properly set the level type of a slicing Copper hierarchy
- PIVOT-5016 Fix
ArrayIndexOutOfBoundsException
in theSetExpander
of the Mdx engine. - PIVOT-5038 Fixed errors in usage counting caused by decrementing the usage too many times.
- PIVOT-5050 Fix distributed post-processors initialization. This issue happened when chaining at least three post-processors where the bottom one in the chain is not initialized and caused missing dependencies propagation in some cases.
- PIVOT-5062 Fix
NullPointerException
inPointListRetrievalResult
in the case of a polymorphic distributed setup. - PIVOT-5068 Fix
NullPointerException
in the AggregatesResultCopier used by postprocessors. - PIVOT-5074 Fix
SubCubeProperties#removeRestriction(dimension, hierarchy)
. The issue happened in distributed polymorphic setup where the same dimension from two data nodes includes different hierarchies and one of the dimension hierarchies cardinality is greater than 1. - PIVOT-5078 The Parquet Source correctly parses boolean vectors.
- PIVOT-5097 Fix
NullPointerException
when registering aGetAggregatesQuery
build using a location Collections not supportingnull
elements. - PIVOT-5115 Fix
IllegalArgumentException
during query plan logging whenexecutionTimingPrint
is enabled . - PIVOT-5119 Fix rollback process of store transactions.
- PIVOT-5128 Fix a memory leak that could happen upon query failure/cancellation if an
AggregatesCache
was used. - PIVOT-5137 Fix
LevelIdentifier.fromDescription
andHierarchyIdentifier.fromDescription
. They did not handle correctly the escaping of@
. - PIVOT-5151 Fix
QfsStrings.escape
with$
as escape symbol. - PIVOT-5156 Fixed missing query plan retrievals. The issue happened when underlying retrievals with different partitioning were merged. Note that this does not affect query results nor the query plan export feature, but only the query plan printing.
- PIVOT-5162 Fix potential NPE in
PrimitiveAnalysisAggregationRetrieval
when joining results.