> ## Documentation Index > Fetch the complete documentation index at: https://docs.activeviam.com/llms.txt > Use this file to discover all available pages before exploring further. # Post-Processors ## Introduction Atoti introduced a while ago a key feature called Post-Processors, which provides the ability to evaluate business logic for each request to aggregated data. The Post-Processors were called such because they are computed after the basic aggregated measures. However, they must not be confused with the Transform phase of ETL (which happens when loading data in the datastore). Post-processors are in fact applied on-the-fly, only on relevant data, during a query. Post-processors are defined on the server side with Java code. The Post-Processors are the "old way" to add your own business logic to the computations done in Atoti. Since then, we introduced the [Copper API](../copper/copper_intro), which is much easier to use. Post-processors are available for backward compatibility and for complex use cases where the Copper API does not yet offer a satisfying alternative. Post-Processors are extremely flexible and can use the following types of data to compute their results: * Pre-aggregated data * Custom aggregations * Other post-processor results * External resources (e.g. share prices, FOREX) Most of the time the Post-Processor retrieves underlying measures at some location, performs additional computation with the (optional) help of some services deployed in the application, and writes its own output to the result cell set. Post-Processors advantages include the following: * The end user does not need to know if a measure is post-processed or just pre-aggregated data; everything is just a Measure. * Post-Processors are written in Java. This allows you to use many external systems and provides more flexibility than MDX's calculated measures. * Post-Processors offer infinite possibilities for computations. Atoti's post-processors can easily and accurately compute the PnL of a trade, using real-time data streams (e.g. share prices and FOREX). * The results of any Post-Processor can be computed across any dimension. * Post-Processors are more flexible than traditional database-stored procedures. * The results can depend on the query context (e.g. on the user). ## Post-Processed Measures Below are some common business cases that use post-processors: * Simple arithmetic operations (e.g. calculating a ratio between levels) * Currency conversions (e.g. using external data to convert a trade's currency to a common currency) * Presentation effects (e.g. hiding meaningless values from users) * Displays of market data (e.g. showing the market data used in a currency conversion) * Running totals (e.g. showing how cash flows accumulate over a given time period) * Computing non-aggregating values (e.g. VaR) * Real-time market data updates (e.g. FX and P\&L) ## IPostProcessor `IPostProcessor` is the base interface for all PostProcessors. The `compute` method is responsible for implementing the actual business logic: ```java theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}} /** * Main evaluation method of the post processor The post processor computes the aggregates within * the scope of a range location, with the help of an aggregate retriever it can use to retrieve * dependent aggregates. * *

The post processor can retrieve any kind of aggregate from the retriever, included * aggregates that have been previously computed by another post processor. The query engine is in * charge of arranging post processor chains accordingly. * *

The result of this post-processor evaluation should be written to the result using the given * retriever. * * @param location The location to evaluate this post-processor for * @param retriever The retriever that can provide underlying aggregated values * @throws ActiveViamException if an exception occurs during this computation */ void compute(ILocation location, IAdvancedAggregatesRetriever retriever) throws ActiveViamException; ``` Note that the `IPostProcessor` is created only once per version of the cube and then shared between queries, so you must be careful not to maintain state in instance variables. Once created, the post-processor is initialized with its properties via the `init` method: if several measures configured in the cube use the same post-processor plugin key with different properties, one instance will be created for each measure. ## Provided Post-Processors Source code of some post-processors is provided as an example for you to write your own. The different classes are packaged in a zip called `activepivot-ext` and hosted on Artifactory. Note that, most of the time, you won't need to use those post-processors since their logic is available through the [Copper API](../copper/copper_intro), which is much simpler to use. All concrete implementations for the built-in post-processors have a Copper counterpart. You can, however, extend the abstract classes for additional flexibility to handle (and potentially optimize) specific use cases. The following diagram illustrates the available post-processors: Post-Processors tree ### AAdvancedPostProcessor This is the abstract base class for all post-processors. When implementing a new post-processor, we recommend extending this class (or one of its existing subclasses) as it contains boilerplate code to retrieve underlying measures and to declare the context dependencies and continuous query handlers. Here is an [example implementation](filtering#trendpostprocessor-example) for AAdvancedPostProcessor. ### ABasicPostProcessor This is an abstract base class for simple post-processors that retrieves its underlying aggregates at the same location for which it computes its result. The `ABasicPostProcessor` allows a calculation to occur for each cell for which the user requests aggregates. Unlike the `ADynamicAggregationPostProcessor`, no extra aggregation occurs on the calculated values. This post-processor allows calculations and cosmetic changes to occur only on the cells requested. Anything that is written as a function of underlying aggregates and external data can be modeled as an `ABasicPostProcessor`. VaR is a common example, where the percentile needs to be retrieved from a vector for each cell in a user's view. This can be combined with user preferences (context values), to allow the user to choose the percentile or the method of calculating VaR (simple percentile, expected shortfall, exponentially weighted expected shortfall), which will be used during the calculation. `ABasicPostProcessor` implements `IPartitionedPostProcessor` and sets its partitioning levels based on the wildcards in the location where it is computed, provided its `partitionedOnRangeLevels` property allows it (or the default value set by `ActiveViamProperty.DEFAULT_PARTITIONING_ON_RANGE_LEVELS_PROPERTY` if the specific property is not set for the post-processor). `ABasicPostProcessor` also implements `IEvaluator`, so any concrete implementation can be used by other post-processors that use an `IEvaluator`, such as the [DynamicAggregationPostProcessor](#dynamicaggregationpostprocessor). `ABasicPostProcessor` is the easiest abstract post-processor implementation to understand. However, it can cover a multitude of business use cases. The business logic should be implemented in the `evaluate` method: ```java theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}} /** * Maps the given aggregated measures, which are found at the given location, and writes it into * the given result cell. * * @param location the current location * @param aggregatedMeasures the measures, aggregated at the given point location * @param resultCell the cell into which the result of the evaluation should be written */ void evaluate(ILocation location, IRecordReader aggregatedMeasures, IWritableCell resultCell); ``` ### BasicPostProcessor This post-processor is a concrete implementation of the `ABasicPostProcessor`, which delegates its evaluation to an `IEvaluator`. Its plugin key is "BASIC". When configuring the `BasicPostProcessor`, you must set its `evaluator` property to the plugin key of the `IEvaluator` you want to use. The use of an `IEvaluator` does not change what has to be implemented (the `evaluate` method signature is the same in the `BasicPostProcessor` and in the `IEvaluator`). However, it allows for easier code reuse, since the implementation of the `IEvaluator` can also be used for a `DynamicAggregationPostProcessor`. > `BasicPostProcessor` usage is deprecated as it is strongly recommended you use > [Copper API](../copper/copper_measures#simple-operations) for defining > simple measures instead. Using the Copper API allows for less > verbose, more maintainable code, and enables a functional definition > of the measures. ### AFilteringPostProcessor This abstract post-processor filters locations based on conditions on levels, by editing the query filter during prefetch. The evaluation is based on an `IEvaluator`. If there is no evaluator defined, but the post-processor has only one underlying measure, it is assumed the post-processor filters that measures's value. ### FilteringPostProcessor This is a concrete implementation of `AFilteringPostProcessor`, which filters on specific members for given levels. The plugin key for this post-processor is "FILTERING". When configuring the `FilteringPostProcessor`, you must define the `leafLevels` property (the levels where you define your filters) and the `membersToFilterFor` property (the member values for each leaf level). > The Copper counterpart to this post-processor is a > [filtered measure](../copper/copper_measures#measure-filtering). ### ABaseDynamicAggregationPostProcessor This is an abstract post-processor with support for high performance dynamic aggregation. The concept of dynamic aggregation is explained in the [Atoti Concepts in a Nutshell article](../concepts/concepts_in_a_nutshell#creating-custom-measures). `ABaseDynamicAggregationPostProcessor` is a dynamic aggregation post-processor that retrieves its underlying aggregated values at the requested leaf levels and reduces them to the requested scope. It can be parameterized by defining the `leafLevels` property (the level to which it should go to for prefetching data and applying a transformation procedure) and the `aggregationFunction` property (the plugin key of the reduction aggregation function). The transformation procedure applied at the leaf level should be defined by implementing the `createLeafEvaluationProcedure` method: ```java theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}} /** * Create the {@link ITransformProcedure transformation} procedure that will be executed on the * underlying {@link IIterableAggregatesRetrievalResult result at the leaf level} and that will * return the actual leaves. * *

Sub-classes can override this method to return their own procedure implementation. * * @param result The result containing the underlying values * @param measureIds The IDs of the underlying measures in the result * @return The procedure used to transform the result */ protected abstract ITransformProcedure createLeafEvaluationProcedure( final IIterableAggregatesRetrievalResult result, final int[] measureIds); ``` `ABaseDynamicAggregationPostProcessor` implements `IPartitionedPostProcessor`. The partitioning levels are set as the leaf levels or their parents. ### ADynamicAggregationPostProcessor This class provides a leaf evaluation procedure at one less level of abstraction than the `ABaseDynamicAggregationPostProcessor` so that you only need to implement `evaluateLeaf`: ```java theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}} /** * Performs the evaluation of the post processor on a leaf (as defined in the properties), and * writes this intermediate result into the given result cell. * * @param leafLocation The leaf location * @param underlyingValues Values of the underlying measures for the leafLocation * @param resultCell the result cell into which the intermediate aggregate is written */ protected abstract void evaluateLeaf( ILocation leafLocation, IRecordReader underlyingValues, IWritableCell resultCell); ``` ### DynamicAggregationPostProcessor This is a concrete implementation of a dynamic aggregation post-processor that delegates its leaf evaluation to an `IEvaluator`. The plugin key for this post-processor is "DYNAMIC". > The Copper counterpart to this post-processor is a > [dynamic aggregation measure](../copper/copper_measures#dynamic-aggregationleaf-operation). ### LeafCountPostProcessor This is a dynamic aggregation post-processor that counts the number of leaves that contribute to the aggregate for each queried aggregate. Configure the depth of the leaves in the same way as for dynamic aggregation post-processors. This post-processor is a concrete implementation, with the plugin key "LEAF\_COUNT". `LeafCountPostProcessor` is used to compile the DistinctCount from MDX queries. Altering its behavior might make the results of the DistinctCount formula differ from the MDX specification. ### ALocationShiftPostProcessor This post-processor allows data to be read from one location and written to the location specified in the query. This is done by overriding the `shiftLocation` method: ```java theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}} /** * Shift the evaluation location into the read location. * *

This method is used to shift the input evaluated location to retrieve the result set from * which the returned values will be read. * *

This method can return {@code null} to skip the evaluation of the post processor on the * specified location. * * @param evaluationLocation A location at which the post processor is evaluated * @return the read location at which the post processor will actually read the underlying * aggregates from which it will derive the result associated with the evaluation location */ protected abstract ILocation shiftLocation(ILocation evaluationLocation); ``` This allows post-processors to be chained together for performing calculations using measures for different locations. It does this by attaching those measures to locations that other post-processors can use. The most common example is calculating the difference between two days, where the implementation of the `ALocationShiftPostProcessor` shifts the location to retrieve data for 'Yesterday'. > The Copper counterpart to this post-processor is a > [location shifting measure](../copper/copper_measures#shift-measures). ### ATimeLinePostProcessor This post-processor builds its query evaluation by applying logic on the different values of a measure along a time hierarchy. This post-processor is configured by providing several properties: * `timeHierarchyName`: the time hierarchy upon which we are operating, * `streamMeasureName`: the measure that is considered as a "stream" along the time hierarchy, * `positionType`: how we depend on the stream measure along the time hierarchy. In fact, the post-processor can depend on the current value or not: * previous values, including the current one (positionType = "CURRENT\_STREAM") * previous values, excluding the current one (positionType = "PREVIOUS\_STREAM") The available methods allow you to choose the locations along the time hierarchy you are interested in, along with how to aggregate the measure on those locations: ```java theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}} /** Returns the value before considering the first cell of the timeline. */ protected abstract OutputT getInitialPosition(); /** * Returns the aggregation of current cell with previous position. * * @param previousPosition Value on the previous element of the timeline. It may be mutated * @param currentValue value of the underlying measure on the current cell of the timeline */ protected abstract OutputT aggregateNextEntry(OutputT previousPosition, Object currentValue); @Override public abstract Object[] computeRequiredLocations( ILocation location, Collection rangeLocationArrays, Collection locations); ``` For continuous queries, this post-processor uses `TimeLineHandler` to calculate its impact (the only core implementation of handler that depends on its measure, since the handler needs to know which hierarchy is the time hierarchy that the measure is interested in). > The Copper counterpart to this post-processor is an > [analytic function](../copper/copper_measures#analytic-functions). ### AStream2PositionPostProcessor This post-processor is an abstract implementation of the `ATimeLinePostProcessor` that specializes in computing the cumulative aggregation of a measure along the time hierarchy. Simple applications include calculating the cumulative sum of a measure over time, or computing the minimal value reached so far for a measure. > The Copper counterpart to this post-processor is an > [aggregate analytic function](../copper/copper_measures#aggregate-analytic-function). ### PreviousValuePostProcessor This post-processor is a concrete implementation of the `ATimeLinePostProcessor`, which, given a cell and a time hierarchy, returns the `ATimeLinePostProcessor` underlying the time hierarchy. Its plugin key is "PreviousValue". > The Copper counterpart to this post-processor is the `lag` > [navigation analytic function](../copper/copper_measures#navigation-analytic-function). ### FormulaPostProcessor This post-processor applies an `IFormula` defined in its initialization properties. It is a concrete implementation, with the plugin key "FORMULA". The formula is written in [Reverse Polish Notation](https://en.wikipedia.org/wiki/Reverse_Polish_notation), with operators specified after their operands. The operands are extended plugins of the `IOperand` interface. It means that you can extend the set of operands by adding new `IOperand` extended plugin values. The built-in operands are: | Operand | Description | Example | | --------------------- | --------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | double | Creates a specific double value | `double[1.5]` | | float | Creates a specific float value | `float [1]` | | int | Creates a specific integer value | `int[1]` | | long | Creates a specific long value | `long[1]` | | string | Creates a specific string value | `string[example]` | | null | Always evaluated to `null` | `null[]` | | aggregatedValue | Fetches a measure on the current location | `aggregatedValue[pnl.SUM]` | | parentAggregatedValue | Fetches a measure on the parent node of the current location | `parentAggregatedValue[pnl.SUM::Underlyings@Underlyings]`
Evaluates the measure `pnl.SUM` for the parent location on hierarchy `Underlyings` of the evaluated location | | childAggregatedValues | Fetches a measure for all children of the current location | `childAggregatedValues[delta.SUM::Underlyings::true]`
Evaluates `pnl.SUM` on the children relative to hierarchy `Underlyings` of the evaluated location, and returns a sorted list of the results. | | levelValue | The name of the member at the given level, or `null` | `levelValue[BookId@Desk@Booking]`
Returns the id of the book of the location being evaluated, or `null` if the location is not deep enough | | dynamicAggregation | Re-aggregates the result from the leaf levels on the given hierarchy using the given aggregation function | `dynamicAggregation[Currency@Underlyings::delta.SUM::AVG]`
Expands the evaluated location on the level `Currency` of the `Underlyings` hierarchy, and aggregates all the delta.SUM results using the AVG aggregation function | The operators are plugins of the `IOperator` interface. This means that the set of operators can be extended by adding new `IOperator` plugin values. The built-in operators are: | Operator | Description | | --------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | + | Sums of all previous elements | | - | Subtracts the last value from the penultimate value | | CONCAT | Concatenates (as in string) all previous elements | | / | Divides the penultimate value by the last value | | \* | Multiplies all previous elements | | SUBVECTOR | Creates a subvector of a vector: operands are the vector, the included from index, the excluded to index | | div | Ternary divide operator: numerator, denominator, and an optional third argument that is the value to return when the numerator is not null, and the denominator is null or 0 | The `FormulaPostProcessor` supports grouping operations through the usage of brackets. You can define a sub-formula inside another formula by setting an entire formula between brackets. Sub-formula evaluation will precede the surrounding formula evaluation. Example: `(int[4], int[5], *), (int[3], int[2], +), -` \<=> (5 × 4) - (3 + 2) \<=> 15 > This post-processor can be replaced by combining various measures and operations through the > [Copper API](../copper/copper_measures). The `FormulaPostProcessor` is evaluated only for the locations where `contributors.Count` is > 0. So, if you use operands that are post-processed measures that produce values even for locations where the contributors count is 0 (such as a `PreviousValuePostProcessor`, for example), you might end up with a null result for your formula even if the operands are not null. To overcome this behavior, there is an alternative core post-processor, called `ArithmeticFormulaPostProcessor`. ### ArithmeticFormulaPostProcessor Like the `FormulaPostProcessor`, the `ArithmeticFormulaPostProcessor` applies an `IFormula` defined in its initialization properties. However, that formula is an `ArithmeticFormula`, e.g. it only allows the `double`, `float`, `int`, `long`, `null` and `aggregateValue` operands, and only allows `+`, `-`, `*`, `/` and `div` operators. This post-processor is a concrete implementation, with the plugin key "ARITHMETIC\_FORMULA". Contrary to the `FormulaPostProcessor`, the `ArithmeticFormulaPostProcessor` is evaluated where the measures defined in the formula as `aggregatedValue` are not `null` (instead of where `contributors.COUNT` is >0). > The Copper counterpart to this post-processor is an > [arithmetic operation](../copper/copper_measures#arithmetic-operations). ### MinimumLevelsPostProcessor This post-processor hides underlying measures when the queried locations are higher than some given levels in their respective hierarchies. In some use cases the aggregation of base data above a given level does not make sense; for instance, summing amounts in different currencies, or summing data from different historical dates. This post-processor is a concrete implementation, with the plugin key "MINIMUM\_LEVELS". > The Copper counterpart to this post-processor is the > [`doNotAggregateAbove` method](../copper/copper_measures#aggregation-levels-hiding). ## Special Post-Processor Interfaces For Evaluation Performances ### IPartitionedPostProcessor The `IPartitionedPostProcessor` allows you to split a partitioned post-processor computation across multiple partitions. Results from each partition are then reduced back into a final result. Partitioned vs. non-partitioned post-processors This allows for better performance over waiting for all underlying measures from all partitions to be available before computing the post-processor's results. Implement the `reduce` method to define how each partition's results are reduced into a final result: ```java theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}} /** * Reduces the partial {@link IAggregatesRetrievalResult results} of the execution of this * post-processor on each partition into a global result and contributes it to the {@link * IPostProcessedRetrievalResultWriter writer}. * * @param location The location on which this post-processor is being executed * @param partialResults The results of the execution of this post-processor on each partition * @param writer The writer to contribute to */ void reduce( ILocation location, List partialResults, IPostProcessedRetrievalResultWriter writer); ``` Then override the `setPartitioningLevels` method (defaulted to doing nothing in parent `IPostProcessor` interface). ```java theme={"languages":{"custom":["/engine/python-sdk/0.9/languages/pycon.tmLanguage.json"]}} @Override default void setPartitioningLevels( final ILocation location, final Collection partitioningLevels) { throw new UnsupportedOperationException( getClass().getSimpleName() + " must define the levels on which they partition."); } ``` ### IDistributedPostProcessor The `IDistributedPostProcessor` is a distributed, partitioned post-processor that may be distributed on each data cube, regardless of its partitioning levels. More information on distributed post-processors can be found in the [Distributed Architecture](../distributed/distributed_post_processors) section of the documentation. ### ILinearPostProcessor The `ILinearPostProcessor` is a linear post-processor that allows results to be aggregated linearly. The result of a post-processor applied on a 'big' aggregate will be the same as when you aggregate the outputs of a post-processor on each element of any partition of the 'big' aggregates. For example, a basic post-processor that multiplies an underlying measure by 2 will give the same results if it is applied on a grand total, or if we sum its results on any partition of that total. Implementing this interface lets the query engine know that it can deduce results for that post-processor from less aggregated results using the given aggregation function (this functionality is called "range sharing").