Advanced Topics

We give here additional information that is not mandatory in order to use CoPPer in a project but might prove useful when encountering issues.

Column Graph and Measures

The datasets and columns you manipulate via CoPPer's API are all immutable objects. They are here only to represent the calculation the user wants to perform. In the end, each column is either calculated from other columns or a store field, a context value or a constant value column. So all the columns CoPPer works with form a graph, with column operators as vertices and store field, context values and constant columns as leafs of the graph. The edges of the graph are between a column and its operands. In the later debugging section we explain how to see this graph. But the main take from this graph is that a CoPPer calculation is split into multiple columns linked between them via operators. All these columns belong to multiple datasets as well. Each time the user calls a dataset operator it creates a new dataset containing new columns. In the following sections, what we call the first dafirst datasets are the datasets retrieved via context.createDatasetFromFacts() or context.createDatasetFromStore(String), and the last dataset is the one on which the user called .toCellSet(String), or .publish().

In all the examples above, we used as measure names the name of a column in the final dataset. It always worked well, but one can wonder:

  1. Are all columns of the final dataset converted as measures?
  2. What happens if we have multiple columns with the same name?
  3. What about intermediate columns, are they also converted to measures?

For 1., CoPPer analyses all the columns of the last dataset and decide for each of them if they correspond to a calculation that should be used as a measure, or instead as an additional hierarchy or not. This is done to minimize the number of measures CoPPer creates in the cube that would clutter the UIs. If for instance a column represents directly a cube level, or a constant or a context value, we won't convert them to measure.

This automated choice by CoPPer of which column correspond to a measure should cover all basic use cases, but if you created a column that was not converted into a measure by CoPPer you should:

  1. Report us this missing measure conversion (see Reporting an Issue below)
  2. As a workaround until the issue is fixed: aggregate this column after your last calculation via dataset.agg(Columns.customAgg(yourColumnName, CopyFunction.KEY)). The copy aggregation function is a safe choice if your calculation is supposed to work everywhere in the cube.

Error Handling

When you use CoPPer in your project you will use the CopperActivePivotDescriptionPostProcessor as explained under API. The calculations you describe and publish in your lambda will throw runtime exceptions if CoPPer could not convert them. We recommend you don't try to catch them and let your project startup fail when one of these calculations is wrong. They should give a detailed error message helping you fix the issue; and are of different type:

UserErrorException are the exceptions that will be thrown when you describe a calculation that has a mistake in it, the usual suspects are:

  • Using an incorrect column name or store name or store field name
  • Using a plugin key not registered for the given plugin of the registry
  • Assuming the wrong data type for a column, like calling .multiply() on a column holding String values.

InvalidCalculationException also happens when your calculations contains mistakes and you should fix them. The difference between this exception and UserErrorException is that here the mistake is more complex to detect and can only be detected at a later stage. The UserErrorException stack trace will give you the place where you made a mistake, whereas a InvalidCalculationException will contain the location of the offending code in its message.

The usual suspects for this exceptions are: trying to use .partitionBy() or .orderBy() with columns that do not represent a level.

NotSupportedOperationException tells you the API call you made is not supported yet.

CouldNotCastException Is the exception you will probably get the most. This exception means that CoPPer didn't detect any issue with your calculations but still could not convert it to a measure. This can be due to:

  • CoPPer understood what you were trying to do but detected that it doesn't support it yet. If there is a workaround the error message will contain it, otherwise you should report this issue (see Reporting an Issue below).
  • CoPPer couldn't even understand what you were trying to do. In this case it will suggest one or multiple fixes that you should try. If none of them is satisfactory or works you should also report this issue (see below).

Testing

One of the objectives of CoPPer is to be easily testable. This is why it bundles a lot of utilities and APIs to help test your calculations. The main one is the CubeFoundryTester (see "Why Copper"? for the name).

Since CubeFoundryTester and ComponentsRule (see below) are only needed for tests and require test dependencies they are not present in activepivot-copper main maven module. You should import them using the test jar type (with test-jar):

<dependency>
 <groupId>com.activeviam.activepivot</groupId>
 <artifactId>activepivot-copper</artifactId>
 <type>test-jar</type>
 <scope>test</scope>
</dependency>

CubeFoundryTester

This object receives the scaffolding in its constructor and provide a BuildingContext that can be used exactly the same way than with CopperActivePivotDescriptionPostProcessor. It is thus quite easy to test the dataset that you publish in your cube. Let's say for instance that your main measure is called pnlForex. You can create it in a method like this:

public Dataset pnlForex(BuildingContext context) {
    StoreDataset forex = context.createDatasetFromStore("Forex");
 
    return context.createDatasetFromFacts()
        // The rest of the calls is not important here
        // .agg(...)
        // .select(...)
           ;
}

This way you can call this method from your CopperActivePivotDescriptionPostProcessor and from your unit tests. In CopperActivePivotDescriptionPostProcessor you should do

buildingContext -> {
      // Here you can describe your CoPPer measures with the building context.
      pnlForex(context).publish()
 }

And in a single unit test you can do:

/**
 * Tests the values returned by the PP created by the PnL Forex calculation.
 */
@Test
public void testPnlForex() {
    pnlForex(context)
    .toCellSet("SELECT .....")
    .getTester()
    .hasCellCountOf(2)
    .hasOnlyOneCellHavingInItsCoordinates("EUR").containing(10.6)
    .hasOnlyOneCellHavingInItsCoordinates("USD").containing(3.6)
    ;
}

Here context is a test class attribute built by the test using the CubeFoundryTester. It can easily be accessed from it by reading CubeFoundryTester.buildingContext.

ComponentsRule

As explained above, the toCellset method of a dataset works by creating a manager and a cube on top of an existing datastore, perform the query and stop the cube and manager. These are a lot of ActiveViam components that must be created and stopped automatically around the test methods, and it is important to do it nicely otherwise a failing test will very likely break all the tests running after because of some component conflict. This is why we introduced in CoPPer a ComponentsRule. This rule is a jUnit rule that starts and stops cleanly ActivePivot cubes and managers cleanly around the tests, even if they fail.

If we use all these components together we can write the following simple test class:

public class TestBasic {
 
    /** The components rule. */
    @Rule
    public ComponentsRule cr = new ComponentsRule();
    /** The context built by the tester. */
    public BuildingContext context = cr.useForClass(TestBasic::createTester).buildingContext;
 
    /**
     * Creates the tester on a very simple dataset.
     *
     * @return The tester.
     */
    public static CubeFoundryTester createTester() {
        IDatastoreSchemaDescription datastoreDescription = new DatastoreSchemaDescriptionBuilder()
                .withStore(new StoreDescriptionBuilder()
                        .withStoreName("sa")
                        .withField("f", ILiteralType.STRING)
                        .withField("m", ILiteralType.DOUBLE)
                        .withField("l", ILiteralType.LONG)
                        .withVectorField("v", ILiteralType.DOUBLE).ofSize(2)
                        .withoutKey()
                        .build())
                .build();
 
        ISelectionDescription selection = StartBuilding.selection()
                .fromBaseStore("sa")
                .withField("f")
                .withField("m")
                .withField("l")
                .withField("v")
                .build();
 
        IActivePivotInstanceDescription cubeDescription = StartBuilding.cube("C")
                .withDimension("d1")
                    .withHierarchy("h1")
                        .withLevel("f")
                .build();
 
        TransactionsBuilder data = SimpleTransactionsBuilder.start()
                .inStore("sa")
                .add("A", 1, 1L, new double[] {1.0, 2.0})
                .add("B", 2, 2L, new double[] {2.0, 3.0})
                .end();
 
        return new CubeFoundryTester(datastoreDescription, selection, data, cubeDescription);
    }
 
    /**
     * Tests the grand total of measure 1.
     */
    @Test
    public void testMeasure1GrandTotal() {
        myDataset(context)
        .toCellSet("SELECT FROM [C] WHERE [Measures].[myMeasureName]")
        .getTester()
        .hasOnlyOneCell().containing(6.0);
    }
}

Here we see that we call cr.useForClass(TestBasic::createTester), this tells the components rule that the tester passed as argument will be needed for the whole test class and the components it created should be stopped only at the end of the class. There are multiple other methods available on a ComponentsRule and its javadoc should help you choose which one you need if you have a different test use case.

The getTester() call gives you a ICellSetTester that contains a lot of API to test the content of the cellSet, which is very useful when it contains multiple cells, like:

  • hasOnlyOneCellHavingInItsCoordinates() that searches for a cell when you have a coordinate (member) that appears only once a cellSet axis. This is useful for simple queries with one cell per member value for instance.
  • hasOnlyOneCellWith() is a more advanced construct that helps you search for a cell even if you have crossJoins in your query that make member appear multiple time in your axes. After this call you can chain calls to coordinate(), each one give the member value for one level.

If for instance your query crossjoins the levels date and to of your cube, each containing two members, you can test like this:

toCellSet(mdxQuery)
  .getTester()
  .hasCellCountOf(4)
  .hasOnlyOneCellWith()
  .coordinate("to", "A")
    .coordinate("date", date1).containing(7.0)
    .coordinate("date", date2).containing(14.0)
  .coordinate("to", "B")
    .coordinate("date", date1).containing(4.0)
    .coordinate("date", date2).containing(4.0);
← Arithmetic Operations and Edge Cases