Performance benchmarks

​

Benchmark methodology

The following results were gathered on the FRTB 4.0.0 release. This section explains how the benchmark results were gathered.

​

Benchmark results overview

The benchmark results are displayed in tables with two axes; Cores and Dataset Size. This indicates what size machine and what size dataset the tests were run on. Each cell of these tables represents an individual performance test. Each Approach (ex: IMA-DRC) was run independently.

​

Performance test procedure

The performance test was run many times on multiple Azure machines to generate the Benchmark results. Each iteration of the performance test did the following:

• Create a new Azure VM with a specific number of CPUs
• Load the approach’s dataset onto the Azure VM
• Load the FRTB 4.0 Accelerator docker image onto the Azure VM
• Start the FRTB 4.0 Accelerator
• Run Queries

For each run of the benchmark, the following three properties were configured with one value from each list. The performance test was run for all permutations of inputs resulting in a total of 36 runs (3 * 3 * 4)

• Approach (IMA-DRC, IMA-ES, SA-DRC, SA-SBM)
• Number of CPUs: (16, 32, 64)
• Dataset Size: (Small, Medium, Large)

​

Creating the Azure instance

Each iteration of the performance test was run on a new, clean Azure VM running Linux with a specific number of CPUs. For these tests the Azure EdsV5 Virtual Machines were used. For more information, see EdsV5 VMs on Azure’s details page. The following machines were allocated with the stated docker images:

Azure VM allocations:

Docker container CPU limit:

Large VMs are allocated and the CPUs the Docker Image can use are then limited. This is done because the Azure instances for smaller VMs do not have enough memory for these tests. The tests need enough memory overhead so the JVM is not forced to perform many Garbage Collections.

​

Datasets

Each approach has its own three datasets, one for each size (Small, Medium, Large). This allows control over how much data is injected into the Cube. Having one dataset per approach provides the following benefits:

• Ensure that only the necessary data for that approach and the approach’s queries is loaded
• Tailor the datasets to maximize the number of locations the queries have to aggregate
• Generate more accurate memory statistics.

In each approach, queries are run to gather the number of facts from the Cube - these queries are from the Data Sanity Check dashboard, which can be used to compare a cube to the testing cube. Here is an example of the files that were loaded for the IMA-DRC datasets:

• /configuration/*
• /date/General/*
• /date/DRC/*
• /historical/fx-data/*
• /historical/drc-summary/*

DRC vectors

The DRC vectors used have a length of 400,000 and a density of 6% (94% sparse vectors).

​

Running queries

Once FRTB is up and running, one warmup query was run, and then the average of running another 12 queries was taken. This warmup query gets the JVM fully ready because when the JVM has just started, the first query will be significantly slower and does not accurately represent query times. A Garbage Collection was forced in between each Query execution to ensure that queries had the most memory to run with - this also lowers the chance that a Garbage Collection would occur during a running query and alter performance. All aggregates caches were also disabled, so query results are not cached. Otherwise, all query times would have been nearly zero as the query results would be fetched from the cache rather than re-computing the entire query.

​

Benchmark results by version

Results can be found on the following page:

• Benchmark Results 6.0
• Benchmark Results 5.3
• Benchmark Results 5.2
• Benchmark Results 5.1