Performance Benchmarks
Benchmark Methodology
The following results were gathered on the FRTB 4.0.0 release. Here we explain how we gathered the benchmark results.
Benchmark Results
The benchmark results are displayed in tables with two axes; Cores and Dataset Size. This tells us what size machine and what size dataset the tests were run on. Each cell of these tables represent an individual performance test. Each Approach (ex: IMA-DRC) was run independently.
Performance Test Procedure
The performance test was run many times on multiple Azure machines to generate the Benchmark results. Each iteration of the performance test did the following:
- Create a new Azure VM with a specific number of CPUs
- Load the approach’s dataset onto the Azure VM
- Load the FRTB 4.0 Accelerator docker image onto the Azure VM
- Start the FRTB 4.0 Accelerator
- Run Queries
For each run of the benchmark we configured the following three properties with one value from each list. We ran the performance test for all permutations of inputs resulting in a total of 36 runs (3 * 3 * 4)
- Approach (IMA-DRC, IMA-ES, SA-DRC, SA-SBM)
- Number of CPUs: (16, 32, 64)
- Dataset Size: (Small, Medium, Large)
Creating Azure Instance
Each iteration of the performance test was run on a new, clean Azure VM running Linux with a specific number of CPUs. For these tests we used the Azure EdsV5 Virtual Machines. For more information, see EdsV5 VMs on Azure’s details page.
When allocating the machines we allocated the following machines with the stated docker images:
Azure VM Allocations:
| Number Of CPUs | Small Dataset | Medium Dataset | Large Dataset |
|---|---|---|---|
| 16 | E32ds V5 | E32ds V5 | E64ds V5 |
| 32 | E32ds V5 | E32ds V5 | E64ds V5 |
| 64 | E64ds V5 | E16ds V5 | E64ds V5 |
Docker Container CPU limit:
| Number Of CPUs | Small Dataset | Medium Dataset | Large Dataset |
|---|---|---|---|
| 16 | 16 | 16 | 16 |
| 32 | 32 | 32 | 32 |
| 64 | 64 | 64 | 64 |
The reason we allocate large VMs and then limit the CPUs the Docker Image can use is because the Azure instances for smaller VMs do not have enough memory for our tests. We want to allow our tests to have enough memory overhead so the JVM is not forced to perform many Garbage Collections.
Datasets
Each approach has its own three datasets, one for each size (Small, Medium, Large). This allows us to control how much data is injected into the Cube. Having one dataset per approach provides the following benefits:
- Ensure that we are only loading the necessary data for that approach and the approach’s queries
- Tailor the datasets to maximize the number of locations our queries have to aggregate
- Generate more accurate memory statistics.
In each approach we run queries to gather the number of facts from the Cube - these queries are from the Data Sanity Check bookmark which you can use to compare your cube to our testing cube.
Here is an example of the files that were loaded for the IMA-DRC datasets:
- /configuration/*
- /date/General/*
- /date/DRC/*
- /historical/fx-data/*
- /historical/drc-summary/*
DRC Vectors
The DRC vectors used have a length of 400,000 and a density of 6% (94% sparse vectors).
Running Queries
Once FRTB is up and running, we ran one warmup query, and then took the average of running another 12 queries. This warmup query gets the JVM fully ready because when the JVM has just started, the first query will be significantly slower and does not accurately represent query times.
A Garbage Collection was forced in between each Query execution to ensure that queries had the most memory to run with - this also lowers the chance that a Garbage Collection would occur during a running query and alter performance.
We also disabled all aggregates caches, so we do not cache query results. If we had not, then all query times would have been nearly zero as we would be fetching the query results from the cache rather than re-computing the entire query.
Benchmark Results
Results can be found on the following page: