Off-Heap Memory Monitoring

There exist good general-purpose tools that analyze the on-heap memory consumption of Java applications. However, Atoti Server stores most of its data off-heap, and as such these tools often miss a lot of information regarding memory ownership, which makes it hard to draw relevant conclusions.

Off-Heap Memory Events

Atoti tracks its off-heap memory allocations, and exposes this information through JFR events. However, these events are disabled by default. To enable, it is necessary to:

start your application with the option -Dactiveviam.mmap.tracking=true
start a JFR with the appropriate options, see JFR events The associated events are under the namespace atoti.server.memory.

Check the JFR documentation for more details.

Memory Analysis Report

A complete off-heap memory report of your Atoti application can be produced using an IMemoryAnalysisService. These memory reports list all off-heap memory used by Atoti, with detailed ownership information (e.g. which store allocates which type of memory, which dictionaries, indexes, …). This service can be instantiated and used programmatically, or it can be exposed by setting up a JMX bean. These reports can then be analyzed using the Memory Analysis Cube project (MAC). It is a specialized tool enabling in-depth analysis of the off-heap memory consumption of Atoti applications.

For more detail on the analysis process, please refer to the documentation of the MAC repository.

Export Memory Reports Programmatically

The basic implementation of IMemoryAnalysisService is MemoryAnalysisService. It can be instantiated through its constructor, passing the Datastore and ActivePivotManager we want to export.

final IMemoryAnalysisService service =
    new MemoryAnalysisService(datastore, manager, exportPath);

All reports will be exported in a sub-folder inside the specified exportPath.

// exports the application based on the most recent version
path = service.exportMostRecentVersion("report");
// exports all versions of the application
path = service.exportApplication("report");
// exports the heads of all specified branches
path = service.exportBranches("report", Set.of("master", "branch1", "branch3"));
// exports the application for all specified epoch ids
path = service.exportVersions("report", new long[] {1L, 3L, 12L});

The first string argument of these methods is the name of the subdirectory the report should be stored in. Each method returns the Path of the exported report.

Export Memory Reports through an MBean

The IMemoryAnalysisService can be created and exposed through a JMX MBean, using the following snippet:

@Bean
public JMXEnabler jmxMemoryMonitoringServiceEnabler() {
  return new JMXEnabler(
      new MemoryAnalysisService(
          datastore, manager, Paths.get(System.getProperty("java.io.tmpdir"))));
}

In the above code, we created a new service using the application Datastore andActivePivotManager. We also specified the directory of the export folder as the OS temp directory. All generated reports will be created inside that directory as sub-folders. The implementation MemoryAnalysisService is MBean friendly and will automatically name itself as MemoryAnalysisService.

Generating the Report

To export a new report, connect to your application using your favorite MBean browser. In this guide, we will be using Java Mission Control. Alternatives are JConsole or JVisualVM - which are proprietary software of Oracle - or the open-sourced VisualVM. In the MBean folder com.activeviam, you will see the MBean MemoryAnalysisService.

The attribute JmxDumpFolder corresponds to the path to the service’s export directory, in case you are unaware of the application configuration. Within the operation screen, you have the ability to export a report for the entire application - see the following screenshot - or for a selected range of epochs. Exporting a range allows exporting only the memory required for a given epoch or series of epochs. Exporting the whole application helps detect how much memory is truly retained (very handy for spotting objects leaking memory).

Full Export

When exporting the whole application, you are asked to provide a name for the export. This name is used to create a sub-folder inside the application export directory. The result of the operation call tells us exactly where the report is located. For our example, we want to generate a report in the folder export-1, that will be located under /tmp/export-1. In case we already have a folder export-1, the application will add a timestamp at the end of the specified folder name.

Version Export

Doing an export of the application for a range of epochs is very similar to doing the full application export. In addition to the export folder, the list of epochs to export needs to be specified. This list is a comma-separated list of ranges. A range can be one of the following:

a single value like 1, that will export Epoch 1,
a range like 1-3, that will export Epochs 1, 2 and 3

For example, the epoch list 1,4-6,9 represents Epochs 1, 4, 5, 6 and 9. In our example, we export Epochs 1, 2 and 3 into export-2.

To get a list of the available epochs, various MBeans exist. For example, you can go to the Epoch Manager and look at the epoch usage.

Documentation Index

​Off-Heap Memory Monitoring

​Off-Heap Memory Events

​Memory Analysis Report

​Export Memory Reports Programmatically

​Export Memory Reports through an MBean

​Generating the Report