Monitoring via the Global Health Dispatcher

Since ActivePivot 5.7, monitoring of an ActivePivot application has become easier through the introduction of events that can be listened to.

Introduction

There is now a global health event dispatcher. Whenever something interesting happens, ActivePivot notifies the health event dispatcher which then notifies all the handlers that are registered to it.

You can register your handler to the health event dispatcher and get notifications of health events you have asked for via the System property activeviam.health.log. This system property works in a similar fashion to the unified logging API introduced in the JVM in Java 9.

By default, when you start ActivePivot, a handler is already registered to the health event dispatcher. This handler simply logs events generated by the application, like events for executed queries, loaded CSV files, memory and thread consumption.

How the System property works

Health events have a set of tags - such as "activepivot", "query", "realtime") and a level (for example, "FINE", "INFO"). The system property allows you to define which tag you are interested in and at what level. The events that match those tags and level are forwarded to the listeners. The ones that don't match are ignored.

The following syntax is used:

-Dactiveviam.health.log=[:option]
   option := [<what>]
             'disable'
   what := <selector>[,...]
   selector := <tag-set>[*][=<level>]
   tag-set := <tag>[+...]
              'all'
   tag := name of tag
   level := OFF
            FINEST
            FINER
            FINE
            INFO
            WARNING
            SEVERE

Default configuration

By default all health events of level INFO or higher are logged.

Usages

Configure logs for an exact tag

-Dactiveviam.health.log=query=FINE matches all health events tagged only with query, at the level FINE or above. Specifically, any event tagged with query plus any other tag is ignored by this configuration.
Default policy is active for other health events.

Configuring logs for many tags

-Dactiveviam.health.log=query*=FINE matches all health events whose tag list contains query and level is FINE or above.
Default policy is active for other health events.

-Dactiveviam.health.log=activepivot+query*=FINE activates all health events tagged with at least activepivot and query using level INFO or above.
Default policy is active for other health events.

Configuring logs for all tags

-Dactiveviam.health.log=all=FINE matches all health events regardless of the tag, at the level FINE or above.

Disabling logs for all tags except a selection

-Dactiveviam.health.log=disable,activepivot+query=INFO turns off all health events, except those tagged exactly with activepivot and query using level INFO or above.

-Dactiveviam.health.log=disable,activepivot+query*=FINE,csv+source+parsing=FINE turns off all health events, except those tagged with at least activepivot and query using level FINE or above and events tagged exactly with csv, source and parsing.

List of tags

The following lists all tags currently defined:

  • activepivot
  • csv
  • datastore
  • jvm
  • line
  • memory
  • parsing
  • query
  • realtime
  • source
  • transaction
  • distribution
  • membership

Registering a custom health event handler

By default, a health event handler is a simple lambda receiving a health event. You can register the handler like this:

final IHealthEventHandler handler = event -> System.out.println(event.toString());
IHealthEventDispatcher.addHandler(handler);

However, such a listener may be overly simple for you. If you want to do something more complex depending on different types of event, we recommend implementing IGlobalHealthEventHandler. This handler provides specific methods based on the exact type of event, so that you can handle each one differently.

For example:

class ExampleHealthEventHandler implements IGlobalHealthEventHandler {
    @Override
    public void onSlowActivePivotQuery(final SlowActivePivotQuery q) {
        sendEmail("A query was too slow " + q.getQuery());
        onUnknownEvent(q);
    }
    @Override
    public void onCsvFileParsingFailure(final CsvFileParsingFailure csv) {
        sendEmail("Failed to parse file " + csv.getFileName());
        onUnknownEvent(csv);
    }
    @Override
    public void onUnknownEvent(final IHealthEvent e) {
        System.out.println("Generic event " + e);
    }
}
final ExampleHealthEventHandler handler = new ExampleHealthEventHandler();
IHealthEventDispatcher.addHandler(handler);

IGlobalHealthEventHandler extends various interfaces, each dedicated to a different component of ActivePivot. You can then implement only one of the extended interfaces, covering exactly the perimeter to monitor.

Default Health Event Handler

ActivePivot comes with a default Health Event Handler, registered as a QuartetType in the core product: LoggingGlobalHealthEventHandler. A single instance of this type is created at the startup of any application and registered as the first handler of the HealthEventDispatcher. This class sends every event to the JUL Logger named com.activeviam.health.monitor.ILoggingHealthEventHandler. This means that in addition to the property -Dactiveviam.health.log, you can control the level of the Java logger itself.

As it is a QuartetType, you can replace the default implementation with your own. The following code block implements a dummy handler focusing on ActivePivot query completions.

@QuartetType(description = "A very simple event handler", intf = IHealthEventHandler.class)
public static class ExampleHealthEventHandler implements IActivePivotHealthEventHandler {
    private static final Logger LOGGER = Logger
            .getLogger(ExampleHealthEventHandler.class.getName());
    @Override
    public void onActivePivotQueryDone(final ActivePivotQueryDone q) {
        StringBuilder message = new StringBuilder();
        message.append("|QUERY=").append(q.getQuery());
        message.append("|DURATION=").append(q.getDuration());
        message.append("|USER=").append(q.getUserName());
        LOGGER.log(Level.INFO, message.toString());
    }
    @Override
    public void onUnknownEvent(final IHealthEvent e) {
    }
}

Formatting events for export

The generated Health Events are designed to serve monitoring purposes. The default handler outputs the content to human-readable logs. Often, this is not the best for software tools that tend to prefer something more formatted, like JSON content.

There are various alternatives to produce software-oriented content from Health Events. This section describe these approaches, with their pros and cons based on the premise that we want to turn the memory and thread report into JSON content.

We recommend approach 2 - extending the default handler - as it allows you to keep standard outputs while customizing the relevant events.

Dedicated handler

The easiest approach is to register a special handler, focused on JvmMemoryReport, exporting them as JSON. This handler implements IComposerHealthEventHandler as illustrated below to produce its JSON content. Once the class is defined, register an instance of it as a new handler, as described in this section.

class JsonMemoryHandler implements IComposerHealthEventHandler {
    @Override
    public ITagAcceptor getTagAcceptor() {
        // Optional: this avoids this handler process other events while this is not needed.
        final Set<String> memoryTags = new HashSet<>();
        Collections.addAll(memoryTags, "jvm", "memory");
        return new TagAcceptor(
                Collections.singletonMap(memoryTags, Level.ALL),
                Collections.emptyMap(),
                Level.ALL);
    }
    @Override
    public void onJvmMemoryReport(final JvmMemoryReport report) {
        System.out.println(
                "{\"threads\": " + report.getThreadCount()
                        +", \"maxHeap\": " + report.getMaxHeap()
                        + "}");
    }
    @Override
    public void onUnknownEvent(IHealthEvent e) {/* Ignore this */}
}

Pros:

  • A dedicated class produces JSON content. This is controlled code, owned by users.

Cons:

  • Often, the JSON produced is passed to the logs, to then be captured by tools like Kibana for later analysis. Thus, the memory report is displayed twice, once textually and again as JSON.

Extending the LoggingGlobalHealthEventHandler

A second approach consists in overriding the default Handler provided by ActivePivot. LoggingGlobalHealthEventHandler implements the method #onJvmMemoryReport, turning a JvmMemoryReport into text. We can extend the class and completely override this method to produce a JSON output from the JvmMemoryReport.

Then, we must register the extended class as a QuartetType, as described in this section.

Pros:

  • Other log entries are still available in a human-readable format.
  • The memory and thread reports are available as JSON.

Cons:

  • The memory and thread are in JSON, and may be less readable for users.

Writing a new EventHandler

Another approach would be to write a new IHealthEventHandler from scratch and register it as a QuartetType. Given that it is likely that all events - not only the JvmMemoryReport - need to be formatted as JSON, it avoids multiple overrides to a core-product class.

For the purpose of the memory report, as with the additional handler solution, we implement IComposerHealthEventHandler and produce a JSON output.

Pros:

  • Full control of the exported events, in the case where all events are exported.

Cons:

  • No more logging of application activity.
← OverviewQuery Execution Plan →