5.8.13

Monitoring via the Global Health Dispatcher

Since ActivePivot 5.7, monitoring of an ActivePivot application has become easier through the introduction of events that can be listened to.

Introduction

There is now a global health event dispatcher. Whenever something interesting happens, ActivePivot notifies the health event dispatcher who will then notify all the handlers that are registered to it.

You can register your handler to the health event dispatcher and you will be notified of health events you have asked for via the System property activeviam.health.log. This system property works in a similar fashion to the unified logging API introduced in the JVM in Java 9.

By default, when you start ActivePivot, a handler is already registered to the health event dispatcher. This handler simply logs events generated by the application, like events for executed queries, loaded CSV files, memory and thread consumption.

How does the System property work

Health events have a set of tags - such as "activepivot", "query", "realtime", ...) and a level ("FINE", "INFO", ...). The system property allows you to define which tag and at what level you are interested in. The events that match those tags and level will be forwarded to the listeners. The ones that don't match will be ignored.

The following syntax is used:

-Dactiveviam.health.log=[:option]
   option := [<what>]
             'disable'
   what := <selector>[,...]
   selector := <tag-set>[*][=<level>]
   tag-set := <tag>[+...]
              'all'
   tag := name of tag
   level := OFF
            FINEST
            FINER
            FINE
            INFO
            WARNING
            SEVERE

Default configuration

By default all health events of level INFO or higher are logged.

Usages

Configure logs for an exact tag

-Dactiveviam.health.log=query=FINE matches all health events tagged with query only, at the level FINE or above. Particularly, any event tagged with query and any other tag will be ignored by this configuration.
Default policy is active for other health events.

Configuring logs for many tags

-Dactiveviam.health.log=query*=FINE matches all health events whose tag list contains query and whose level is FINE or above.
Default policy is active for other health events.

-Dactiveviam.health.log=activepivot+query*=FINE activates all health events tagged with at least activepivot and query using INFO or above level.
Default policy is active for other health events.

Configuring logs for all tags

-Dactiveviam.health.log=all=FINE matches all health events regardless of the tag, at the level FINE or above.

Disabling logs for all tags but a selection

-Dactiveviam.health.log=disable,activepivot+query=INFO turns off all health events, except those tagged exactly with activepivot and query using INFO or above level.

-Dactiveviam.health.log=disable,activepivot+query*=FINE,csv+source+parsing=FINE turns off all health events, except those tagged at least with activepivot and query using FINE or above level and events tagged exactly with csv, source and parsing.

List of tags

The following lists all tags currently defined:

  • activepivot
  • csv
  • datastore
  • jvm
  • line
  • memory
  • parsing
  • query
  • realtime
  • source
  • transaction
  • distribution
  • membership

Registering a custom health event handler

By default, a health event handler is a simple lambda receiving a health event. They can be registered like this:

final IHealthEventHandler handler = event -> System.out.println(event.toString());
HealthEventDispatcher.addHandler(handler);

However, such a listener may be overly simple to you. You may want to do something different depending on the type of event. In which case we recommend implementing IGlobalHealthEventHandler. This handler provides specific method depending on the exact type of event, so that you can handle them differently.

For example:

class ExampleHealthEventHandler implements IGlobalHealthEventHandler {
    @Override
    public void onSlowActivePivotQuery(final SlowActivePivotQuery q) {
        sendEmail("A query was too slow " + q.getQuery());
        onUnknownEvent(q);
    }
    @Override
    public void onCsvFileParsingFailure(final CsvFileParsingFailure csv) {
        sendEmail("Failed to parse file " + csv.getFileName());
        onUnknownEvent(csv);
    }
    @Override
    public void onUnknownEvent(final IHealthEvent e) {
        System.out.println("Generic event " + e);
    }
}
final ExampleHealthEventHandler handler = new ExampleHealthEventHandler();
HealthEventDispatcher.addHandler(handler);

IGlobalHealthEventHandler extends various interfaces, each dedicated to a different component of ActivePivot. It is then possible to implement only one of the extended interfaces, covering exactly the perimeter to monitor.

Default Health Event Handler

ActivePivot comes with a default Health Event Handler, registered as a QuartetType in the core product: LoggingGlobalHealthEventHandler. A single of this type is created at the startup of any application and registered as the first handler of the HealthEventDispatcher. This class sends every event to the JUL Logger named com.activeviam.health.monitor.ILoggingHealthEventHandler. This means that in addition to the property -Dactiveviam.health.log, you can control the level of the Java logger itself.

Being a QuartetType, it is possible to replace the default implementation with your own. The following code block implements a dummy handler focusing on ActivePivot query completions.

@QuartetType(description = "A very simple event handler", intf = IHealthEventHandler.class)
public static class ExampleHealthEventHandler implements IActivePivotHealthEventHandler {
    private static final Logger LOGGER = Logger
            .getLogger(ExampleHealthEventHandler.class.getName());
    @Override
    public void onActivePivotQueryDone(final ActivePivotQueryDone q) {
        StringBuilder message = new StringBuilder();
        message.append("|QUERY=").append(q.getQuery());
        message.append("|DURATION=").append(q.getDuration());
        message.append("|USER=").append(q.getUserName());
        LOGGER.log(Level.INFO, message.toString());
    }
    @Override
    public void onUnknownEvent(final IHealthEvent e) {
    }
}

Formatting events for export

The generated Health Events are designed to serve purposes of monitoring. The default handler outputs the content to logs, in a human-readable format. Often, this is not the best for software tools, that tend to prefer a more formatted content, like JSON content.

There are various alternatives to produce software-oriented content from Health Events. The following paragraphs describe these approaches, with their pros and cons.
When describing the different approaches, not to be to general, we will consider that we want to turn the memory and thread report into JSON content.

We recommend approach 2 - extending the default handler - as it allows to keep standard outputs while customizing the relevant events.

Dedicated handler

The easiest approach is to register a special handler, focused on JvmMemoryReport, exporting them as JSON. This handler shall implement IComposerHealthEventHandler as illustrated below to produce its JSON content. Once the class is defined, register an instance of it as a new handler, as described in this section.

class JsonMemoryHandler implements IComposerHealthEventHandler {
    @Override
    public ITagAcceptor getTagAcceptor() {
        // Optional: this avoids this handler process other events while this is not needed.
        final Set<String> memoryTags = new HashSet<>();
        Collections.addAll(memoryTags, "jvm", "memory");
        return new TagAcceptor(
                Collections.singletonMap(memoryTags, Level.ALL),
                Collections.emptyMap(),
                Level.ALL);
    }
    @Override
    public void onJvmMemoryReport(final JvmMemoryReport report) {
        System.out.println(
                "{\"threads\": " + report.getThreadCount()
                        +", \"maxHeap\": " + report.getMaxHeap()
                        + "}");
    }
    @Override
    public void onUnknownEvent(IHealthEvent e) {/* Ignore this */}
}

Pros:

  • A dedicated class is producing JSON content. This is controlled code, owned by users.

Cons:

  • Often, the produced JSON is passed to the logs, to be then captured by tools like Kibana for later analysis. Thus, the memory report is displayed twice, once textually and another time as JSON.

Extending the LoggingGlobalHealthEventHandler

A second approach consists in overriding the default Handler provided by ActivePivot. LoggingGlobalHealthEventHandler implements the method #onJvmMemoryReport, turning a JvmMemoryReport into text. We can extend the class and completely override this method in order to produce a JSON output from the JvmMemoryReport.

Then, the extended class must be registered as a QuartetType, as describe in this section.

Pros:

  • Other logs entries are still available in a human-readable format;
  • The memory and thread reports are available as JSON.

Cons:

  • The memory and thread are in JSON, and may be less readable for users.

Writing a new EventHandler

Another approach would be to write from scratch a new IHealthEventHandler and register it as a QuartetType. Because it is likely that all events - not only the JvmMemoryReport - need to be formatted as JSON, it avoids multiple overrides to a core-product class.

For the purpose of memory report, as the additional handler solution, we shall implement IComposerHealthEventHandler and produce a JSON output.

Pros:

  • Full control of the exported events, in the case where all events shall be exported.

Cons:

  • No more logging about the activity of the application.
← MDX SnippetsQuery Execution Plan →