Application Performance Monitoring

ActivePivot APM (Application Performance Monitoring) is a solution for monitoring the healthiness and performance of ActivePivot instances. It provides several features easing the support work, and reducing the burden of maintaining and troubleshooting ActivePivot.

ActivePivot APM provides the following main features:

• Query monitoring: Monitor in-progress, successful or failed MDX queries, their associated execution logs, and the historical performances
• Datastore monitoring: Display the trend of datastore sizes over time; Monitor the datastore transaction times; View successful and failed CSV loading, and their associated execution logs.
• Netty monitoring: Monitor the size of data transferred between nodes, in terms of single query execution, all queries executed by a user, or all activities, etc.
• Monitoring activities by user: Monitor activities (e.g. query execution) on a user level.
• JVM monitoring: Overall JVM status, including CPU, heap and off-heap memory, threads, GC, etc.

APM is built with a number of open-source tools. Below is a high-level illustration of an end-to-end monitored ActivePivot relying on APM library, additional monitoring tools and their interaction.

Configuration

Adding APM in an ActivePivot project

The APM library relies on Extended Plugins or Types overriding to extend core component with monitoring capabilities. In order to load monitoring classes, grab the APM jar and add it as a dependency of your project

<dependency>
  <groupId>com.activeviam.apm</groupId>
  <artifactId>apm</artifactId>
  <version>version of activepivot</version>
</dependency>

List of Extended Plugins

Please find the list of extended plugins provided into the APM library

Class name	Plugin Key	Core class	Description
MonitoredActivePivotQueryExecutor	BASE	ActivePivotQueryExecutor	Replacing the default ActivePivotQueryExecutor with adding ability to monitor query execution
MonitoredMDXStream	MDX	MdxStream	MdxStream with monitoring ability. Also append trace id to the error message to be displayed on Atoti UI
LogNettyMessenger	NETTY	NettyMessenger	Override to add more logging when a query is sent and when we receive the results
MonitoredSynchronousAggregatesContinuousQueryEngine	SYNC	SynchronousAggregatesContinuousQueryEngine	Overrides with extra logs on event handling and tracing capability covering all the subsequence activities

Monitored Spring Configuration

APM library relies on Spring BeanPostProcessor to automatically customize and add monitoring capabilities within ActivePivot configuration classes.
Additionally, the library offers additional configuration classes detailed in the following table:

Spring Config	Description
MonitoredDataLoadingConfig	Add the configuration of the IMessageHandler as well a REST service to check on the file loading status. Check more in the "Data Loading" chapter
MonitoringJmxConfig	Exposes some MBeans related to the previously mentioned monitored services.
QueryPerformanceEvaluatorConfig	Enable query performance evaluation and slow query detection
ExtraLoggingConfig	Add the configuration bringing additional logging (e.g. environment properties)
ConsulServiceConfig	(Optional) Service discovery configuration using Consul server

In case there is a need to wait for the APM stack to be started before triggering some action, one can use the following Spring annotation on the required bean:

@DependsOn(value = TracingConfig.APM_TRACING)

Logging

APM relies on SLF4J with a Logback binding for the logging part.

ActivePivot IHealthEventHandler monitoring

The ActivePivot core code provides some monitoring features through health events. APM defines a specific QuartetType for the IHealthEventHandler class which overrides all the underlying health event handlers to make sure they are printed properly to SLF4J. Additionally, ActivePivot core forces the logs generated by the ILoggingHealthEventHandler implementations to be under the same logger name (com.activeviam.health.monitor.ILoggingHealthEventHandler). We do not think the best approach as it could be necessary, in some occasion, to split the logs into different files. This is why the library contains the following overrides:

Event Handler Interface	Description
IActivePivotHealthEventHandler	Custom implementation MonitoredLoggingActivePivotHealthEventHandler which logs the messages into the logger com.activeviam.health.monitor.IActivePivotHealthEventHandler
IComposerHealthEventHandler	Custom implementation MonitoredLoggingComposerHealthEventHandler which logs the messages into the logger com.activeviam.health.monitor.IComposerHealthEventHandler
ICsvSourceHealthEventHandler	Custom implementation MonitoredLoggingCsvHealthEventHandler which logs the messages into the logger com.activeviam.health.monitor.impl.ICsvSourceHealthEventHandler
IDatastoreHealthEventHandler	Custom implementation MonitoredLoggingDatastoreHealthEventHandler which logs the messages into the logger com.activeviam.health.monitor.IDatastoreHealthEventHandler
IHealthEventHandler	New QuartetType MonitoredLoggingGlobalHealthEventHandler overriding the core LoggingGlobalHealthEventHandler class. It makes sure we are using the previously mentioned event handlers

Configuration

Logback supports custom conversion specifiers allowing to display specific information properly within the logs. APM defines three custom converter detailed below:

Converter	Description
com.activeviam.apm.logging.impl.LogUserConverter	Allow to display the current user for the thread. The user is either taken from the security layer or from the health event
com.activeviam.apm.logging.impl.LogThreadConverter	Allow to display the current thread. This is useful for the health event. This converter can override the default Logback one
com.activeviam.apm.logging.impl.LogInstanceConverter	Required only if ELK setup and JSON logs are used. This converter provides the node instance name specified in property node.instance.name so the source node of a log can be identified in a centralized logging system like ELK

Then, Logback configuration example can make use of these converters as follows:

<?xml version="1.0" encoding="UTF-8"?>
<!-- Example LOGBACK Configuration File http://logback.qos.ch/manual/configuration.html -->
<configuration>
  <shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook"/>
  <jmxConfigurator/>
  <!-- Exposing the conversion rules -->
  <conversionRule conversionWord="thread"
          converterClass="com.activeviam.apm.logging.impl.LogThreadConverter"/>
  <conversionRule conversionWord="user"
          converterClass="com.activeviam.apm.logging.impl.LogUserConverter"/>

  <appender name="MAIN" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${custom.log.dir}/${project.artifactId}.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
      <fileNamePattern>${custom.log.dir}/${project.artifactId}_%d{yyyy-MM-dd}.log.gz
      </fileNamePattern>
      <maxHistory>30</maxHistory>
      <cleanHistoryOnStart>true</cleanHistoryOnStart>
    </rollingPolicy>

    <encoder>
      <!-- Using the converter in the appender pattern layout -->
      <pattern>%d{yyyy-MM-dd HH:mm:ss.SSS} [Thread: %thread; User: %user] %-5level %logger{35} -
        %msg%n
      </pattern>
    </encoder>
  </appender>
  [...]
  <root level="INFO">
    <appender-ref ref="MAIN"/>
  </root>
  <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
    <resetJUL>true</resetJUL>
  </contextListener>
</configuration>

If ELK setup is used, the library is ready for producing JSON logs for feeding Logstash and Elasticsearch. This can be done through an additional JSON appender as follows:

<?xml version="1.0" encoding="UTF-8"?>
<!-- Example LOGBACK Configuration File http://logback.qos.ch/manual/configuration.html -->
<configuration>
  <shutdownHook class="ch.qos.logback.core.hook.DelayingShutdownHook"/>
  <jmxConfigurator/>
  <conversionRule conversionWord="thread"
          converterClass="com.activeviam.apm.logging.impl.LogThreadConverter"/>
  <conversionRule conversionWord="user"
          converterClass="com.activeviam.apm.logging.impl.LogUserConverter"/>
  <conversionRule conversionWord="instance"
          converterClass="com.activeviam.apm.logging.impl.LogInstanceConverter"/>

  <appender name="JSON" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <file>${custom.json.log.dir}/activepivot.log</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
      <fileNamePattern>${custom.json.log.dir}/activepivot_%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
      <maxFileSize>1GB</maxFileSize>
      <maxHistory>30</maxHistory>
      <cleanHistoryOnStart>true</cleanHistoryOnStart>
    </rollingPolicy>

    <encoder class="net.logstash.logback.encoder.LogstashEncoder">
      <!--
          write the timestamp value as a numeric unix timestamp(number of milliseconds since unix epoch)
          Check the link https://github.com/logstash/logstash-logback-encoder/tree/logstash-logback-encoder-5.2#customizing-timestamp
       -->
      <timestampPattern>[UNIX_TIMESTAMP_AS_NUMBER]</timestampPattern>

      <!-- Exclude user name and thread name from MDC and standard fields to avoid duplication -->
      <excludeMdcKeyName>user_name</excludeMdcKeyName>
      <excludeMdcKeyName>thread_name</excludeMdcKeyName>
      <fieldNames>
        <timestamp>log_timestamp</timestamp>
        <thread>[ignore]</thread>
      </fieldNames>

      <!-- Append customized fields  -->
      <provider class="net.logstash.logback.composite.loggingevent.LoggingEventPatternJsonProvider">
        <pattern>
          {
          "user_name": "%user",
          "thread_name": "%thread",
          "instance_name": "%instance"
          }
        </pattern>
      </provider>
    </encoder>
  </appender>
  [...]
  <root level="INFO">
    <appender-ref ref="JSON"/>
  </root>
  <contextListener class="ch.qos.logback.classic.jul.LevelChangePropagator">
    <resetJUL>true</resetJUL>
  </contextListener>
</configuration>

Loggers Categorization

APM loggers could be categorized as follows:

Category	Logger(s)
Feeding	com.activeviam.apm.source com.activeviam.health.monitor.impl.ICsvSourceHealthEventHandler com.activeviam.health.monitor.IDatastoreHealthEventHandler
Feed loading	com.activeviam.apm.loaded.feed Used if you choose the feed.loading.message.handler.type option 'LOG'. In this case you more likely prefer the output to be made in a separate file
Queries	com.activeviam.apm.pivot com.activeviam.apm.tracing com.activeviam.apm.web com.activeviam.health.monitor.IActivePivotHealthEventHandler
Distribution	com.activeviam.apm.messenger
Health Agent	com.activeviam.health.monitor.IComposerHealthEventHandler

Data loading

The MonitoredDataLoadingConfig class provides the bean messageHandler() which creates the implementation of the IMessageHandler. In your source Spring config class you could do the following:

...
@Autowired
private MonitoredDataLoadingConfig monitoredDataLoadingConfig;

@Bean
protected IMessageHandler<IFileInfo<Path>> messageHandler() {
  return monitoredDataLoadingConfig.messageHandler();
}

@Bean(destroyMethod = "close")
public CustomCSVSource dataCSVSource() {
  final CSVSource csvSource = new CSVSource(DATA_TAG);
  ...
  return csvSource;
}

protected List<IStoreMessageChannel<IFileInfo<Path>, ILineReader>> dataCSVChannels(
  final CSVMessageChannelFactory<Path> dataCSVChannelFactory,
  final IMessageHandler<IFileInfo<Path>> messageHandler) {
  
    final List<IStoreMessageChannel<IFileInfo<Path>, ILineReader>> channels = new ArrayList<>();
        ITuplePublisher<IFileInfo<Path>> tuplePublisher = ...;
        IStoreMessageChannel<IFileInfo<Path>, ILineReader> channel = dataCSVChannelFactory.createChannel(topicName, storeName, tuplePublisher);
        channel.withPublisher(tuplePublisher);
        channel.withMessageHandler(messageHandler());
        channels.add(channel);
}

Depending on the properties, the bean allows to:

• log start/end processing of some input data, as well as the errors, through the logger com.activeviam.apm.source
• log the rejections of the input data through the logger com.activeviam.apm.source.rejection
• specify the property feed.loading.message.handler.type to select the type of the message handler: if set to DEFAULT, simply logs detailed information about loading file if set to LOG, logs the feed loading in an external file (CSV style) upon completion using the logger com.activeviam.apm.loaded.feed ** if set to WEBHOOK, call a webhook (external service) upon completion of the file loading

The import of the Spring Config class MonitoredDataLoadingConfig implicitly brings the Spring Config MonitoredDataLoadingCacheConfig and MonitoredDataLoadingRestServicesConfig.

• MonitoredDataLoadingCacheConfig creates a cache allowing to keep all the files loaded as well as their status. It is used later on by the REST service to return the loaded status of a file.
• MonitoredDataLoadingRestServicesConfig exposes a REST service allowing the get the loading status of a file. The service is accessible alongside the ActivePivot core RESTful services, through the endpoint /cube/apm/dataloading.

Properties exposed

List of the properties exposed as well as their default value

Property	Description	Default value
activeviam.apm.config.hide.properties.pattern	Specifies the pattern used to hide sensitive properties when logged	*(password\|key).*
activeviam.apm.enable.all.apmanager.statistics	Enable the MBeans statistics for the ActivePivot manager	true
activeviam.apm.enable.xmla.servlet.logging	Enable the logging of the XMLA Servlet	true
activeviam.apm.enable.json.queries.service.logging	Enable the logging of the JSON Queries Service	true
activeviam.apm.enable.json.queries.service.detailed.logging	Enable the detailed logging of the JSON Queries Service	false
activeviam.apm.enable.json.queries.service.log.memory.stats	Enable the logging of the memory stats for the JSON Queries Service	false
activeviam.apm.enable.queries.service.logging	Enable the logging of the Queries Service	true
activeviam.apm.enable.queries.service.detailed.logging	Enable the detailed logging of the Queries Service	false
activeviam.apm.enable.queries.service.log.memory.stats	Enable the logging of the memory stats for the Queries Service	false
activeviam.apm.enable.streaming.service.logging	Enable the logging of the Streaming Service	true
activeviam.apm.enable.streaming.service.detailed.logging	Enable the detailed logging of the Streaming Service	false
activeviam.apm.enable.streaming.service.log.memory.stats	Enable the logging of the memory stats for the Streaming Service	false
activeviam.apm.enable.web.socket.logging	Enable the logging of Web Socket	true
activeviam.apm.enable.web.socket.detailed.logging	Enable the detailed logging of the Web Socket	false
activeviam.apm.enable.web.socket.log.memory.stats	Enable the logging of the memory stats for the Web Socket	false
activeviam.apm.feed.loading.cache.loaded.files.num.entries	Property defining the number of entries kept in the feed loading cache	1,000,000
activeviam.apm.feed.loading.message.handler.type	Property defining the feed loading message handler type	DEFAULT
activeviam.apm.feed.loading.message.handler.webhook.url	Property defining the webhook URL for the feed loading message handler
activeviam.apm.feed.loading.message.handler.webhook.num.retries	Property defining the number of retries for the webhook feed loading message handler	Integer.MAX_VALUE
activeviam.apm.feed.loading.message.handler.webhook.time.to.wait	Property defining the time to wait (ms) between each retry for the webhook feed loading message handler	3000 (3s)
activeviam.apm.jmx.thread.status.cache.timeout	Thread status cache value timeout in milliseconds. Thread status cache value is refreshed automatically when timeout.	60000 (60s)
activeviam.apm.node.instance.name	Unique name of the current node. It must be provided if logging/monitoring stack (e.g. ELK) is used.	activepivot
activeviam.apm.consul.host	Host or IP of the consul service
activeviam.apm.consul.port	Port of the consul service	8500
activeviam.apm.activepivot.service.name	Service name of the instance. This is for service grouping and filtering in consul	activepivot
activeviam.apm.node.host	Host or IP of the node instance for registering to consul	The first non loop back host ip found in network interfaces
activeviam.apm.node.healthcheck.port	Port which consul should call for healthcheck. This should be port used for jmx explorer, e.g. -javaagent:jmx_prometheus_javaagent-0.3.1.jar=8081	8081
activeviam.apm.node.healthcheck.path	Path which consul should call for healthcheck	metrics
activeviam.apm.node.healthcheck.interval	Interval between health check attempts	5m
activeviam.apm.node.healthcheck.timeout	Timeout for a health check attempt	30s
activeviam.apm.node.healthcheck.deregister.time	Consul will de-register the instance from the registry if it loses contact for more that a specific time	30m
activeviam.apm.query.performance.evaluation.min.sample.size	The minimum amount of past performance required as samples for evaluation	10
activeviam.apm.query.performance.evaluation.max.sample.size	The maximum amount of past performance taken into account for evaluation	30
activeviam.apm.query.performance.evaluation.coefficient	The coefficient for adjusting the threshold	1
activeviam.apm.query.performance.evaluation.elasticsearch.urls	URL(s) to Elasticsearch instance(s) that provides the historical performance. e.g. http://localhost:9200	null
activeviam.apm.query.performance.evaluation.elasticsearch.truststore.path	Path to the trust store that used to have encrypted communication with Elasticsearch	null
activeviam.apm.query.performance.evaluation.elasticsearch.truststore.type	Type of the trust store to be loaded for encrypted communication with Elasticsearch	null
activeviam.apm.query.performance.evaluation.elasticsearch.truststore.password	Password to open the trust store	null
activeviam.apm.query.performance.evaluation.elasticsearch.username	Username for login Elasticsearch if authentication is required	null
activeviam.apm.query.performance.evaluation.elasticsearch.password	Password for login Elasticsearch if authentication is required	null