Data Connectors Overview

This is the documentation for ActiveViam’s Data Connectors library, with a section for users and a section for developers for both the Data Load Controller and Data Extract Engine libraries.

What is Data Connectors?

Data Connectors is a scalable, standard service to handle data loading/extraction within an ActivePivot application. This abstracts specific data loading/extraction implementations away from the client. Data Connectors sits on top of an ActivePivot application. Data Connectors contains target-specific modules that each yield a JAR. Developers can pick and choose to include the artifacts they need in their POM file.

Requirements

ActivePivot Version

The Data Connectors JARs rely on local ActivePivot repositories. We achieve this by marking all ActivePivot dependencies as <scope>provided</scope> within Data Connectors POM files. This allows the ActivePivot dependencies to be inferred from the local file system and to flexibly use different minor versions. ActivePivot upgrades are easier as there is no need to rebuild/upgrade the Data-Connectors tool. The same is expected for the javax.servlet:javax.servlet-api archetype if you are using the monitoring for XMLA.

Configuration Classes

An example of including these configuration classes can be seen as follows:

    @Configuration
    @Import({
        // Required Configurations for Tracing and APM
        APMTracingConfig.class,
        APMActivePivotServicesConfig.class
    })
    public class ApplicationConfig {
    }

Registry Setup

• APM must be defined with high precedence in ActivePivot’s Registry. This is because some classes are Extended Plugins or Types overriding the core code. It is important to make sure they are picked first by the Registry. Make sure you add the package com.activeviam.apm to be picked first. The below example can be added to your application’s configuration.

/** Before anything else we statically initialize the ActiveViam Registry. */
static {
    Registry.setContributionProvider(
        new ClasspathContributionProvider(
            "com.activeviam.apm",  // First has highest priority
            "com.qfs",
            "com.quartetfs",
            "com.activeviam"
        )
    );
}

Initial Data Load

• If you are executing any DLC tasks via Spring Beans during initialization of your application, you must ensure that the APM_TRACING bean is created first either by the @DependsOn or by Autowiring the APM_TRACING bean. Please see code sample below as an example:

    @Bean
    @DependsOn(value = {APM_TRACING})
    public Void initialLoad() {
        // Load data here...
    }

Module Structure

Here we can see the Data Connectors module structure with all target modules (in blue).

Currently, Data Connectors supports the following targets for the DLC and DEE components. Targets are included in the following maven modules:

Users can import the target module required for either data Loading or Extraction by including the corresponding target’s module within their POM. Then the user will have access to both the Data Load Controller and Data Extract Engine components.

Here is an example of including Data Connectors for RabbitMq:

<dependency>
    <groupId>com.activeviam.io</groupId>
    <artifactId>data-connectors-rabbitmq</artifactId>
    <version>2.0.0-AP5.10</version>
</dependency>

REST Services

Services URL

Both the DLC and DEE components are found at the following REST URLs:

/services/connectors/rest/dlc/<REST-version>/
/services/connectors/rest/dee/<REST-version>/

REST Service Version

Data Connectors REST endpoints contain a version number. This version number helps remote users ensure they are using the correct REST contract. The REST version is appended to the end of the base REST endpoint for both DLC and DEE components. More on each component can be found here: DLC REST Services and DEE REST Services.

Below we can see the REST version matrix for the different componenets of Data Connectors: