DirectQuery - Atoti Hub

Atoti Market Data provides a way to store various types of market data and retrieve it, efficiently, during a query. These retrievals are executed as Get-by-key or List queries on the individual market data tables. In the course of a complex calculation, such as PnL Explain, there may be many thousands of these queries. With an in-memory Atoti application, the datastore is highly optimized to handle these operations and, even with high volumes, they are executed quickly. However, in an Atoti application using DirectQuery, each of these queries must be executed on the external database. This may mean many thousands of database queries over a network, which will drastically slow any user workflows. The DirectQuery Local Cache is a solution to this problem. Atoti Market Data is designed to work with this cache. Please see the DirectQuery Local Cache Atoti documentation to understand how the cache works, and for details on how to use it in an Atoti application. For this guide, we will assume you have read the documentation, and you are using the cache in your project.

Concepts

Filling the cache

Each of the non-abstract post-processors in Atoti Market Data are configured to use the cache. If these post-processors are part of your query, they will submit a request to fill the cache with market data during the prefetch phase. When they later run Get-by-key or List queries, these queries will use the cached data, improving retrieval speed.

Cache Partitions

A fundamental aspect of the DirectQuery Local Cache design is the use of partitions. A cache does not contain the entire underlying table but instead holds only a subset, i.e., a partition. Partitions are configurable per cache. In the case of Atoti Market Data, a partition is defined as a unique combination of AsOfDate, and MarketDataSet. This means that when Atoti Market Data is used to retrieve a market data value, the cache with be filled with all market data values related to the AsOfDate and MarketDataSet for the request. Any subsequent requests for values with the same AsOfDate and MarketDataSet will be retrieved from the cache, making them significantly faster.

Set up

Cache descriptions

To cache data from a table, you must provide a cache description when the application starts. For each market data store there is a corresponding configuration class that exposes a SingleTableCacheDescription bean. These classes are available in the market-data-config module. These beans should be collected and used when configuring your Atoti application. Each cache description is created with a cache capacity, which determines the maximum number of rows or partitions to keep in the cache. You can configure the cache capacity by exposing a supplier bean for the relevant cache.

@Import(value = {
        // Import the cache configuration for each store you are using.
        CubeMarketDataDirectQueryCacheConfig.class,
        CurveMarketDataDirectQueryCacheConfig.class,
        FxRateMarketDataDirectQueryCacheConfig.class,
        SpotMarketDataDirectQueryCacheConfig.class,
        SurfaceMarketDataDirectQueryCacheConfig.class
})
@Configuration
public class CacheDescriptionConfig {

    // Specify the cache capacity for the cube cache. This is injected to the CubeMarketDataDirectQueryCacheConfig class imported above.
    @Bean
    public CubeMarketDataDirectQueryCacheConfig.CacheCapacitySupplier cubeCacheCapacitySupplier() {
        return () -> CacheCapacity.MaxPartitionsCount.of(6);
    }

    // Specify the cache capacity for the curve cache. This is injected to the CurveMarketDataDirectQueryCacheConfig class imported above.
    @Bean
    public CurveMarketDataDirectQueryCacheConfig.CacheCapacitySupplier curveCacheCapacitySupplier() {
        return () -> CacheCapacity.MaxPartitionsCount.of(4);
    }

    // Specify the cache capacity for the FX rate cache. This is injected to the FxRateMarketDataDirectQueryCacheConfig class imported above.
    @Bean
    public FxRateMarketDataDirectQueryCacheConfig.CacheCapacitySupplier fxRateCacheCapacitySupplier() {
        return () -> CacheCapacity.MaxPartitionsCount.of(4);
    }

    // Specify the cache capacity for the spot cache. This is injected to the SpotMarketDataDirectQueryCacheConfig class imported above.
    @Bean
    public SpotMarketDataDirectQueryCacheConfig.CacheCapacitySupplier spotCacheCapacitySupplier() {
        return () -> CacheCapacity.MaxPartitionsCount.of(2);
    }

    // Specify the cache capacity for the surface cache. This is injected to the SurfaceMarketDataDirectQueryCacheConfig class imported above.
    @Bean
    public SurfaceMarketDataDirectQueryCacheConfig.CacheCapacitySupplier surfaceCacheCapacitySupplier() {
        return () -> CacheCapacity.MaxPartitionsCount.of(10);
    }
}

DatabaseCacheManager injection

Each of the non-abstract post-processors in Atoti Market Data is configured to use the cache but, to do so, they must be injected with the IDatabaseCacheManager. There is a configuration class in market-data-config that will inject the IDatabaseCacheManager into all relevant post-processors. You can import this class in your project, and you must also expose IDatabaseCacheManager as a bean.

@Import(value ={
        // This configuration class will inject the cache manager to relevant post-processors.
        DatabaseCacheManagerInjectionConfig.class
})
public class PostProcessorCacheConfig {

    // Expose the cache manager as a bean. This is required by the DatabaseCacheManagerInjectionConfig class.
    @Bean
    public IDatabaseCacheManager databaseCacheManager(IDirectQueryDatabase database) {
        return database.getCacheManager();
    }
}

AllMarketDataDirectQueryConfig

The market-data-config includes an AllMarketDataDirectQueryConfig class that will import the cache descriptions for all stores, and the post-processor injection configuration.

@Import(value ={
        // Contains the cache descriptions for all stores and the injection configuration.
        AllMarketDataDirectQueryConfig.class
})
public class AtotiMarketDataCacheConfig {

    // Expose the cache manager as a bean.
    @Bean
    public IDatabaseCacheManager databaseCacheManager(IDirectQueryDatabase database) {
        return database.getCacheManager();
    }

    // Specify the cache capacity for the cube cache. This is injected to the CubeMarketDataDirectQueryCacheConfig class imported above.
    @Bean
    public CubeMarketDataDirectQueryCacheConfig.CacheCapacitySupplier cubeCacheCapacitySupplier() {
        return () -> CacheCapacity.MaxPartitionsCount.of(6);
    }

    // Specify the cache capacity for the curve cache. This is injected to the CurveMarketDataDirectQueryCacheConfig class imported above.
    @Bean
    public CurveMarketDataDirectQueryCacheConfig.CacheCapacitySupplier curveCacheCapacitySupplier() {
        return () -> CacheCapacity.MaxPartitionsCount.of(4);
    }

    // Specify the cache capacity for the FX rate cache. This is injected to the FxRateMarketDataDirectQueryCacheConfig class imported above.
    @Bean
    public FxRateMarketDataDirectQueryCacheConfig.CacheCapacitySupplier fxRateCacheCapacitySupplier() {
        return () -> CacheCapacity.MaxPartitionsCount.of(4);
    }

    // Specify the cache capacity for the spot cache. This is injected to the SpotMarketDataDirectQueryCacheConfig class imported above.
    @Bean
    public SpotMarketDataDirectQueryCacheConfig.CacheCapacitySupplier spotCacheCapacitySupplier() {
        return () -> CacheCapacity.MaxPartitionsCount.of(2);
    }

    // Specify the cache capacity for the surface cache. This is injected to the SurfaceMarketDataDirectQueryCacheConfig class imported above.
    @Bean
    public SurfaceMarketDataDirectQueryCacheConfig.CacheCapacitySupplier surfaceCacheCapacitySupplier() {
        return () -> CacheCapacity.MaxPartitionsCount.of(10);
    }
}

Advanced customization

If you are using the default Atoti Market Data stores and post-processors, simply follow the Set up steps to start caching market data. If you have made customizations or are not using all the components of Atoti Market Data you may need to make further changes.

Cache partition customizations

A cache partition is defined as a combination of AsOfDate and MarketDataSet for all Atoti Market Data stores. We believe this provides the best level of granularity for most projects. If, however, you require a different partitioning, you must define your own cache definition for the table. Be sure not to also import the provided cache config class.

@Configuration
public class CubeMarketDataDirectQueryCacheConfig {

    // Provide a custom cache description for the CubeMarketData store.
    @Bean
    public SingleTableCacheDescription cubeStoreCacheDescription() {
        return SingleTableCacheDescription.builder()
                .tableName(MarketDataConstants.CUBE_MARKET_DATA_STORE)
                .cachePartitioningFields(List.of("AsOfDate", "Custom")) // Insert custom partitioning fields here.
                .capacity(CacheCapacity.MaxPartitionsCount.of(6))
                .build();
    }
}

The post-processors in Atoti Market Data are designed to work with the default cache descriptions and the partitioning they define. They extract the AsOfDate and MarketDataSet from the location and use this to feed the cache. If you change the partitioning, the post-processors will need to be updated. This is done by extending the post-processor and overriding the getConverter method.

public class CustomCubeMarketDataPostProcessor extends CubeMarketDataPostProcessor {

    public CustomCubeMarketDataPostProcessor(String name, IPostProcessorCreationContext creationContext) {
        super(name, creationContext);
    }

    // Define your new converter, based on the new partitioning requirements of the cache.
    @Override
    public ILocationToCachePartitionConverter getConverter(Properties properties) {
        return new BestEffortLocationToCachePartitionConverter(List.of(
                new LevelIdentifier("Date", "Dates", "AsOfDate"),
                new LevelIdentifier("Custom", "Custom", "Custom")));
    }
}

Store customizations

The cache descriptions are configured based on the default structure of the underlying store. Most importantly, they require that the store has an AsOfDate, and a MarketDataSet field as these are used for the cache partitions. If you have customized the store and these fields are no longer present, you must define your own cache definition for the table and update relevant post-processors. This follows the same steps as to modify the cache partitioning.

New stores and post-processors

If you write new post-processors, you will need to ensure these are set up to work with the cache. If the new post-processor is retrieving market data from a custom store, you will need to provide your own cache description for the store.

@Configuration
public class CustomDirectQueryCacheConfig {

    @Bean
    public SingleTableCacheDescription customStoreCacheDescription() {
        return SingleTableCacheDescription.builder()
                .tableName("storeName")
                .cachePartitioningFields(List.of(...))
                .capacity(CacheCapacity.MaxPartitionsCount.of(6))
                .build();
    }
}

Your custom post-processors are responsible for filling the cache as described in the Atoti documentation. We offer two classes to help with this. ADirectQueryCachingMarketDataPostProcessor is an abstract class that configures a DatabaseCachePrefetcher for you. It uses getMarketDataRetrievalContainer().retriever().getTableName() as the name of the cache, and it assumes the cache partitioning uses AsOfDate and MarketDataSet, pulling these levels from the first two entries in the AMarketDataPostProcessor.requiredLevels array.

@AtotiExtendedPluginValue(intf = IPostProcessor.class, key = CustomMarketDataPostProcessor.PLUGIN_KEY)
public class CustomMarketDataPostProcessor extends ADirectQueryCachingMarketDataPostProcessor {

    public static final String PLUGIN_KEY = "CustomMarketDataPostProcessor";

    protected CustomMarketDataPostProcessor(String name, IPostProcessorCreationContext creationContext) {
    super(name, creationContext);
    }

    @Override
    protected IContextualMarketDataRetriever getMarketDataRetriever(ILocation location) {
        return ...;
    }

    @Override
    public String getType() {
        return PLUGIN_KEY;
    }
}

If your post-processor does not meet the requirements of ADirectQueryCachingMarketDataPostProcessor, for instance, you have a different cache partitioning, you can instead implement the IDirectQueryCachingPostProcessor interface. By implementing IDirectQueryCachingPostProcessor, your post-processor will be injected with the IDatabaseCache (provided you import the injection configuration class) and there is a helper method to add the DatabaseCachePrefetcher. However, you must specify the cache name and the converter.

@AtotiExtendedPluginValue(intf = IPostProcessor.class, key = CustomMarketDataPostProcessor.PLUGIN_KEY)
public class CustomMarketDataPostProcessor extends ADefaultMarketDataPostProcessor implements IDirectQueryCachingPostProcessor {

    public static final String PLUGIN_KEY = "CustomMarketDataPostProcessor";

    // DatabaseCacheManager with getters and setters for injection.
    @Setter @Getter protected IDatabaseCacheManager databaseCacheManager;

    protected CustomMarketDataPostProcessor(String name, IPostProcessorCreationContext creationContext) {
        super(name, creationContext);
    }

    // Initialize prefetchers with the cache prefetcher, utilizing the addDatabaseCachePrefetcher helper method
    @Override
    public List<IPrefetcher<?>> initializePrefetchers(Properties properties) {
        var superPrefetchers = super.initializePrefetchers(properties);
        return addDatabaseCachePrefetcher(superPrefetchers, properties, getActivePivot());
    }

    // Declare the cache name.
    @Override
    public String getCacheName(Properties properties) {
        return "CustomCache";
    }

    // Declare the cache converter.
    @Override
    public ILocationToCachePartitionConverter getConverter(Properties properties) {
        return new BestEffortLocationToCachePartitionConverter(...);
    }

    @Override
    protected IContextualMarketDataRetriever getMarketDataRetriever(ILocation location) {
        return ...;
    }

    @Override
    public String getType() {
        return PLUGIN_KEY;
    }
}

​Concepts

​Filling the cache

​Cache Partitions

​Set up

​Cache descriptions

​DatabaseCacheManager injection

​AllMarketDataDirectQueryConfig

​Advanced customization

​Cache partition customizations

​Store customizations

​New stores and post-processors