Unload Topic

Unload topics allow you to remove facts from the datastore based on a specified condition. The condition can be generated from the DlcScope.

Configuration

Yaml Configuration

The unload topics can be defined via yaml configuration. The topics name must match a datastore’s name.

The list of field names per topic name must all be present in the scope when performing the unload operation for that topic. The unload operation will remove all facts from the store where the field equals the value provided in the scope.

The usage of implicit topics can be enabled or disabled.

dlc:
  unload:
    implicit-topics-enabled: true
    topics:
      BaseStore:
        - CobDate
        - ParameterSet

Java Configuration

The unload topics can be configured by defining UnloadTopicDescription beans in your application.

UnloadTopicDescription

The UnloadTopicDescription is used to define the configuration of an unload topic. The following properties are available:

Removal Condition Factory

The removalConditionFactory is a BiFunction that takes an IStoreDescription and a DlcScope and returns an ICondition. This condition is used to filter the facts to be removed from the store.

The UnloadTopicDescription.of(String name, Map<String, Set<String>> fieldNamesPerStore) will create a removalConditionFactory that will, for each store (fieldNamesPerStore keys), create a condition where each field matches the provided field value in the scope. This factory will only work if all field names are provided in the scope.

Example

We can remove all facts that have an AsOfDate of 2021-01-01 from our store by specifying a scope of:

DlcScope.of(
		"AsOfDate", "2021-01-01"
);

We can remove all facts that have an AsOfDate of 2021-01-01 and that have a Currency of usd from our store by specifying a scope of:

DlcScope.of(
		"AsOfDate", "2021-01-01",
        "Currency", "USD"
);

Data Type Considerations

It is important to note the following:

• The removalConditionFactory should be able to ensure the DlcScope keys are properly typed. Since IConditions are strict filters, if the datatype of the scope value does not align with the datatype of the store column, then no data will be unloaded.
• When unloading a DlcScope that was provided via a REST request, the scope values may be of type String.

Custom Unload Topics

You can create custom unload topics by defining UnloadTopicDescription beans in your application. You can customize the removal condition factory or rely on the DlcUnloadOperationHelperUtil to create removal conditions for you.

Here we will create an unload topic that will remove facts from the “StoreName” store where “FiledName_1” and “FieldName_2” field values come from the scope.

@Bean
public UnloadTopicDescription simpleUnloadTopic(){
	return UnloadTopicDescription.of(
			"CustomUnloadTopic",
			Map.of("StoreName", Set.of("FieldName_1", "FieldName_2"))
	);
}

Here we will create an unload topic that will remove all trades that contain a specific trade id:

@Bean
public UnloadTopicDescription fuzzyUnloadTopic(){
	return UnloadTopicDescription.builder("UnloadFuzzyTradeTopic")
			.stores(Set.of("BaseStore"))
			.removalConditionFactory(
					(store, scope) -> {
						// Verify scope contains the expected keys
						assert(scope.containsKey("tradeId"));

						// Scope value may end with a wildcard
						String tradeSearchValue = (String) scope.get("tradeId");
						tradeSearchValue = tradeSearchValue.replace("*", "");

						// Remove all trades that contain the provided search value
						return BaseConditions.containsIgnoreCase(
								FieldPath.of("tradeId"),
								tradeSearchValue
						);
					}
			)
			.build();
}

This topic can be used with a scope of ("tradeId", "Trade_123*") to remove all trades that have a trade id containing the substring "Trade_123".

Implicit Unload Topics

The usage of implicit unload topics can be enabled with the following configuration property:

dlc:
  unload:
    implicit-topics-enabled: true

The DLC will create implicit UnloadTopicDescriptions for each store in your datastore schema. This allows you to unload from any store without having to define an UnloadTopicDescription for each store. The UnloadTopicDescription can also be easily referenced by name.

For each store, an implicit topic will be created with the configuration shown in the following table:

Implicit UnloadTopicDescriptions

Garbage Collection

It may be required to perform a Garbage Collection after an unload operation. Please see the Garbage Collection section for details.