Skip to main content
Custom unload topics are defined with a removalConditionFactory, while implicit unload topics are created automatically from the request scope.

​
Configuration

​
YAML configuration

The unload topics can be defined via YAML configuration. The topics name must match a datastore table’s name. The list of field names per topic name must all be present in the scope when performing the unload operation for that topic. The unload operation will remove all facts from the store where the field equals the value provided in the scope. The DLC also has a feature to create implicit unload topics. This feature can be enabled or disabled. 
dlc:
  unload:
    implicit-topics-enabled: true
    topics:
      BaseStore:
        - CobDate
        - ParameterSet

​
Java configuration

The unload topics can be configured by defining UnloadTopicDescription beans in the application.

​
UnloadTopicDescription

The UnloadTopicDescription is used to define the configuration of an unload topic. The following properties are available:
ParameterRequiredTypeDefaultDescription
nameYStringName of the topic.
storesYSet<String>Empty SetSet of stores this topics will unload from.
removalConditionFactoryYBiFunction<IStoreDescription, DlcScope, ICondition>Factory that creates a False condition.Factory that creates the condition to filter the facts to be removed.

​
Removal condition factory

The removalConditionFactory is a BiFunction that takes an IStoreDescription and a DlcScope and returns an ICondition. This condition is used to filter the facts to be removed from the store. The UnloadTopicDescription.of(String name, Map<String, Set<String>> fieldNamesPerStore) will create a removalConditionFactory that will, for each store (fieldNamesPerStore keys), create a condition where each field matches the provided field value in the scope. This factory will only work if all field names are provided in the scope.

​
Example

All facts that have an AsOfDate of 2021-01-01 can be removed from the store by specifying a scope of: 
DlcScope.of(
		"AsOfDate", "2021-01-01"
);
All facts that have an AsOfDate of 2021-01-01 and that have a Currency of usd can be removed from the store by specifying a scope of: 
DlcScope.of(
		"AsOfDate", "2021-01-01",
        "Currency", "USD"
);

​
Data type considerations

It is important to note the following:
  • The removalConditionFactory should be able to ensure the DlcScope keys are properly typed. Since IConditions are strict filters, if the datatype of the scope value does not align with the datatype of the store column, then no data will be unloaded.
  • When unloading a DlcScope that was provided via a REST request, the scope values may be of type String.

​
Custom unload topics

Custom unload topics can be created by defining UnloadTopicDescription beans in the application. The removal condition factory can be customized, or the DlcUnloadOperationHelperUtil can be used to create removal conditions. The following example creates an unload topic that removes facts from the “StoreName” store where “FiledName_1” and “FieldName_2” field values come from the scope. 
@Bean
public UnloadTopicDescription simpleUnloadTopic(){
	return UnloadTopicDescription.of(
			"CustomUnloadTopic",
			Map.of("StoreName", Set.of("FieldName_1", "FieldName_2"))
	);
}
The following example creates an unload topic that removes all trades that contain a specific trade id: 
@Bean
public UnloadTopicDescription fuzzyUnloadTopic(){
	return UnloadTopicDescription.builder("UnloadFuzzyTradeTopic")
			.stores(Set.of("BaseStore"))
			.removalConditionFactory(
					(store, scope) -> {
						// Verify scope contains the expected keys
						assert(scope.containsKey("tradeId"));

						// Scope value may end with a wildcard
						String tradeSearchValue = (String) scope.get("tradeId");
						tradeSearchValue = tradeSearchValue.replace("*", "");

						// Remove all trades that contain the provided search value
						return BaseConditions.containsIgnoreCase(
								FieldPath.of("tradeId"),
								tradeSearchValue
						);
					}
			)
			.build();
}
This topic can be used with a scope of ("tradeId", "Trade_123*") to remove all trades that have a trade id containing the substring "Trade_123".

​
Implicit unload topics

Implicit unload topics allow data to be unloaded from a datastore table without explicitly defining an unload topic in the configuration.
Please be aware of the following when using implicit topics:
  • The usage of implicit topics exposes the application to requests that may be overly flexible, allowing users to unload data from any table for any field/value pair.
  • The request must know the details of the datastore schema.
The usage of implicit unload topics can be enabled via the implicit-topics-enabled property 
dlc:
  unload:
    implicit-topics-enabled: true
When using implicit unload topics:
  • The topic name must match the name of an existing datastore table.
  • The removal condition will be resolved from the scope of the incoming request.
Example
{
  "operation": "UNLOAD",
  "topics": ["TradeTable"],
  "scope": {
    "tradeID": "IRS_GBP_CPI 35a38b80"
  }
}
The DLC will create an implicit UnloadTopicDescription at request time when a topic is requested that both matches the name of a datastore table and no UnloadTopicDescription exists in the DLC configuration.

​
Implicit UnloadTopicDescriptions

ParameterValue
nameName of the store.
Example: "TradeBaseStore".
storesAs above, the name of the store in question.
Note: The value is just the one store.
Example: List ["TradeBaseStore"]
removalConditionFactoryDefault IUnloadTopicDescription.SCOPE_TO_REMOVAL_CONDITION

​
Garbage collection

It may be required to perform a Garbage collection after an unload operation. Please see the Garbage collection section for details.