Parquet loading topic

Configuration properties

Properties for a Parquet Loading Topic are defined by name in the dlc.parquet.topics namespace, and are picked up as configurations for the DLC.

Key	Type	Default	Description
file-pattern	`String`		Path matcher.
columns	`List<String>`	Columns will implicitly come from the store which is being published to.	The order of columns in the file. If a column name matches a datastore column, its parser will be implied. For columns that do not match a datastore column, a parser must be provided in the channel as a Custom Field. Columns without a matching datastore column or parser are skipped.
source-store-mapping	`Map<String, String>`		Ordered map of fields in the record from the remote topic or queue.
channels	`Set<Channel>`	A channel is created using the topic’s name as the name of the target.	Channels.
restrict-to-sources	`Set<String>`		Sources to which the topic is restricted. See source to topic matching.
restrict-from-sources	`Set<String>`		Sources from which the topic is restricted. See source to topic matching.

YAML example

This configuration does not specify a target for the data or the data format.

If the topic is not named the same as the store, the target must be explicitly specified in the channel of the topic.

The target is implicitly the Atoti table named “Trades”.
The format implicitly comes from the columns of the Atoti table.

dlc:
  parquet:
    topics:
      Trades:
        file-pattern: glob:trades*.parquet

Java configuration

ParquetTopicDescription Spring Beans are picked up as configurations for the DLC.

Parameter	Required	Type	Default	Description
name	Y	`String`		Name of the topic.
filePattern		`String`		Path matcher.
columns		`List<String>`	Columns will implicitly come from the store which is being published to.	The order of columns in the file. If a column name matches a datastore column, its parser will be implied. For columns that do not match a datastore column, a parser must be provided in the channel as a Custom Field. Columns without a matching datastore column or parser are skipped.
sourceStoreMapping		`Map<String, String>`		Ordered map of fields in the record from the remote topic or queue.
channels		`Set<ChannelDescription>`	A channel is created using the topic’s name as the name of the target	Channel Descriptions which belong to the topic.
restrictToSources		`Set<String>`		Sources to which the topic is restricted. See source to topic matching.
restrictFromSources		`Set<String>`		Sources from which the topic is restricted. See source to topic matching.

Java example

This configuration does not specify a target for the data or the data format.

If the topic is not named the same as the store, the target must be explicitly specified in the channel of the topic.

The target is implicitly the Atoti table named “Trades”.
The format implicitly comes from the columns of the Atoti table.

@Bean
public ParquetTopicDescription tradesTopic() {
    return ParquetTopicDescription.builder("Trades", "glob:trades*.parquet")
            .build();
}

Operations

Request API

Configuration

Components

Functional

Extensions

Parquet loading topic

Configuration properties

YAML example

Java configuration

Java example

​Configuration properties

​YAML example

​Java configuration

​Java example

Configuration properties

YAML example

Java configuration

Java example