☰
Navigation :

Parquet Topic

Configuration Properties

Properties for a Parquet Topic are defined by name in the dlc.parquet.topics namespace, and are picked up as configurations for the DLC.

Key Required Type Default Description
file-pattern String Path matcher.
column-names List<String> Columns will come from the store which is being published to Column of the file. You must specify all column names in the file. The ones that do not correspond to a matching column in the datastore are skipped.
source-store-mapping Map<String, String> Ordered map of fields in the record from the remote topic or queue
channels Set<Channel> A channel is created using the topic’s name as the name of the target. Channels.
restrict-to-sources Set<String> Sources to which the topic is restricted.
restrict-from-sources Set<String> Sources from which the topic restricted.

YAML Example

dlc:
  parquet:
    topics:
      Trades:
        file-pattern: trades*.parquet

Java Configuration

ParquetTopicDescription Spring Beans are picked up as configurations for the DLC.

Parameter Required Type Default Description
name Y String Name of the topic.
filePattern String Path matcher.
columnNames List<String> Columns will come from the store which is being published to Column of the file. You must specify all column names in the file. The ones that do not correspond to a matching column in the datastore are skipped.
sourceStoreMapping Map<String, String> Ordered map of fields in the record from the remote topic or queue
channels Set<ChannelDescription> A channel is created using the topic’s name as the name of the target Channel Descriptions which belong to the topic.
restrictToSources Set<String> Sources to which topic is restricted.
restrictFromSources Set<String> Sources from which topic is restricted.

Java Example

@Bean
public ParquetTopicDescription tradesTopic() {
    return ParquetTopicDescription.builder("Trades", "trades*.parquet")
            .build();
}