Navigation :

CSV Parser

Configuration Properties

Properties for the CSV Parser are provided inside the configuration of a CSV Topic.

Key	Type	Default	Description
charset	Charset	UTF-8	Character set used to decode the file.
number-skipped-lines	int	1	The number of lines to skip at the beginning of each file (header).
accept-overflowing-lines	boolean	false	When set to `false`, a record that contains more elements is rejected, otherwise the additional elements are ignored.
accept-incomplete-lines	boolean	true	When set to `false`, a record that contains fewer elements is rejected. If set to `true`, null values are inserted into the missing columns.
separator	char	,	Character which separates fields in the file.
columns	`List<String>`	Columns will implicitly come from the store which is being published to.	Column of the file. You must specify all column names. The ones that do not correspond to a matching column in the datastore are skipped.
process-quotes	boolean	true	Sets the behavior of the parser regarding quotes.

YAML Example

dlc:
  csv:
    topics:
      trades:
        parser:
          accept-overflowing-lines: true
          separator: "\t"

Java Configuration

CsvParserDescription Java object can be used for configuration in Java. The object is used in the configuration of a CSV Topic.

CsvParserDescription

Parameter	Type	Default	Description
charset	Charset	UTF-8	Character set used to decode the file.
numberSkippedLines	int	1	The number of lines to skip at the beginning of each file (header).
acceptOverflowingLines	boolean	false	When set to `false`, a record that contains more elements is rejected, otherwise the additional elements are ignored.
acceptIncompleteLines	boolean	true	When set to `false`, a record that contains fewer elements is rejected. If set to `true`, null values are inserted into the missing columns.
separator	char	,	Character which separates fields in the file.
columns	`List<String>`	Columns will implicitly come from the store which is being published to.	Column of the file. You must specify all column names. The ones that do not correspond to a matching column in the datastore are skipped.
processQuotes	boolean	true	Sets the behavior of the parser regarding quotes.

Java Example

@Bean
public CsvTopicDescription tradesTopic() {
    return CsvTopicDescription.builder("Trades", "glob:trades*.csv")
            .parser(
                    CsvParserDescription.builder()
                            .separator('\t')
                            .acceptOverflowingLines(true)
                            .build()
            ).build();
}