This section describes configurations for loading data via messaging streams using consumers.
Messaging Sources are sources that require an open connection to stream data from the source to the consumer.
Messaging Consumers
Messaging consumers are responsible for connecting to the remote messaging system, listening constantly for new data,
and consuming data when available.
The consumers are created when the request to start “listening” to the remote topic is submitted at runtime. The underlying
Kafka client consumers are created and managed by the messaging consumer implementation - KafkaMessageConsumer.
The remote topics are polled for records when the consumer thread begins.
Messaging Transaction Controller Tasks
The messages collected in the consumers are applied to the Datastore in bulk operations. The cadence of
these transactions can be tuned via the TxControllerTaskDescription.
Messaging Topics
The Messaging Topic Description contains an SQL statement to retrieve data from a messaging producer.