Messaging source - Kafka and RabbitMQ

To start consumers for loading data from a messaging system, a START_LISTEN operation needs to be invoked.

Here, “scope” does not need to be provided. However, if one needs to load records with a particular CobDate, or from a particular offset in kafka, the relevant scope can be provided (examples further below).

Payload to start listening to a topic:

{
   "operation":"START_LISTEN",
   "topics":["Trade"]
}

To stop the consumers, a STOP_LISTEN request needs to be submitted. This request will stop all the consumers for the topic, and, as such, there is no need to provide a “scope”.

{
   "operation":"STOP_LISTEN",
   "topics":["Trade"]
}

If one needs to load records with a particular CobDate, following payload can be provided. The consumers will then discard records whose CobDate do not match the one specified. This is useful if one needs to strictly load a particular date, but it may make the consumer slow as it will check the date in each record. In most cases, a dedicated queue/topic is maintained for feeding to AP, therefore only the data that needs to be loaded to AP, will be submitted to the remote queue/topic.

{
   "operation":"START_LISTEN",
   "topics":["Trade"],
   "scope":{
      "CobDate":"2019-11-21"
   }
}

In case of Kafka, one may want to consume a topic from a particular offset (the position of a particular record). Below payload maps the topic’s partition id to an offset. This will cause the consumers to be assigned to the particular partition, and they will start consuming from the given offset.

{
   "operation":"START_LISTEN",
   "topics":["Trade"],
   "scope" :{
       "offsets" : {
                "partition-0" : "1234",
                "partition-1" : "0",
                "partition-2" : "1550919"
            }
   }
}