JDBCSource configuration

Parameters impacting the JDBCSource:

There are several parameters that can have an impact on the JDBCSource and its performances :

• Source Properties : those properties are either a field of the JDBC source or a property of one of its fields.
  • poolSize : Input argument given to the Source Constructor that drive the size of the ThreadPool in which the JDBCTask will be executed. Defaults to 2. It can be an impacting factor when trying to fetch data of the underlying topic from numerous channels at the same time.
  • appendQueueSize : size of the append queue that will receive the parsed Records of a given JDBCTask . This property should be set at a larger value than batchSize and JDBCTopic.fetchSize, since those properties correspond to push/pop operations on a Collection.
  • batchSize : size of the collection a JDBCAppendRunnable performs on the append Queue.
• Topic Properties : these properties are not directly linked to a JDBCSource, but will impact the Source Performances when asked to fetch the topic-related data.
  • JDBCTopic.fetchSize : size used to receive SQL request answer. While fetching the data from a java.sql.ResultSet, we fetch only fetchSize-sized pieces of the ResultSet at once, this is can be used to control the network usage congestion. This property can be the performance bottleneck of the source.
  • JDBCTopic.nbAppendThreads : size of the thread pool in which the JDBCAppendRunnable are executed.
  • JDBCTopic.chunkSize: size of the data chunks sent to the ChannelMessage in order to be sent to the DataStore.

While fetching data with a JDBCSource, the size of the append queue is monitored. If the queue gets full, it becomes a limiting factor for the data fetching operation one can consider either increase the value of the property, the number of JDBCAppendRunnable thread-pool size, or the batchSize property value .