JDBCSource configuration
Parameters impacting the JDBCSource:
There are several parameters that can have an impact on the JDBCSource and its performances :
- Source Properties : those properties are either a field of the JDBC source or a property of one of its fields.
poolSize
: Input argument given to the Source Constructor that drive the size of the ThreadPool in which theJDBCTask
will be executed. Defaults to 2. It can be an impacting factor when trying to fetch data of the underlying topic from numerous channels at the same time.appendQueueSize
: size of the append queue that will receive the parsed Records of a givenJDBCTask
. This property should be set at a larger value thanbatchSize
andJDBCTopic.fetchSize
, since those properties correspond to push/pop operations on a Collection.batchSize
: size of the collection aJDBCAppendRunnable
performs on the append Queue.
- Topic Properties : these properties are not directly linked to a JDBCSource, but will impact the Source Performances when asked to fetch the topic-related data.
JDBCTopic.fetchSize
: size used to receive SQL request answer. While fetching the data from ajava.sql.ResultSet
, we fetch only fetchSize-sized pieces of theResultSet
at once, this is can be used to control the network usage congestion. This property can be the performance bottleneck of the source.JDBCTopic.nbAppendThreads
: size of the thread pool in which theJDBCAppendRunnable
are executed.JDBCTopic.chunkSize
: size of the data chunks sent to theChannelMessage
in order to be sent to the DataStore.
While fetching data with a JDBCSource
, the size of the append queue is monitored. If the queue gets full, it becomes a limiting factor for the data fetching operation one can consider either increase the value of the property, the number of JDBCAppendRunnable
thread-pool size, or the batchSize
property value .