How to use Transaction Record Blocks

Transaction Record Blocks provide a high-performance alternative to addAll/removeAll for bulk record operations, available outside a transaction. The standard addAll method takes a Collection<Object[]>, which forces every primitive value to be boxed (int to Integer, double to Double, etc.). Record blocks avoid this boxing overhead by offering typed write methods (writeInt, writeDouble, writeLong, etc.) that work directly with primitives. Record blocks also handle automatic dictionarization of values and support BlockOptions to further optimize loading for common patterns such as single-value or default-value columns. Multiple threads can build and submit blocks concurrently within a single transaction, enabling parallel loading.

A single block is not thread-safe. Use separate blocks per thread instead of sharing one.
A block must not be used after it has been submitted to the Transaction Manager via addRecords or removeRecords.

Experimental

This feature is experimental and must be enabled with the following system property:

activeviam.feature.experimental.advanced_datastore_transaction_manager_operations.enabled=true

The next examples use a store with the following schema:

private static final IStoreDescription STORE_DESCRIPTION =
    StoreDescription.builder()
        .withStoreName("Trades")
        .withField("tradeId", ILiteralType.INT)
        .asKeyField()
        .withField("date", ILiteralType.LOCAL_DATE)
        .withField("product", ILiteralType.STRING)
        .withField("quantity", ILiteralType.DOUBLE, 0.0)
        .build();

Create a block and add records

Blocks can be created and filled outside a transaction. Only addRecords and removeRecords require an active transaction. This allows you to prepare data ahead of time and keep the transaction window as short as possible. A block has a capacity and a size. The capacity is the maximum number of records the block can hold, set when creating the factory with createBlockFactory(tableName, capacity). The size is the number of records actually written to the block. By default, the size equals the capacity. If you fill fewer rows than the capacity, call setSize to indicate how many rows should be submitted.

Performance

Prefer multiple reasonably-sized blocks over one large block. A single block is processed sequentially at submission time, whereas multiple smaller blocks can be filled in parallel from different threads within the same transaction.

To add records using a block:

Get the ITransactionManager from the datastore.
Create an ITransactionRecordBlockFactory for the target table, specifying a capacity and optionally providing BlockOptions.
Create an ITransactionRecordBlock from the factory with createBlock().
Write values using typed methods (e.g. writeInt, writeDouble, write).
Call setSize with the actual number of records written.
Submit the block with addRecords within a transaction.

final ITransactionManager tm = datastore.getTransactionManager();
// Create a factory for the "Trades" table with a block capacity of 1024 records
final ITransactionRecordBlockFactory factory = tm.createBlockFactory("Trades", 1024);
final ITransactionRecordBlock block = factory.createBlock();
// Retrieve field positions from the record format
final IRecordFormat format = block.getRecordFormat();
final int idIndex = format.getFieldIndex("tradeId");
final int dateIndex = format.getFieldIndex("date");
final int productIndex = format.getFieldIndex("product");
final int quantityIndex = format.getFieldIndex("quantity");
// Write records row by row using typed methods to avoid boxing
for (int row = 0; row < 3; ++row) {
  block.writeInt(row, idIndex, row);
  block.write(row, dateIndex, LocalDate.of(2025, 6, 1));
  block.write(row, productIndex, "Product" + row);
  block.writeDouble(row, quantityIndex, row * 10d);
}
// Set the actual number of records written (may be less than the block capacity)
block.setSize(3);
// Submit the block within a transaction
tm.startTransaction("Trades");
tm.addRecords(block);
tm.commitTransaction();

Use writeDefault to write the schema default value for a column.

Performance

Although the underlying data structure of a record block is row-oriented (since records are added row by row in a datastore transaction), filling it field by field (i.e. in a columnar fashion) is usually more efficient. This is because dictionary cache locality matters more: when writing all values for the same dictionarized field consecutively, the dictionary stays hot in cache.

Block options

BlockOptions allow you to optimize blocks when some columns have predictable values.

Single-value columns

Single-value columns are bound to a fixed value shared by all rows in the block. They are immutable after block creation. This is useful when all records in a block share the same value for a column (e.g. a partition date), as it avoids redundant per-row writes.

Default-value columns

Default-value columns are initialized with their default values from the table schema. Unlike single-value columns, they can be overwritten per row. This is useful when many fields have defaults and only a few rows need explicit values. A column cannot be both a single-value and a default-value column.

// "Date" is the same for all rows in the block and cannot be modified after creation
// "Quantity" is initialized with its schema default (0.0) but can be overwritten per row
final ITransactionRecordBlockFactory factory =
    tm.createBlockFactory(
        "Trades",
        1024,
        BlockOptions.builder()
            .singleValueColumns(Map.of("date", LocalDate.of(2025, 6, 1)))
            .defaultValuesColumns(List.of("quantity"))
            .build());
final ITransactionRecordBlock block = factory.createBlock();
final IRecordFormat format = block.getRecordFormat();
final int idIndex = format.getFieldIndex("tradeId");
final int productIndex = format.getFieldIndex("product");
final int quantityIndex = format.getFieldIndex("quantity");
for (int row = 0; row < 3; ++row) {
  block.writeInt(row, idIndex, row);
  block.write(row, productIndex, "Product" + row);
}
// Overwrite the default quantity only for specific rows
block.writeDouble(1, quantityIndex, 50d);
block.setSize(3);
tm.startTransaction("Trades");
tm.addRecords(block);
tm.commitTransaction();

Remove records

To remove records using a block, create a block dedicated for removal with createBlockForRemoval(). Only key fields need to be written. Then submit the block with removeRecords within a transaction.

It is the caller’s responsibility to pass the right type of block to the right method:

Blocks created with createBlock() must be submitted with addRecords.
Blocks created with createBlockForRemoval() must be submitted with removeRecords.

final ITransactionRecordBlockFactory factory = tm.createBlockFactory("Trades", 1024);
// Create a block dedicated for removal: only key fields need to be written
final ITransactionRecordBlock block = factory.createBlockForRemoval();
final int keyFieldIndex = block.getKeyFields()[0];
// Remove records with key 1 and 3
block.writeInt(0, keyFieldIndex, 1);
block.writeInt(1, keyFieldIndex, 3);
block.setSize(2);
tm.startTransaction("Trades");
tm.removeRecords(block);
tm.commitTransaction();

Transfer existing records

The transfer method copies an existing record from the most recent version of the store into a block at a given row, looked up by key. The store must have key fields defined. This enables an update-where pattern outside a transaction: transfer a record, modify specific fields according to existing data, and submit the block back. transfer returns true if the record was found, false otherwise. In both cases, the key values are always written into the block. Record blocks also provide read methods (readInt, readDouble, read, …) that return undictionarized values, which can be used to inspect transferred data before modifying it. Fields configured as single-value columns are not transferred, as they are already set at block creation time. For batch transfers, transferRange transfers several records at once for better performance.

final ITransactionRecordBlockFactory factory = tm.createBlockFactory("Trades", 1024);
final ITransactionRecordBlock block = factory.createBlock();
final IRecordFormat format = block.getRecordFormat();
final int quantityIndex = format.getFieldIndex("quantity");
// Transfer the existing record with key 1 into the block at row 0.
// All field values are copied from the store into the block.
final boolean found = tm.transfer(block, 0, 1);
if (found) {
  // Modify the quantity of the transferred record
  block.writeDouble(0, quantityIndex, block.readDouble(0, quantityIndex) * 2d);
}
block.setSize(1);
// Submit the modified record back to the store (this performs an update since the key exists)
tm.startTransaction("Trades");
tm.addRecords(block);
tm.commitTransaction();

These methods do not require an active transaction. For update-where operations inside a transaction, prefer the regular updateWhere API.

Create blocks from a list query

The createBlocksFromQuery method creates blocks populated with all records from the most recent version of a store matching a given query’s condition. Unlike transfer which looks up individual records by key, this method works with arbitrary conditions and does not require the store to have key fields. The query’s selected fields determine which columns are copied into the blocks. All selected fields must be fields of the block’s table, though they can be reached through references if the query starts from a different table. Fields configured as single-value columns are not copied, as they are already set at block creation time. This method benefits from the same optimizations as a regular list query. Dictionarized data is copied directly from the store’s internal storage into the blocks, avoiding intermediate buffering. Since the number of matching records is not known upfront and because blocks are produced in parallel for each partition, the method creates blocks as needed and returns them as a collection. Some blocks may not be full. This method does not require an active transaction.

final ITransactionRecordBlockFactory factory = tm.createBlockFactory("Trades", 1024);
final ListQuery query =
    datastore
        .getQueryManager()
        .listQuery()
        .forTable("Trades")
        .withCondition(BaseConditions.equal(FieldPath.of("date"), LocalDate.of(2025, 6, 2)))
        .selectingAllTableFields()
        .toQuery();
// Create blocks containing all records where Date = 2025-06-02
final Collection<ITransactionRecordBlock> blocks = tm.createBlocksFromQuery(factory, query);
// The returned blocks contain the matching records with all field values copied
// They can be read, modified, and submitted back to the store
final IRecordFormat format = factory.getRecordFormat();
final int quantityIndex = format.getFieldIndex("quantity");
for (final ITransactionRecordBlock block : blocks) {
  for (int row = 0; row < block.size(); ++row) {
    // Double the quantity of each matching record
    block.writeDouble(row, quantityIndex, block.readDouble(row, quantityIndex) * 2d);
  }
}
tm.startTransaction("Trades");
for (final ITransactionRecordBlock block : blocks) {
  tm.addRecords(block);
}
tm.commitTransaction();

Copy rows between blocks

The copyFrom method copies rows directly from one block to another at the dictionarized level, without undictionarizing and re-dictionarizing each value. This is useful when generating new records from existing blocks, for example creating risk scenarios from reference data. Both blocks must be associated with the same table and have the same set of single-value column names. Single-value column values are not copied: the destination block retains its own values. This method does not require an active transaction, and does not update the destination block’s size. The following example creates a source block, copies all its rows into a destination block, modifies the keys, and submits both blocks:

final ITransactionRecordBlockFactory factory = tm.createBlockFactory("Trades", 1024);
// Fill a source block with records
final ITransactionRecordBlock source = factory.createBlock();
final IRecordFormat format = source.getRecordFormat();
final int idIndex = format.getFieldIndex("tradeId");
final int dateIndex = format.getFieldIndex("date");
final int productIndex = format.getFieldIndex("product");
final int quantityIndex = format.getFieldIndex("quantity");
for (int row = 0; row < 3; ++row) {
  source.writeInt(row, idIndex, row);
  source.write(row, dateIndex, LocalDate.of(2025, 6, 1));
  source.write(row, productIndex, "Product" + row);
  source.writeDouble(row, quantityIndex, (row + 1) * 10d);
}
source.setSize(3);
// Copy all rows from the source block into a new destination block
final ITransactionRecordBlock dest = factory.createBlock();
dest.copyFrom(source, 0, source.size(), 0);
// Modify the copied records: assign new keys
for (int row = 0; row < 3; ++row) {
  dest.writeInt(row, idIndex, row + 100);
}
dest.setSize(3);
// Submit both blocks
tm.startTransaction("Trades");
tm.addRecords(source);
tm.addRecords(dest);
tm.commitTransaction();

Documentation Index

​Experimental

​Create a block and add records

​Performance

​Performance

​Block options

​Single-value columns

​Default-value columns

​Remove records

​Transfer existing records

​Create blocks from a list query

​Copy rows between blocks

Experimental

Create a block and add records

Performance

Performance

Block options

Single-value columns

Default-value columns

Remove records

Transfer existing records

Create blocks from a list query

Copy rows between blocks