Package com.activeviam.source.common.api.impl

Class Fetch<I,E>

java.lang.Object
com.activeviam.source.common.api.impl.Fetch<I,E>
Type Parameters:
I - the type of the identifier of messages
E - the type of the data elements appended into message chunks
public final class Fetch<I,E> extends Object
Fetch data from a data source and into several stores, using a channel factory. The fetch is performed within one single transaction of the stores.

The Fetch class offers several constructors, which depending on what the user has already defined, will make it easy to use.

If you are using a channel factory to set up your loading, and have relatively few stores / topics, the Fetch(IStoreMessageChannelFactory) constructor is the way to go. If you want to fetch only from some stores, use the Fetch(IStoreMessageChannelFactory, Collection) constructor.

If you want to use a publisher which is different from the default publisher of your channel (i.e. TuplePublisher for most of the sources) you can use the Fetch(IStoreMessageChannelFactory, Collection, Map) constructor.

If you just have a ITransactionalWriter (that you may have retrieved from the datastore using myDatastore.getTransactionManager() method), use the Fetch(ITransactionalWriter, List) constructor. To fetch data only from some stores, use the Fetch(ITransactionalWriter, List, List) constructor.

Author:
ActiveViam

  • Constructor Details

    • Fetch

      public Fetch(ITransactionalWriter transactionalWriter, List<? extends IMessageChannel<I,E>> channels)
      Creates a fetch operation into the given list of channels. This will start the transaction on all stores.
      Parameters:
      transactionalWriter - transactional writer (must also be a transaction manager)
      channels - the list of channels to fetch the data from

    • Fetch

      public Fetch(ITransactionalWriter transactionalWriter, List<? extends IMessageChannel<I,E>> channels, @Nullable List<String> stores)
      Creates a fetch operation into the given list of channels.
      Parameters:
      transactionalWriter - transactional writer (must also be a transaction manager)
      channels - the list of channels to fetch the data from
      stores - stores on which the transaction should be started, null means all stores

    • Fetch

      public Fetch(IStoreMessageChannelFactory<I,E> channelFactory)
      Fetches of all the stores, mapped automatically to the source topics that have the same name as the store.
      Parameters:
      channelFactory - the channel factory of the source to fetch the data from

    • Fetch

      public Fetch(IStoreMessageChannelFactory<I,E> channelFactory, @Nullable Collection<String> stores)
      Fetches of a selection of stores, mapped automatically to the source topics that have the same name as the store.
      Parameters:
      channelFactory - the channel factory of the source to fetch the data from
      stores - stores on which the transaction should be started, null means all stores

    • Fetch

      public Fetch(IStoreMessageChannelFactory<I,E> channelFactory, @Nullable Collection<String> stores, Map<String,ITuplePublisher<I>> tupleMapping)
      Fetches of a selection of stores, mapped automatically to the source topics that have the same name as the store.
      Parameters:
      channelFactory - the channel factory of the source to fetch the data from
      stores - stores on which the transaction should be started, null means all stores
      tupleMapping - mapping between the source topics and their respective tuple publishers

    • Fetch

      public Fetch(IStoreMessageChannelFactory<I,E> channelFactory, Map<String,String> mapping, @Nullable Map<String,ITuplePublisher<I>> tupleMapping)
      Fetches a selection of topics into a selection of stores, with respect to the given topic/store mapping.
      Parameters:
      channelFactory - the channel factory of the source to fetch the data from
      mapping - mapping between source topics and destination stores
      tupleMapping - mapping between the source topics and their respective tuple publishers

  • Method Details

    • defaultMapping

      public static Map<String,String> defaultMapping(Collection<String> stores)
      Creates a mapping between stores and topic, with stores names equals to the topic names.
      Parameters:
      stores - a collection of stores on which to create the mapping
      Returns:
      default mapping for the given store names

    • fetch

      public IDatastoreSchemaTransactionInformation fetch(ISource<I,E> source)
      Fetches data into the IEpoch.MASTER_BRANCH_NAME branch.
      Parameters:
      source - the data source to fetch data from
      Returns:
      the schema transaction information

    • fetch

      public IDatastoreSchemaTransactionInformation fetch(ISource<I,E> source, String branch)
      Fetches data in the specified branch branch. If the branch does not already exist, it will be created.
      Parameters:
      source - the data source to fetch data from
      branch - the branch of the datastore to be updated
      Returns:
      the schema transaction information

    • fetch

      public IDatastoreSchemaTransactionInformation fetch(ISource<I,E> source, String branch, String parent)
      Main fetching method. This method:
      • starts the transaction
      • creates the channels to add all the tuples that are parsed and fetches the data in a synchronous way
      • commits the transaction on the specified branch from the specified parent branch (if branch does not exist)
      Parameters:
      source - the data source to fetch data from
      branch - the name of branch to start the transaction on
      parent - the name of a branch on which branch will be started. If parent is not null, then branch must not be the name of an existing branch. When parent is null, branch starts on IEpoch.MASTER_BRANCH_NAME.
      Returns:
      the schema transaction information