Package com.activeviam.risk.ref.cfg.impl
Class ACSVSourceConfig
- java.lang.Object
-
- com.activeviam.risk.ref.cfg.impl.ACSVSourceConfig
-
- Direct Known Subclasses:
ASensiSourceConfig
,PnLImportSourceConfig
,PnLSourceConfig
,VaRImportSourceConfig
,VaRSourceConfig
public abstract class ACSVSourceConfig extends Object
-
-
Field Summary
Fields Modifier and Type Field Description protected com.qfs.server.cfg.IDatastoreConfig
datastoreConfig
static String
DEFAULT_CSV_BUFFER_SIZE
static String
DEFAULT_CSV_LINES_TO_SKIP
static String
DEFAULT_CSV_PARSER_THREADS
static String
DEFAULT_CSV_SEPARATOR
protected org.springframework.core.env.Environment
env
protected String
folder
protected static Logger
LOGGER
-
Constructor Summary
Constructors Constructor Description ACSVSourceConfig()
-
Method Summary
All Methods Instance Methods Abstract Methods Concrete Methods Modifier and Type Method Description protected com.activeviam.tools.dataloadcontroller.core.source.csv.CsvScopedFetchSource<Path>
createCsvSource(String sourceName)
Create a CSV source object suitable for use with theIDataLoadController
.protected <I> SimpleSourceConfig.ITopicConfig<com.qfs.msg.csv.IFileInfo<Path>>
createTopicConfig(String topic, String store, com.qfs.source.ITuplePublisher<com.qfs.msg.csv.IFileInfo<Path>> tuplePublisher)
Create anSimpleSourceConfig.ITopicConfig
object.protected com.qfs.source.impl.CSVMessageChannelFactory<Path>
csvChannelFactory()
This function must be overridden in order to create a specific beanabstract com.activeviam.tools.dataloadcontroller.core.source.csv.CsvScopedFetchSource<Path>
csvSource()
com.activeviam.tools.dataloadcontroller.core.source.csv.CsvScopedFetchSource<Path>
csvSource(String sourceName)
Generates a specific source with topicsProperties
csvSourceConfiguration()
The CSVSource propertiesvoid
defineTopic(com.qfs.msg.csv.impl.CSVSource<Path> source, String filePath, String topicName)
protected List<String>
getFileColumns(String topicName, Map<String,com.quartetfs.fwk.IPair<String,String>> topicToStoreAndFilePatternMap, com.qfs.store.IDatastoreSchemaMetadata schemaMetadata)
Retrieves the list of file columns for a given store, either from a custom list where configured, or from the store configuration.protected int
getLinesToSkip(String topic)
protected Map<String,SimpleSourceConfig.ITopicConfig<com.qfs.msg.csv.IFileInfo<Path>>>
getTopicToPublisherMap()
protected abstract Map<String,com.quartetfs.fwk.IPair<String,String>>
getTopicToStoreAndFilePatternMap()
protected com.qfs.source.ITuplePublisher<com.qfs.msg.csv.IFileInfo<Path>>
getTuplePublisher(String topic)
This function is intended to be overridden in order to specify some specific tuple publishersvoid
verifyFolder(String dataSet)
Verify that a folder is valid
-
-
-
Field Detail
-
LOGGER
protected static Logger LOGGER
-
DEFAULT_CSV_PARSER_THREADS
public static final String DEFAULT_CSV_PARSER_THREADS
- See Also:
- Constant Field Values
-
DEFAULT_CSV_BUFFER_SIZE
public static final String DEFAULT_CSV_BUFFER_SIZE
- See Also:
- Constant Field Values
-
DEFAULT_CSV_LINES_TO_SKIP
public static final String DEFAULT_CSV_LINES_TO_SKIP
- See Also:
- Constant Field Values
-
DEFAULT_CSV_SEPARATOR
public static final String DEFAULT_CSV_SEPARATOR
- See Also:
- Constant Field Values
-
env
@Autowired protected org.springframework.core.env.Environment env
-
datastoreConfig
@Autowired protected com.qfs.server.cfg.IDatastoreConfig datastoreConfig
-
folder
@Autowired @Qualifier("folder") protected String folder
-
-
Method Detail
-
getTopicToStoreAndFilePatternMap
protected abstract Map<String,com.quartetfs.fwk.IPair<String,String>> getTopicToStoreAndFilePatternMap()
-
csvSource
@Bean public abstract com.activeviam.tools.dataloadcontroller.core.source.csv.CsvScopedFetchSource<Path> csvSource()
-
defineTopic
public void defineTopic(com.qfs.msg.csv.impl.CSVSource<Path> source, String filePath, String topicName)
-
createTopicConfig
protected <I> SimpleSourceConfig.ITopicConfig<com.qfs.msg.csv.IFileInfo<Path>> createTopicConfig(String topic, String store, com.qfs.source.ITuplePublisher<com.qfs.msg.csv.IFileInfo<Path>> tuplePublisher)
Create anSimpleSourceConfig.ITopicConfig
object.- Type Parameters:
I
- type of identifier of the files- Parameters:
topic
- the topic namestore
- the topic's store name.tuplePublisher
- The channel's tuple puslisher- Returns:
- the topic config
-
getTuplePublisher
protected com.qfs.source.ITuplePublisher<com.qfs.msg.csv.IFileInfo<Path>> getTuplePublisher(String topic)
This function is intended to be overridden in order to specify some specific tuple publishers- Parameters:
topic
- The current topic- Returns:
- A tuple publisher if any or null
-
createCsvSource
protected com.activeviam.tools.dataloadcontroller.core.source.csv.CsvScopedFetchSource<Path> createCsvSource(String sourceName)
Create a CSV source object suitable for use with theIDataLoadController
.- Parameters:
sourceName
- name of the CSV source.- Returns:
- the CSV source
-
csvChannelFactory
@Bean protected com.qfs.source.impl.CSVMessageChannelFactory<Path> csvChannelFactory()
This function must be overridden in order to create a specific bean- Returns:
- the CSVMessageChannelFactory
-
csvSourceConfiguration
@Bean public Properties csvSourceConfiguration()
The CSVSource properties- Returns:
- properties
-
csvSource
public com.activeviam.tools.dataloadcontroller.core.source.csv.CsvScopedFetchSource<Path> csvSource(String sourceName)
Generates a specific source with topics- Parameters:
sourceName
- name of the source- Returns:
- the fetch source
-
verifyFolder
public void verifyFolder(String dataSet)
Verify that a folder is valid- Parameters:
dataSet
- folder
-
getTopicToPublisherMap
protected Map<String,SimpleSourceConfig.ITopicConfig<com.qfs.msg.csv.IFileInfo<Path>>> getTopicToPublisherMap()
-
getLinesToSkip
protected int getLinesToSkip(String topic)
-
getFileColumns
protected List<String> getFileColumns(String topicName, Map<String,com.quartetfs.fwk.IPair<String,String>> topicToStoreAndFilePatternMap, com.qfs.store.IDatastoreSchemaMetadata schemaMetadata)
Retrieves the list of file columns for a given store, either from a custom list where configured, or from the store configuration.- Parameters:
topicName
- The name of the destination store.topicToStoreAndFilePatternMap
- resouce used to find-out the store name from a specific topicschemaMetadata
- The schema metadata.- Returns:
- The list of columns in the source file.
-
-