Atoti supports both one-off and continuous data loading strategies to populate and update the datastore. One-off loading is ideal for static datasets or initial ingestion. Continuous loading ensures that updates are automatically detected and processed in real time or near real time.Documentation Index
Fetch the complete documentation index at: https://docs.activeviam.com/llms.txt
Use this file to discover all available pages before exploring further.
What is the difference between one-off and continuous loading?
Understanding the difference between one-off and continuous loading helps choose the right strategy to balance performance and data freshness.| One-off loading | Continuous loading | |
|---|---|---|
| Purpose | Initial ingestion or manual refresh | Automatic updates from changing data |
| Trigger | Manual or scheduled | Event-driven or polling-based |
| Use cases | Historical data and static datasets | Streaming dashboards and live monitoring |
| Performance impact | Depends on dataset size | Depends on update frequency and volume |
| Monitoring and reporting | Optional parsing reports | Optional parsing reports and stream events |
How does Atoti detect changes for continuous loading?
For file-based sources, Atoti uses a Watcher service that listens for file system events and reacts immediately when files are created or modified. This provides low-latency detection of changes. For message-based sources such as Kafka or JMS, data is pushed directly to Atoti as messages arrive, enabling continuous ingestion.What sources support continuous loading?
Each source has its own capabilities and limitations when it comes to detecting and ingesting changes efficiently. The following categories show which sources support continuous loading, one-off loading, or both:One-off loading sources
These sources are typically used for initial ingestion or manual refreshes. They do not support automatic detection of changes without external orchestration.- Flat files (e.g. CSV, Parquet): Loaded via fetch operations.
- Relational databases (JDBC): Queries are executed to retrieve and load data.
- Cloud platforms (e.g. BigQuery, Snowflake): Used for scheduled or batch-based loading.
Continuous loading sources
These sources support automatic updates and are designed for low-latency ingestion.- Flat files (e.g. CSV, Parquet): Monitored using polling or watcher service.
- Messaging systems (e.g. Kafka, JMS): Designed for streaming ingestion.
- Custom systems (APIs, platforms): Can be integrated to push updates continuously.
Sources that support both
Some sources can be configured for either one-off or continuous loading depending on the use case and integration method.- Flat files: Can be fetched once or monitored continuously.
- Relational databases: Support one-off fetch and incremental refresh; continuous loading may require orchestration.
- Cloud platforms: Typically used for scheduled loads but can be extended for continuous updates using external triggers.
- Custom systems: Behavior depends on the integration; both loading types are possible.
How does continuous loading work?
Continuous loading in Atoti refers to the automatic detection and ingestion of new or updated data without manual intervention. It ensures that the datastore remains synchronized with external data sources in near real time. To help clarify the different mechanisms available, the following numbered list outlines the main approaches supported by Atoti for continuous loading. Each method serves a distinct purpose and applies to different types of data sources and integration patterns.- Atoti Python SDK uses other Python based libraries to manage some continuous loading
- Atoti Java SDK has in built APIs
1. File-based continuous loading
Use this approach when your data is stored in flat files such as CSV or Parquet. Files are monitored using:- Polling: Periodically checks file modification timestamps.
- Watcher service: Listens for file system events and reacts immediately (for example a new file created in a folder).
- When working with file-based data pipelines.
- When you need low-latency updates and the file system supports event notifications (recommended: watcher service).
- When you want a simple setup without external orchestration.
Note: When monitoring files for changes, be aware that Atoti will reload all lines, including those that are unmodified. This is acceptable if a small percentage of the lines are unchanged. However, if you only intend to modify or add a few lines, it’s more efficient to place these lines in a new file. This allows the system watcher service to detect its arrival.
2. Streaming query updates
Use this approach when you need real-time updates to query results, such as in dashboards or alerting systems. Atoti’s Streaming API allows clients to:- Subscribe to a query once.
- Receive updates automatically when the result changes.
- When building interactive dashboards that must reflect live data.
- When you want to avoid polling the server for query results.
- When using Atoti UI or custom clients that support WebSocket communication.
3. Long polling for remote clients
Use this approach when your client cannot maintain a persistent connection, such as in .NET environments or restricted networks. Long polling simulates streaming by:- Submitting a listen request.
- Waiting for updates or timeout.
- Resubmitting the request immediately.
- When WebSocket is not available or not supported by the client.
- When integrating with remote systems that require HTTP-based communication.
- When you need to multiplex updates from multiple domains over a single connection.
4. Incremental refresh from external databases (DirectQuery only)
Use this approach when your data is stored in an external database and you want to update specific rows. Incremental refresh:- Targets specific changes using conditions (e.g., by date or ID).
- Supports multi-table updates.
- Can use unknown conditions when precise scopes are not available.
- When working with DirectQuery schemas.
- When you want to avoid full reloads and reduce database load.
Related reading
Find out more about file based continuous loading by following these links:- Atoti Java SDK
- Atoti Python SDK uses other Python based libraries