Configuration Properties
This section documents the different configuration properties that can be used when running Atoti FRTB with DirectQuery enabled.
By default, the properties are defined in the file frtb-starter/src/main/resources/application.yaml for optimal readability. You can also define these properties in another .yaml or
.properties files. If defining in a .properties file, you must specify the full property path.
Properties
Here is a list of all DirectQuery configuration properties available for all Databases. Note that the properties can be defined with kebab-case or camelCase in the .properties,.yaml or as command line arguments.
Deployment Type
Base path: starter.deployment.type
This property is used to specify the deployment type of the application. To enable the use of direct query, the deployment type should be set to direct-query.
Common DirectQuery Properties
Base path: directquery
These properties are applied to any database the application is connecting to.
Dates Filter
Base path: directquery.dates-filter
Default Behavior: The DirectQuery data node will include all dates in the remote database.
These properties are used to specify the dates that the DirectQuery data node has access to. This is important as the DirectQuery data node should not contain any dates included in any other data node as when using distribution, two nodes of the same cube cannot contain the same date.
The dates filter can be customized to handle custom filtering use cases. Please see the Dates Filter Bean documentation for more information.
By default, the DirectQuery data node will include all dates in the remote database. If you want to include only a subset of dates, you can specify them here.
dates-to-include
Default Value: Empty List
Only the specified dates will be included in the DirectQuery data cube, all other dates will be excluded.
This property cannot be set along with dates-to-exclude.
dates-to-exclude
Default Value: Empty List
The specified list od dates will be excluded from the DirectQuery data cube and all other dates on the database will be included.
This property cannot be set along with dates-to-include.
Schema Validation
Base path: directquery.schema.validation
These properties are used to specify when and how to validate the remote database schema. If the remote database schema is bad or invalid, this can result in significant performance issues. A schema with duplicate keys can result in longer than normal aggregation and application refresh times.
validate-on-startup
Default Value: true
This will initiate a validation check of the remote database when the application starts. This validation will check that all remote tables are readable and
that they do not contain any duplicate keys. If there are any errors they will be logged and, if configured with the fail-startup-on-bad-schema property, an
exception will be thrown.
fail-startup-on-bad-schema
Default Value: true
This property sets what you want the system to do when a bad schema is encountered when the application is starting: either fail or just log warnings.
Clustering
Base path: directquery.clustering
Clustering properties are specified per database, for example, when specifying the clustering fields for the database in use:
directquery:
clustering:
enabled: true
clustering-per-table:
SASensitivities: AsOfDate
Clustering allows Atoti Server to break up queries into smaller ones that can be run in parallel. This improves the initial aggregation time on the remote database as the aggregation query can be run in parallel. See the Atoti Server documentation on clustering fields for more information.
Database Specific Properties
Base path: directquery.database.xxx
These properties are applied to the specific database the application is connecting to and may be unique per database.
type
Default Value: clickhouse
The database to use in the current instance of the application. This value must be the name of the database in lowercase.
enabled
Default Value: false
If we want to enable clustering for this database.
clustering-per-table
Default Value: empty list
Specifies a mapping of tables to a list of clustering fields.
Here is an example of how to specify the key values:
clustering-per-table:
TableName1: Field_1, Field_2
TableName2: Field_1
etc...
Snowflake Database Properties
Base path: directquery.database.snowflake
These properties only apply to the Snowflake database.
connection-string
Default Value: null
The connection string used to connect to the remote Snowflake instance.
username
Default Value: null
The username to use to authenticate the connection to Snowflake.
password
Default Value: null
The password to use to authenticate the connection to Snowflake.
warehouse
Default Value: null
The warehouse (compute) to use when executing queries on Snowflake.
feeding-warehouse
Default Value: directquery.database.snowflake.warehouse value
The warehouse (compute) to use when computing the initial aggregates on Snowflake. This will also be the warehouse to use when performing an Application Refresh.
The idea is to use a larger instance so the initial aggregation time can be faster. This warehouse will only be used for the initial aggregation and refreshes,
any other user query that hits Snowflake will be executed using the directquery.database.warehouse warehouse.
database
Default Value: null
The Snowflake Database to connect to and use for queries.
schema
Default Value: null
The Snowflake Schema to connect to and use for queries.
ClickHouse Database Properties
Base path: directquery.database.clickhouse
These properties only apply to the ClickHouse database.
username
Default Value: null
The optional username to use to authenticate the connection to ClickHouse. If no authentication is defined on your ClickHouse database, these properties can be ignored.
password
Default Value: null
The optional password to use to authenticate the connection to ClickHouse. If no authentication is defined on your ClickHouse database, these properties can be ignored.
port
Default Value: null
The port the ClickHouse database is exposed on.
host-name
Default Value: null
The hostname the ClickHouse database is hosted from. For example, when running locally this would be localhost
database
Default Value: null
The ClickHouse Database to connect to and use for queries.
schema
Default Value: null
The ClickHouse Schema to connect to and use for queries.
Databricks Database Properties
Base path: directquery.database.databricks
These properties only apply to the Databricks database.
timeTravelPolicy
Default Value: DISABLED
The property that allows for time-travel based on the policy specified.
connectionString
Default Value: null
The connection string used to connect to the remote Databricks cluster.
heavyLoadingConnectionString
Default Value: null
The heavy loading connection string computes the aggregates and is only used when the application starts or is refreshed.
database
Default Value: null
The Databricks Database to connect to and use for queries.
schema
Default Value: null
The Databricks Schema to connect to and use for queries.