Skip to main content
The Atoti DirectQuery Databricks connector uses the Databricks JDBC driver to connect to a lakehouse. The driver supports several authentication types, and only some of them suit a long-running server. Read access to the queried data is granted separately, through Unity Catalog. Creating the connector itself is described in Create the database connector.

​
Credentials

DirectQuery passes both the connection string parameters and the explicitly set JDBC properties to the Databricks JDBC driver DataSource. An explicitly set property takes precedence over a connection string parameter of the same name.
Secrets do not belong in the connection string, which can end up in log files. Pass every secret as an additionalOption on DatabricksConnectionProperties instead.

​
How to authenticate with OAuth machine-to-machine

OAuth machine-to-machine is the recommended authentication type for a self-hosted Atoti Server. A Databricks service principal authenticates with a client ID and a client secret. The driver obtains and renews access tokens by itself. No operator action is needed for the lifetime of a long-running server. Set AuthMech=11, Auth_Flow=1 and OAuth2ClientId in the connection string:
Pass the client secret as an additional option so that it stays out of the connection string:
The service principal setup is described in How to create the service principal and OAuth application.

​
How to authenticate with a service principal private key

This type replaces the shared client secret with a private key held by the service principal. Add UseJWTAssertion=1 to the OAuth machine-to-machine parameters. The driver then signs a JWT assertion with the key. Access tokens are still renewed by the driver itself. These parameters describe the key:
  • Auth_JWT_Key_File: path to the private key file.
  • Auth_KID: key identifier.
  • Auth_JWT_Alg: signing algorithm. Defaults to RS256.
  • Auth_JWT_Key_Passphrase: passphrase, when the key file is encrypted.
Set Auth_Scope=sql as well, because the driver sends no scope of its own on this type.
On Azure, a connection that sets AzureTenantId uses the Azure service principal type instead, which ignores UseJWTAssertion.

​
How to authenticate with an Azure managed identity

An Atoti Server running on Azure compute can authenticate to an Azure Databricks workspace with an Azure managed identity. This type is only available on Azure. No secret has to be stored in the Atoti Server configuration. Access tokens are renewed by the driver itself. Set AuthMech=11 and Auth_Flow=3, together with azure_workspace_resource_id, which the driver requires for this type. For a user-assigned managed identity, add OAuth2ClientId to select which identity to use.

​
How to authenticate with a personal access token

A Databricks personal access token is the quickest type to set up. Set AuthMech=3 and UID=token in the connection string:
Pass the token itself as the PWD property, available as DatabricksConnectionProperties.PASSWORD_PROPERTY_KEY:
Personal access tokens expire and have to be rotated by an operator. OAuth machine-to-machine is preferable for a long-running server.

​
How to authenticate with browser-based OAuth

Browser-based OAuth uses AuthMech=11 with Auth_Flow=2. It requires a person to sign in through a browser. It is therefore unsuitable for a server deployment. Use it for local development and exploration only. The driver caches the resulting token on disk by default, and refuses to connect until that cache is configured. Either set TokenCachePassPhrase, which encrypts the cache, or turn the cache off with EnableTokenCache=0.

​
How does Databricks token federation apply?

Token federation is not an authentication type. It is a layer that the driver applies on top of the configured authentication type. Token federation is enabled by default, because the driver’s EnableTokenFederation parameter defaults to 1. When the workspace is set up for OAuth token federation, the driver exchanges the token for a workspace token automatically. The Identity_Federation_Client_Id parameter names the OAuth application used for the exchange. No extra configuration is needed. Configure OAuth machine-to-machine as described in How to authenticate with OAuth machine-to-machine, and token federation applies on top of it. The exchange only takes place when the token was issued by a provider other than the Databricks workspace itself.

​
Permissions

The user or service principal used by DirectQuery needs read access to the queried data, granted through Unity Catalog. The minimum privileges are:
  • USE CATALOG on the target catalog
  • USE SCHEMA on the target schema
  • SELECT on the queried tables and views
These three privileges cover the complete DirectQuery read path. They are sufficient for the default time-travel behavior, which reads table history, and for the detection of views. DirectQuery does not write to the lakehouse. The aggregate tables feature does not change these privileges. Atoti generates the CREATE TABLE and INSERT INTO statements but never executes them: they are run separately, under whichever Databricks identity is chosen for that purpose.

​
How to create the service principal and OAuth application

  1. Create a service principal in the Databricks account console. See Manage service principals.
  2. Grant the service principal USE CATALOG, USE SCHEMA, and SELECT on the target catalog, schema, and tables. See Manage privileges in Unity Catalog.
  3. Create an OAuth secret for the service principal. This step yields the client ID and the client secret. See Authorize service principal access to Databricks with OAuth.
  4. Supply the client ID as OAuth2ClientId and the client secret as OAuth2Secret, as described in How to authenticate with OAuth machine-to-machine.
The service principal also needs access to the compute it queries through. Grant it at least the CAN USE permission on the target SQL warehouse or cluster. See SQL warehouse ACLs. The compute types and the Databricks runtime versions that DirectQuery supports are listed in the compatibility matrix.