Databricks

Introduction

This guide explains how to setup Databricks for use with Cyclr, as well as installing a Databricks Connector.

This guide assumes that you wish to use your own Databricks account whenever a Databricks Connector is installed by providing your own Databricks authentication at the Cyclr Partner-level.

If you instead wish to work with the Databricks accounts of your customers, you should leave the Partner-level values blank so that Cyclr requests them each time a Databricks Connector is installed.

Each of your customers will then need to follow the steps below to setup their own Databricks service principal and obtain a Client ID and Secret. Those details would then be provided to Cyclr when installing a Databricks Connector.

Setup & Authentication

Overview

To use our Databricks Connector, you must create a Databricks account and set up a workspace. For more information on how to do this, go to Databricks Get Started guide.

To use your Databricks account on AWS, you need an existing AWS account.

Once your Databricks workspace has been created, you will receive an email with your workspace URL. From that you can extract your Workspace ID.

If your Workspace URL is https://dbc-sampleId.cloud.databricks.com, your Workspace ID would be this:
dbc-sampleId

For information about online training resources, see Get free Databricks training.

Remote Setup - performed by Cyclr Partner

Create a service principal

To add a service principal to your Databricks account using the Databricks account console:

1. As an account admin, log in to the account console.

2. Click User management on the left panel.

3. On the Service principals tab, click Add service principal.

4. Enter a name for the service principal.

5. Click Add.

6. Optionally, on the Roles tab, turn on Account admin to call Databricks' account-level APIs.

You can now assign your service principal to identity federated workspaces.

1. In the account console sidebar, click Workspaces.

2. Click your workspace name.

3. On the Permissions tab, click Add permissions.

4. Search for and select the service principal, assign the permission level (workspace User or Admin), and click Save.

Assign workspace-level permissions to the Databricks service principal

1. If the admin console for your workspace is not already opened, click your username in the top bar and click Settings.

2. Click on the Identity and access tab.

3. Next to Service principals, click Manage.

4. Click the name of your Databricks service principal to open its settings page.

5. On the Configurations tab, check the box next to each entitlement that you want your Databricks service principal to have for this workspace, and then click Update.

6. On the Permissions tab, grant access to any Databricks users, service principals, and groups that you want to manage and use this Databricks service principal. See Manage roles on a service principal.

Create an OAuth secret for a service principal

Before you can use OAuth to authenticate to Databricks, you must first create an OAuth secret, which can be used to generate OAuth access tokens.

A service principal can have up to 5 OAuth secrets.

To create an OAuth secret for a service principal by using the account console:

1. As an account admin, log in to the account console.

2. Click User management on the left panel.

3. On the Service principals tab, select your service principal.

4. Under OAuth secrets, click Generate secret.

5. Copy the displayed Secret and Client ID, and then click Done.

   The Client ID is the same as the service principal’s application ID.

Your OAuth Secret will only be revealed once during creation.

For more information on how to set OAuth on your Databricks account, go to OAuth machine-to-machine (M2M) authentication

Partner Setup in Cyclr Console

Having obtained a Client ID and a Secret, go into your Cyclr Partner Console:

1. Go to Connectors > Application Connector Library.

2. Use the search box to locate the Databricks Connector entry.

3. Select the Pencil button.

4. Select the Settings tab.

5. Enter the below values:

6. Select Save Changes.