Databricks IAC

Example Infrastructure as Code for managing Databricks workspaces, catalogs, users, and groups using Terraform and Terragrunt.

Overview

This project provides a structured approach to managing Databricks resources across multiple environments (dev, prod) with shared common configurations.

Configuration

Prerequisites

To use this project, create a common.secrets.yaml file in the common/ directory. An example file (common.secrets.yaml.example) is provided as a reference.

Setup

1. Copy the example secrets file:

   cp common/common.secrets.yaml.example common/common.secrets.yaml

2. Update common/common.yaml with your own users and groups configuration.

Architecture

Databricks Infrastructure Configuration

This configuration module sets up a complete Databricks environment with the following components:

Metastore:

• Creates a primary metastore for centralized data governance and management

Catalogs:

• dev: Development catalog for testing and experimentation
• prod: Production catalog for stable, live data and workloads

Serverless Workspaces:

• dev: Workspace for development
• prod: Workspace for production

Access Control:

• Defines user groups for role-based access management
• Assigns users to appropriate groups based on responsibilities
• Configures granular permissions on dev and prod catalogs for each group
• Ensures proper isolation and security between development and production environments

graph TD
  common_metastore["common/metastore"]
  common_users["common/users-groups"]

  dev_catalog["dev/catalog"]
  dev_workspace["dev/workspace"]

  prod_catalog["prod/catalog"]
  prod_workspace["prod/workspace"]

  dev_catalog --> common_metastore
  dev_catalog --> dev_workspace

  dev_workspace --> common_metastore
  dev_workspace --> common_users

  prod_catalog --> common_metastore
  prod_catalog --> prod_workspace

  prod_workspace --> common_metastore
  prod_workspace --> common_users

Loading