data_dynamos_exercise

There are two ways to setup and run the code in this repository:

1. Setup Directly on the Machine: Install all the Software directly on your machine and run the code.
2. Setup Using Docker: Build a Docker image containing all the Software already and run the code inside Docker container.

Setup Directly on the Machine

Prerequisites

• Git
• Git LFS — the seed CSVs in data_ingestion/seeds are stored via Git LFS. Without it, those files stay as pointer stubs and dbt seed fails. Install with brew install git-lfs (macOS) or your OS package manager, then run git lfs install once.
• Poetry with Python 3.11 (see .python-version / pyproject.toml)
• Google Cloud SDK (gcloud), authenticated to the training GCP project
• Access to a BigQuery dataset you can read/write in the project (ask your training admin if you don't have one yet — e.g. a personal dataset named like db_<your_name>)

Navigate to the directory where you want to clone the project. For example:

cd path/to/your/directory

Clone the repository and pull the LFS-tracked seed data:

git clone https://github.com/Data-Dynamos/data_dynamos_exercise.git
cd data_dynamos_exercise
git lfs pull

Authenticate with GCP

dbt and the Streamlit app both connect to BigQuery using your local Application Default Credentials (ADC):

gcloud auth login
gcloud config set project <your-gcp-project-id>
gcloud auth application-default login

Install dependencies

poetry install

Set required environment variables

Both dbt projects (data_ingestion, data_transformation) and the Streamlit app (data_visualization) read the target BigQuery project/dataset from environment variables — nothing works without these set:

Variable	Used by	Purpose
GCP_PROJECT_ID	data_ingestion/profiles.yml, data_transformation/profiles.yml	BigQuery project dbt connects to
GCP_DATASET	data_ingestion/profiles.yml, data_transformation/profiles.yml, data_ingestion/models/sources.yml, data_transformation/models/sources.yml	The BigQuery dataset all seeds/models/dashboard queries read and write — use the dataset you have access to (e.g. your personal dataset)
BIGQUERY_PROJECT	data_visualization/utils/session.py	BigQuery project for the Streamlit app's bigquery.Client (defaults to dai-gcp-training-4e58 if unset)
BIGQUERY_LOCATION	data_visualization/utils/session.py	BigQuery job location (defaults to asia-south1 if unset)

Export them in your shell before running any dbt or Streamlit commands (add to ~/.zshrc/~/.bashrc to persist across sessions):

export GCP_PROJECT_ID=<your-gcp-project-id>
export GCP_DATASET=<your-bigquery-dataset>   # e.g. db_jane_doe_thoughtworks_com
export BIGQUERY_PROJECT=<your-gcp-project-id>
export BIGQUERY_LOCATION=asia-south1

Run the dbt projects

Always invoke dbt via poetry run dbt ... (or activate the poetry venv first) rather than a system-wide dbt binary — a different globally installed dbt version can silently require extra IAM roles or fail in confusing ways.

cd data_ingestion
poetry run dbt deps
poetry run dbt seed
poetry run dbt run

cd ../data_transformation
poetry run dbt deps
poetry run dbt run
poetry run dbt test

data_ingestion is provided as-is and should run end-to-end without any changes. data_transformation and data_visualization contain the exercises — look for TODO markers and the exercise docs linked in each file.

To start streamlit instance - run

./scripts/streamlit_bigquery_startup.sh

Setup Using Docker

Prerequisites

• Brew (Use homebrew for mac package manager - https://brew.sh/)
• Docker (Use colima for docker - https://github.com/abiosoft/colima)
• Once Docker is installed, start the docker instance:

colima start
docker ps

• Google Cloud SDK (gcloud) on your host, authenticated to the GCP project

Build the image

docker build -t data_dynamos_exercise .

Usage

Note: The code will be mounted inside the container so whatever changes you make in your local machine will be synced to the container.

Both workflows below need BigQuery access from inside the container. Authenticate once on your host and reuse those credentials by bind-mounting ~/.config/gcloud:

gcloud auth application-default login

They also need the same environment variables described in Set required environment variables above, passed into the container with -e:

-e GCP_PROJECT_ID=<your-gcp-project-id> \
-e GCP_DATASET=<your-bigquery-dataset> \
-e BIGQUERY_PROJECT=<your-gcp-project-id> \
-e BIGQUERY_LOCATION=asia-south1

Data Transformations

• Start the container

docker run --rm -it \
  -v $PWD:/opt/data_dynamos_exercise -v ~/.config/gcloud:/root/.config/gcloud \
  -e GCP_PROJECT_ID=<your-gcp-project-id> -e GCP_DATASET=<your-bigquery-dataset> \
  data_dynamos_exercise bash

• Run the code inside the container

cd data_ingestion
dbt debug # should connect to bigquery successfully

cd ../data_transformation
dbt debug # should connect to bigquery successfully

Data Visualizations

• Start the container

docker run -p 8501:8501 --rm -it \
  -v $PWD:/opt/data_dynamos_exercise -v ~/.config/gcloud:/root/.config/gcloud \
  -e GCP_PROJECT_ID=<your-gcp-project-id> -e GCP_DATASET=<your-bigquery-dataset> \
  -e BIGQUERY_PROJECT=<your-gcp-project-id> -e BIGQUERY_LOCATION=asia-south1 \
  data_dynamos_exercise bash

• Run the code inside the container

./scripts/streamlit_bigquery_startup.sh