GCP Big Data Platform (Terraform)
Terraform stack for a big-data environment on Google Cloud: private VPC, Dataproc cluster with Spark and Jupyter, Airflow on GKE, keyless GitHub Actions CI/CD via Workload Identity Federation, security and cost checks, and a Pandas / Polars / DuckDB / Spark benchmark notebook.
Overview
A fork of a university course template (Big Data Technologies) that provisions Spark and Airflow infrastructure on GCP with Terraform. A bootstrap stage creates the project, state bucket, audit logging and budget alerts; a CI/CD bootstrap configures Workload Identity Federation so GitHub Actions deploys without long-lived keys. The main stack contains a VPC with IAP-only SSH, a Dataproc cluster (Spark, Jupyter, preemptible workers), Airflow on GKE Standard installed with Helm, and Artifact Registry. Pull requests run fmt, validate, Checkov, plan and an Infracost estimate; merges to master run semantic-release and apply. A scheduled auto-destroy workflow removes the environment daily to avoid idle cost. Course template and most module code are by the instructors; this fork adds cost-control, security-scanning and benchmarking work.
Key Highlights
- ✓Keyless GitHub Actions to GCP authentication (Workload Identity Federation)
- ✓Pull-request gates: terraform fmt / validate, Checkov, plan, Infracost cost comment
- ✓Dataproc (Spark, Jupyter, preemptible workers) and Airflow on GKE with Helm
- ✓Private VPC with IAP-only SSH; Terraform state in a dedicated bucket
- ✓Scheduled auto-destroy workflow to control cloud spend
- ✓Benchmark of Pandas, Polars, DuckDB and PySpark on 10M rows, local vs Dataproc
Technical Deep Dive
Problem & Goal
Show how to provision a reproducible Spark and Airflow environment as code, deploy it through a secure CI/CD pipeline, and keep cost under control on a student billing account.
Benchmark Results
Synthetic streaming-platform events, 10M rows (~268 MB Parquet). PyArrow-backed Pandas was 2.3-3.3x faster than the default backend; DuckDB was about 6-7x faster than default Pandas; Polars lazy execution pushed 4 of 14 columns and the filter into the Parquet scan. Spark on Dataproc was about 3x slower than local PySpark at this size because scheduling and GCS I/O dominate, so the notebook recommends staying on a single node until data outgrows one machine.
My Role & Scope
Course project done in a group of three. My changes to the template: the auto-destroy workflow, Infracost usage profile, preemptible Dataproc workers and machine-size tuning, Checkov exceptions and workflow permissions, the Airflow DAG and Spark job fixes, and the Phase 2 benchmark notebook including the Dataproc comparison. Infrastructure modules and the base pipelines come from the course template.
Screenshots & Images
