diff --git a/README.md b/README.md index d1d50a5..e9b2c4f 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,132 @@ -# Streamlit NBA +# 🏀 NBA Analytics — Streamlit & dbt Data Warehouse -NBA Streamlit Analytics +[![NBA Data Pipeline](https://git.turbo-data.com/nprasad2077/Streamlit_NBA/actions/workflows/pipeline.yaml/badge.svg?branch=main)](https://git.turbo-data.com/nprasad2077/Streamlit_NBA/actions?workflow=pipeline.yaml) +[![Live Dashboard](https://img.shields.io/badge/Live%20Dashboard-streamlit.turbo--data.com-FF6B00?logo=streamlit)](https://streamlit.turbo-data.com) -Self Hosted Version +A modern NBA analytics data platform powered by **dbt**, **DuckDB**, and **Streamlit**. Transforms raw basketball event data into a dimensional star schema (32,000+ games, 480,000+ shots from 2002 to 2026), persisting artifacts to **Cloudflare R2 Object Storage** with automated **Gitea Actions** CI/CD and self-hosted **Coolify** deployment. -[Dashboard](https://streamlit.turbo-data.com) +--- -Webhook Added. \ No newline at end of file +## 🏛️ Architecture + +```mermaid +flowchart LR + PG[(Postgres Source)] -->|scripts/extract.sql| LOCAL_DB[(DuckDB 223 MB)] + LOCAL_DB -->|dbt build| MARTS[Marts Star Schema] + MARTS -->|db_storage.py upload| R2[(Cloudflare R2 Storage)] + R2 -->|Smart Sync| COOLIFY[Coolify: streamlit.turbo-data.com] + R2 -->|run_app.sh| LOCAL_APP[Local Streamlit Dashboard] +``` + +### Medallion Data Modeling + +| Layer | Models | Description | +|---|---|---| +| **Staging** | `stg_games`, `stg_line_scores`, `stg_player_game_basic_stats`, `stg_player_game_adv_stats`, `stg_player_shot_charts`, `stg_team_game_basic_stats`, `stg_team_game_adv_stats`, `stg_season_thresholds`, `stg_team_season_thresholds` | Clean raw source data, compute seasonal percentile thresholds, type casting | +| **Intermediate** | `int_games_enriched`, `int_player_performance`, `int_team_performance`, `int_player_shots_enriched` | Entity resolution across team history maps, shot chart & free throw unification | +| **Dimensions** | `dim_teams`, `dim_players`, `dim_seasons`, `dim_dates`, `dim_arenas`, `dim_shot_zones`, `dim_player_game_archetypes` | Conformed entity dimensions | +| **Facts** | `fct_game_results`, `fct_team_game_stats`, `fct_player_game_stats`, `fct_quarter_scoring`, `fct_player_shots`, `fct_player_game_shooting` | High-performance analytical fact tables with surrogate keys | + +--- + +## 🚀 Quickstart & Local Setup + +### 1. Prerequisites +- [uv](https://docs.astral.sh/uv/) (Python package manager) +- [DuckDB CLI](https://duckdb.org/docs/installation/) (v1.3.0+) + +### 2. Clone and Install Dependencies +```bash +git clone https://git.turbo-data.com/nprasad2077/Streamlit_NBA.git +cd Streamlit_NBA + +# Install Python 3.12 and dependencies into .venv +uv sync +``` + +### 3. Configure Environment Variables +Copy the template and fill in your credentials: +```bash +cp .env.example .env +``` + +```ini +# Postgres Source Database (for ETL extraction) +POSTGRES_URL=postgresql://user:password@host:5432/nba + +# Cloudflare R2 / S3 Object Storage +S3_ENDPOINT_URL=https://.r2.cloudflarestorage.com +S3_BUCKET_NAME=dbt-duckdb +S3_ACCESS_KEY_ID=your_r2_access_key +S3_SECRET_ACCESS_KEY=your_r2_secret_key +S3_REGION=auto +S3_DB_KEY=dbt_nba.duckdb +``` + +### 4. Launch the Streamlit Dashboard +```bash +./scripts/run_app.sh +``` +*The launcher automatically verifies and downloads the latest DuckDB database from Cloudflare R2 if it is not already present locally.* + +--- + +## 🗄️ Database Storage Management + +Manage your analytical DuckDB file via the custom storage utility: + +```bash +# Check remote R2 object vs local file status +uv run python scripts/db_storage.py status + +# Download latest database (skips if ETag/size already matches) +uv run python scripts/db_storage.py download + +# Force re-download from Cloudflare R2 +uv run python scripts/db_storage.py download --force + +# Upload local database to Cloudflare R2 +uv run python scripts/db_storage.py upload +``` + +--- + +## ⚙️ Running the ETL Pipeline + +To execute the complete ETL process locally (Extract from Postgres $\rightarrow$ Run dbt transformations $\rightarrow$ Upload artifact to Cloudflare R2): + +```bash +./scripts/pipeline.sh +``` + +--- + +## 🛠️ dbt Development Commands + +Run dbt models and tests directly using `uv`: + +```bash +# Install dbt packages (dbt_utils) +uv run dbt deps + +# Parse and validate project manifest +uv run dbt parse + +# Build all models, seeds, and tests +uv run dbt build + +# Build specific layer by tag +uv run dbt build --select "tag:staging" +uv run dbt build --select "tag:intermediate" +uv run dbt build --select "tag:marts" + +# Run tests only +uv run dbt test +``` + +--- + +## 🚢 CI/CD & Deployment + +- **Gitea Actions (`.gitea/workflows/pipeline.yaml`)**: Runs scheduled daily ETL builds (06:00 UTC) on a custom `ubuntu-latest` runner and updates Cloudflare R2. +- **Coolify Self-Hosting ([streamlit.turbo-data.com](https://streamlit.turbo-data.com))**: Production deployment built via Dockerfile with automated Let's Encrypt SSL and persistent storage caching. \ No newline at end of file