Merge pull request 'README update' (#9) from dev into main

Reviewed-on: #9
2026-09-18 00:41:41 +00:00
parent a85a931133 b76b2719e4
commit 709e14fdac
1 changed files with 128 additions and 5 deletions
@@ -1,9 +1,132 @@
-# Streamlit NBA
+# 🏀 NBA Analytics — Streamlit & dbt Data Warehouse

-NBA Streamlit Analytics
+[![NBA Data Pipeline](https://git.turbo-data.com/nprasad2077/Streamlit_NBA/actions/workflows/pipeline.yaml/badge.svg?branch=main)](https://git.turbo-data.com/nprasad2077/Streamlit_NBA/actions?workflow=pipeline.yaml)
+[![Live Dashboard](https://img.shields.io/badge/Live%20Dashboard-streamlit.turbo--data.com-FF6B00?logo=streamlit)](https://streamlit.turbo-data.com)

-Self Hosted Version
+A modern NBA analytics data platform powered by **dbt**, **DuckDB**, and **Streamlit**. Transforms raw basketball event data into a dimensional star schema (32,000+ games, 480,000+ shots from 2002 to 2026), persisting artifacts to **Cloudflare R2 Object Storage** with automated **Gitea Actions** CI/CD and self-hosted **Coolify** deployment.

-[Dashboard](https://streamlit.turbo-data.com)
+---

-Webhook Added.
+## 🏛️ Architecture
+
+```mermaid
+flowchart LR
+    PG[(Postgres Source)] -->|scripts/extract.sql| LOCAL_DB[(DuckDB 223 MB)]
+    LOCAL_DB -->|dbt build| MARTS[Marts Star Schema]
+    MARTS -->|db_storage.py upload| R2[(Cloudflare R2 Storage)]
+    R2 -->|Smart Sync| COOLIFY[Coolify: streamlit.turbo-data.com]
+    R2 -->|run_app.sh| LOCAL_APP[Local Streamlit Dashboard]
+```
+
+### Medallion Data Modeling
+
+| Layer | Models | Description |
+|---|---|---|
+| **Staging** | `stg_games`, `stg_line_scores`, `stg_player_game_basic_stats`, `stg_player_game_adv_stats`, `stg_player_shot_charts`, `stg_team_game_basic_stats`, `stg_team_game_adv_stats`, `stg_season_thresholds`, `stg_team_season_thresholds` | Clean raw source data, compute seasonal percentile thresholds, type casting |
+| **Intermediate** | `int_games_enriched`, `int_player_performance`, `int_team_performance`, `int_player_shots_enriched` | Entity resolution across team history maps, shot chart & free throw unification |
+| **Dimensions** | `dim_teams`, `dim_players`, `dim_seasons`, `dim_dates`, `dim_arenas`, `dim_shot_zones`, `dim_player_game_archetypes` | Conformed entity dimensions |
+| **Facts** | `fct_game_results`, `fct_team_game_stats`, `fct_player_game_stats`, `fct_quarter_scoring`, `fct_player_shots`, `fct_player_game_shooting` | High-performance analytical fact tables with surrogate keys |
+
+---
+
+## 🚀 Quickstart & Local Setup
+
+### 1. Prerequisites
+- [uv](https://docs.astral.sh/uv/) (Python package manager)
+- [DuckDB CLI](https://duckdb.org/docs/installation/) (v1.3.0+)
+
+### 2. Clone and Install Dependencies
+```bash
+git clone https://git.turbo-data.com/nprasad2077/Streamlit_NBA.git
+cd Streamlit_NBA
+
+# Install Python 3.12 and dependencies into .venv
+uv sync
+```
+
+### 3. Configure Environment Variables
+Copy the template and fill in your credentials:
+```bash
+cp .env.example .env
+```
+
+```ini
+# Postgres Source Database (for ETL extraction)
+POSTGRES_URL=postgresql://user:password@host:5432/nba
+
+# Cloudflare R2 / S3 Object Storage
+S3_ENDPOINT_URL=https://<ACCOUNT_ID>.r2.cloudflarestorage.com
+S3_BUCKET_NAME=dbt-duckdb
+S3_ACCESS_KEY_ID=your_r2_access_key
+S3_SECRET_ACCESS_KEY=your_r2_secret_key
+S3_REGION=auto
+S3_DB_KEY=dbt_nba.duckdb
+```
+
+### 4. Launch the Streamlit Dashboard
+```bash
+./scripts/run_app.sh
+```
+*The launcher automatically verifies and downloads the latest DuckDB database from Cloudflare R2 if it is not already present locally.*
+
+---
+
+## 🗄️ Database Storage Management
+
+Manage your analytical DuckDB file via the custom storage utility:
+
+```bash
+# Check remote R2 object vs local file status
+uv run python scripts/db_storage.py status
+
+# Download latest database (skips if ETag/size already matches)
+uv run python scripts/db_storage.py download
+
+# Force re-download from Cloudflare R2
+uv run python scripts/db_storage.py download --force
+
+# Upload local database to Cloudflare R2
+uv run python scripts/db_storage.py upload
+```
+
+---
+
+## ⚙️ Running the ETL Pipeline
+
+To execute the complete ETL process locally (Extract from Postgres $\rightarrow$ Run dbt transformations $\rightarrow$ Upload artifact to Cloudflare R2):
+
+```bash
+./scripts/pipeline.sh
+```
+
+---
+
+## 🛠️ dbt Development Commands
+
+Run dbt models and tests directly using `uv`:
+
+```bash
+# Install dbt packages (dbt_utils)
+uv run dbt deps
+
+# Parse and validate project manifest
+uv run dbt parse
+
+# Build all models, seeds, and tests
+uv run dbt build
+
+# Build specific layer by tag
+uv run dbt build --select "tag:staging"
+uv run dbt build --select "tag:intermediate"
+uv run dbt build --select "tag:marts"
+
+# Run tests only
+uv run dbt test
+```
+
+---
+
+## 🚢 CI/CD & Deployment
+
+- **Gitea Actions (`.gitea/workflows/pipeline.yaml`)**: Runs scheduled daily ETL builds (06:00 UTC) on a custom `ubuntu-latest` runner and updates Cloudflare R2.
+- **Coolify Self-Hosting ([streamlit.turbo-data.com](https://streamlit.turbo-data.com))**: Production deployment built via Dockerfile with automated Let's Encrypt SSL and persistent storage caching.