8 Commits
Author SHA1 Message Date
nprasad2077 a85a931133 Merge pull request 'feat(ui): responsive sidebar and full-width navigation cards' (#8) from dev into main
test / test (push) Successful in 7s
Reviewed-on: #8
2026-09-18 00:27:15 +00:00
nprasad2077 a87aca36b8 Merge pull request 'complete' (#7) from dev into main
test / test (push) Successful in 8s
Reviewed-on: #7
2026-09-18 00:16:05 +00:00
nprasad2077 3c38855045 Merge pull request 'complete' (#6) from dev into main
test / test (push) Successful in 8s
Reviewed-on: #6
2026-09-18 00:03:21 +00:00
nprasad2077 0cf9e7d8ce Merge pull request 'self host' (#5) from dev into main
test / test (push) Successful in 6s
Reviewed-on: #5
2026-09-17 23:30:28 +00:00
nprasad2077 717384ebfc Merge pull request 'fix(ci): install uv via shell installer in Gitea runner' (#4) from dev into main
test / test (push) Successful in 6s
Reviewed-on: #4
2026-09-17 22:43:21 +00:00
nprasad2077 22346b5748 Merge pull request 'complete app' (#3) from dev into main
test / test (push) Successful in 6s
Reviewed-on: #3
2026-09-17 22:40:14 +00:00
nprasad2077 4356b9b672 Merge pull request 'test actions' (#2) from dev into main
test / test (push) Successful in 5s
Reviewed-on: #2
2026-09-17 21:17:32 +00:00
nprasad2077 ba5d747371 Merge pull request 'init' (#1) from dev into main
Reviewed-on: #1
2026-09-17 20:07:01 +00:00
4 changed files with 12 additions and 206 deletions

No files matched your search

Vendored
BIN
View File
Binary file not shown.
+1 -7
View File
@@ -47,12 +47,6 @@ jobs:
if: env.COOLIFY_WEBHOOK_URL != '' if: env.COOLIFY_WEBHOOK_URL != ''
env: env:
COOLIFY_WEBHOOK_URL: ${{ secrets.COOLIFY_WEBHOOK_URL }} COOLIFY_WEBHOOK_URL: ${{ secrets.COOLIFY_WEBHOOK_URL }}
COOLIFY_API_TOKEN: ${{ secrets.COOLIFY_API_TOKEN }}
run: | run: |
echo "==> Triggering Coolify deployment webhook..." echo "==> Triggering Coolify deployment webhook..."
if [ -n "${COOLIFY_API_TOKEN:-}" ]; then curl -fsSL -X POST "$COOLIFY_WEBHOOK_URL" || echo "Webhook trigger skipped or failed"
curl -fsSL -X POST -H "Authorization: Bearer $COOLIFY_API_TOKEN" "$COOLIFY_WEBHOOK_URL" || echo "Webhook trigger failed"
else
curl -fsSL -X POST "$COOLIFY_WEBHOOK_URL" || echo "Webhook trigger failed"
fi
+6 -71
View File
@@ -1,75 +1,10 @@
name: CI Health Check & Quality Gate # .gitea/workflows/test.yml
name: test
on: on: [push]
push:
branches:
- main
- dev
pull_request:
workflow_dispatch:
jobs: jobs:
healthcheck: test:
runs-on: ubuntu-latest runs-on: ubuntu-latest
name: Code, dbt & Live App Health Check
steps: steps:
- name: Checkout Code - uses: actions/checkout@v4
uses: actions/checkout@v4 - run: node --version
- name: Install System Dependencies
run: |
apt-get update && apt-get install -y curl unzip git ca-certificates
- name: Install uv
run: |
curl -LsSf https://astral.sh/uv/install.sh | sh
echo "$HOME/.local/bin" >> $GITHUB_PATH
echo "/root/.local/bin" >> $GITHUB_PATH
- name: Install Project Dependencies
run: |
export PATH="$HOME/.local/bin:/root/.local/bin:$PATH"
uv sync
- name: Python Syntax & Compilation Verification
run: |
export PATH="$HOME/.local/bin:/root/.local/bin:$PATH"
echo "==> Validating Python scripts compilation..."
uv run python -m py_compile streamlit_app/app.py scripts/db_storage.py
echo "✓ Python scripts compiled without errors"
- name: dbt Manifest & Model Validation
run: |
export PATH="$HOME/.local/bin:/root/.local/bin:$PATH"
echo "==> Validating dbt package dependencies..."
uv run dbt deps --project-dir dbt_nba --profiles-dir dbt_nba
echo "==> Validating dbt models, seeds, and tests compilation..."
uv run dbt parse --project-dir dbt_nba --profiles-dir dbt_nba
echo "✓ All dbt models, seeds, and macros validated"
- name: Cloudflare R2 Storage Probe (Optional if secrets present)
env:
S3_ENDPOINT_URL: ${{ secrets.S3_ENDPOINT_URL }}
S3_BUCKET_NAME: ${{ secrets.S3_BUCKET_NAME }}
S3_ACCESS_KEY_ID: ${{ secrets.S3_ACCESS_KEY_ID }}
S3_SECRET_ACCESS_KEY: ${{ secrets.S3_SECRET_ACCESS_KEY }}
S3_REGION: ${{ secrets.S3_REGION || 'auto' }}
S3_DB_KEY: ${{ secrets.S3_DB_KEY || 'dbt_nba.duckdb' }}
run: |
export PATH="$HOME/.local/bin:/root/.local/bin:$PATH"
if [ -n "${S3_ENDPOINT_URL:-}" ]; then
echo "==> Probing Cloudflare R2 storage status..."
uv run python scripts/db_storage.py status
else
echo "==> S3 secrets not configured for this step; skipping storage probe."
fi
- name: Live Production App Health Probe
run: |
echo "==> Probing production endpoint https://streamlit.turbo-data.com/_stcore/health..."
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://streamlit.turbo-data.com/_stcore/health || echo "000")
if [ "$HTTP_STATUS" = "200" ]; then
echo "✓ Production Streamlit dashboard is healthy (HTTP $HTTP_STATUS)"
else
echo "⚠️ Production health check returned HTTP $HTTP_STATUS (may still be starting or initializing)"
fi
+5 -128
View File
@@ -1,132 +1,9 @@
# 🏀 NBA Analytics — Streamlit & dbt Data Warehouse # Streamlit NBA
[![NBA Data Pipeline](https://git.turbo-data.com/nprasad2077/Streamlit_NBA/actions/workflows/pipeline.yaml/badge.svg?branch=main)](https://git.turbo-data.com/nprasad2077/Streamlit_NBA/actions?workflow=pipeline.yaml) NBA Streamlit Analytics
[![Live Dashboard](https://img.shields.io/badge/Live%20Dashboard-streamlit.turbo--data.com-FF6B00?logo=streamlit)](https://streamlit.turbo-data.com)
A modern NBA analytics data platform powered by **dbt**, **DuckDB**, and **Streamlit**. Transforms raw basketball event data into a dimensional star schema (32,000+ games, 480,000+ shots from 2002 to 2026), persisting artifacts to **Cloudflare R2 Object Storage** with automated **Gitea Actions** CI/CD and self-hosted **Coolify** deployment. Self Hosted Version
--- [Dashboard](https://streamlit.turbo-data.com)
## 🏛️ Architecture Webhook Added.
```mermaid
flowchart LR
PG[(Postgres Source)] -->|scripts/extract.sql| LOCAL_DB[(DuckDB 223 MB)]
LOCAL_DB -->|dbt build| MARTS[Marts Star Schema]
MARTS -->|db_storage.py upload| R2[(Cloudflare R2 Storage)]
R2 -->|Smart Sync| COOLIFY[Coolify: streamlit.turbo-data.com]
R2 -->|run_app.sh| LOCAL_APP[Local Streamlit Dashboard]
```
### Medallion Data Modeling
| Layer | Models | Description |
|---|---|---|
| **Staging** | `stg_games`, `stg_line_scores`, `stg_player_game_basic_stats`, `stg_player_game_adv_stats`, `stg_player_shot_charts`, `stg_team_game_basic_stats`, `stg_team_game_adv_stats`, `stg_season_thresholds`, `stg_team_season_thresholds` | Clean raw source data, compute seasonal percentile thresholds, type casting |
| **Intermediate** | `int_games_enriched`, `int_player_performance`, `int_team_performance`, `int_player_shots_enriched` | Entity resolution across team history maps, shot chart & free throw unification |
| **Dimensions** | `dim_teams`, `dim_players`, `dim_seasons`, `dim_dates`, `dim_arenas`, `dim_shot_zones`, `dim_player_game_archetypes` | Conformed entity dimensions |
| **Facts** | `fct_game_results`, `fct_team_game_stats`, `fct_player_game_stats`, `fct_quarter_scoring`, `fct_player_shots`, `fct_player_game_shooting` | High-performance analytical fact tables with surrogate keys |
---
## 🚀 Quickstart & Local Setup
### 1. Prerequisites
- [uv](https://docs.astral.sh/uv/) (Python package manager)
- [DuckDB CLI](https://duckdb.org/docs/installation/) (v1.3.0+)
### 2. Clone and Install Dependencies
```bash
git clone https://git.turbo-data.com/nprasad2077/Streamlit_NBA.git
cd Streamlit_NBA
# Install Python 3.12 and dependencies into .venv
uv sync
```
### 3. Configure Environment Variables
Copy the template and fill in your credentials:
```bash
cp .env.example .env
```
```ini
# Postgres Source Database (for ETL extraction)
POSTGRES_URL=postgresql://user:password@host:5432/nba
# Cloudflare R2 / S3 Object Storage
S3_ENDPOINT_URL=https://<ACCOUNT_ID>.r2.cloudflarestorage.com
S3_BUCKET_NAME=dbt-duckdb
S3_ACCESS_KEY_ID=your_r2_access_key
S3_SECRET_ACCESS_KEY=your_r2_secret_key
S3_REGION=auto
S3_DB_KEY=dbt_nba.duckdb
```
### 4. Launch the Streamlit Dashboard
```bash
./scripts/run_app.sh
```
*The launcher automatically verifies and downloads the latest DuckDB database from Cloudflare R2 if it is not already present locally.*
---
## 🗄️ Database Storage Management
Manage your analytical DuckDB file via the custom storage utility:
```bash
# Check remote R2 object vs local file status
uv run python scripts/db_storage.py status
# Download latest database (skips if ETag/size already matches)
uv run python scripts/db_storage.py download
# Force re-download from Cloudflare R2
uv run python scripts/db_storage.py download --force
# Upload local database to Cloudflare R2
uv run python scripts/db_storage.py upload
```
---
## ⚙️ Running the ETL Pipeline
To execute the complete ETL process locally (Extract from Postgres $\rightarrow$ Run dbt transformations $\rightarrow$ Upload artifact to Cloudflare R2):
```bash
./scripts/pipeline.sh
```
---
## 🛠️ dbt Development Commands
Run dbt models and tests directly using `uv`:
```bash
# Install dbt packages (dbt_utils)
uv run dbt deps
# Parse and validate project manifest
uv run dbt parse
# Build all models, seeds, and tests
uv run dbt build
# Build specific layer by tag
uv run dbt build --select "tag:staging"
uv run dbt build --select "tag:intermediate"
uv run dbt build --select "tag:marts"
# Run tests only
uv run dbt test
```
---
## 🚢 CI/CD & Deployment
- **Gitea Actions (`.gitea/workflows/pipeline.yaml`)**: Runs scheduled daily ETL builds (06:00 UTC) on a custom `ubuntu-latest` runner and updates Cloudflare R2.
- **Coolify Self-Hosting ([streamlit.turbo-data.com](https://streamlit.turbo-data.com))**: Production deployment built via Dockerfile with automated Let's Encrypt SSL and persistent storage caching.