Streamlit_NBA/.opencode/context.md

Codebase Context — Streamlit_NBA

Updated: 2026-10-27

Overview

Streamlit_NBA is an end-to-end NBA analytics data warehouse and interactive visualization platform. It extracts raw game, team, and player box score and shot telemetry from PostgreSQL into DuckDB, transforms and models the data through a 3-layer medallion star schema (staging views → intermediate tables → dimension/incremental fact marts) using dbt-duckdb, and serves analytical insights via a 7-page Streamlit application and an Evidence BI report suite. Production DuckDB database artifacts are synced to and from Cloudflare R2 / S3 object storage with automated CI/CD and daily scheduled ETL builds orchestrated through Gitea Actions.

Stack

• Languages: Python 3.12 (pinned in pyproject.toml:4, Dockerfile:1), SQL / Jinja (dbt SQL dialect for DuckDB), JavaScript / Markdown (Evidence BI)
• Data Warehouse & Transformation: dbt (dbt-duckdb, dbt Core 1.12.5, dbt adapter 1.11.0 in pyproject.toml:6), DuckDB (duckdb>=1.3.0 in pyproject.toml:8), dbt-labs/dbt_utils 1.3.1 (dbt_nba/packages.yml:2-3)
• Dashboards & Visualization: Streamlit (streamlit>=1.45.1 in pyproject.toml:7), Plotly (plotly>=6.1.2 in pyproject.toml:9), streamlit-echarts>=0.4.0 (pyproject.toml:10), Evidence BI (@evidence-dev/evidence: ^40.1.8 in dbt_nba/reports/package.json:20)
• Storage & Infrastructure: Cloudflare R2 / S3-compatible Object Storage via boto3>=1.34.0 (pyproject.toml:11, scripts/db_storage.py:33-53), Docker (Dockerfile:1-36)
• Package & Runtime Management: uv (pyproject.toml, uv.lock), npm (dbt_nba/reports/package-lock.json)
• CI/CD: Gitea Actions (.gitea/workflows/pipeline.yaml, .gitea/workflows/test.yml)

Layout

• streamlit_app/: Frontend application code (streamlit_app/app.py:1-1464 containing all 7 analytical views and navigation).
• dbt_nba/: Core dbt transformation project:
  • models/staging/: 10 views cleaning raw extracts and deriving dynamic metric thresholds.
  • models/intermediate/: 4 tables performing entity resolution, team abbreviation conforming, and performance joins.
  • models/marts/dimensions/: 7 dimension tables (arenas, dates, player archetypes, players, seasons, shot zones, teams).
  • models/marts/facts/: 6 incremental fact tables with 30-day idempotent refresh windows.
  • seeds/: Reference CSVs for team and arena name/abbreviation history across seasons (seeds/team_maps.csv, seeds/arena_maps.csv).
  • reports/: Evidence BI frontend project (Node.js/npm) with DuckDB connector.
• scripts/: Operational automation:
  • pipeline.sh: Full extraction from Postgres, dbt rebuild, and R2 artifact upload.
  • extract.sql: DuckDB script attaching to Postgres and pulling source tables into main.*.
  • db_storage.py: CLI manager for R2 database upload, download, and hash/status checking.
  • run_app.sh: Streamlit launcher with auto-download fallback from R2.
• data/: Local storage directory for DuckDB database files (data/DB/dbt_nba.duckdb) and court assets (shot_chart.png, shot_chart.svg).
• .gitea/workflows/: CI/CD definitions for scheduled daily ETL (pipeline.yaml) and quality checks (test.yml).
• docs/: Historical documentation, prompts, and tournament records (docs/init_prompt.md, docs/all_nba_cup_games_history.md).

Architecture

• Raw Ingestion Layer: DuckDB PostgreSQL extension (scripts/extract.sql:5-26) attaches to the live PostgreSQL source (POSTGRES_URL) and copies 7 tables (games, line_scores, player_game_basic_stats, player_game_adv_stats, player_shot_charts, team_game_basic_stats, team_game_adv_stats) into main schema.
• Staging Layer (Views): dbt_nba/dbt_project.yml:22-25 materializes models into main_staging. Includes percentile breakpoints (dbt_nba/models/staging/stg_season_thresholds.sql) and dynamic tiering (dbt_nba/models/staging/stg_player_game_adv_stats_extended.sql:1-40).
• Intermediate Layer (Tables): dbt_nba/dbt_project.yml:27-30 materializes into main_intermediate. Handles temporal team mapping (dbt_nba/models/intermediate/int_games_enriched.sql:15-35) and unification of field goal shots with box-score free throws (dbt_nba/models/intermediate/int_player_shots_enriched.sql:1-50).
• Marts Layer (Star Schema): dbt_nba/dbt_project.yml:32-44 materializes into main_marts:
  • 7 Dimensions: dim_teams, dim_players, dim_seasons, dim_dates, dim_arenas, dim_shot_zones, dim_player_game_archetypes.
  • 6 Facts (Incremental): Surrogate keys generated via dbt_utils.generate_surrogate_key (dbt_nba/models/marts/facts/fct_player_game_stats.sql:10-25), using 30-day lookback windows for idempotency.
• Application & Presentation:
  • streamlit_app/app.py:70-83: get_connection() establishes persistent read-only connection to dbt_nba.duckdb; queries cached via @st.cache_data(ttl=3600).
  • Seven dashboard modules: Overview (app.py:151), Teams (app.py:297), Player Lab (app.py:493), Shot Charts (app.py:769), Head to Head (app.py:1052), Game Trends (app.py:1231), Quarter Analysis (app.py:1364).

Entrypoints & flows

• Daily Automated ETL:
  1. Triggered via Gitea Actions cron 0 6 * * * (.gitea/workflows/pipeline.yaml:4-6).
  2. Runs scripts/pipeline.sh (scripts/pipeline.sh:22-45).
  3. Pulls Postgres source into data/DB/dbt_nba.duckdb via DuckDB CLI (scripts/extract.sql:18-24).
  4. Executes dbt deps and dbt build --full-refresh (scripts/pipeline.sh:33-34).
  5. Copies DB to dbt_nba/reports/sources/nba/dbt_nba.duckdb for Evidence BI (scripts/pipeline.sh:37).
  6. Uploads the finalized DuckDB database file to Cloudflare R2 bucket via scripts/db_storage.py upload (scripts/pipeline.sh:41).
• Local Application Startup:
  1. User/container launches ./scripts/run_app.sh or Docker container entrypoint (Dockerfile:36).
  2. Verifies local DuckDB file; if missing or if --sync flag is provided, downloads latest snapshot from Cloudflare R2 (scripts/run_app.sh:23-38).
  3. Launches Streamlit application (streamlit run streamlit_app/app.py).

Data & integrations

• Source Database: PostgreSQL database (POSTGRES_URL in .env), read-only extraction.
• Analytical Storage: DuckDB (dbt_nba.duckdb) containing 6 schemas: main (raw tables), main_staging (views), main_intermediate (tables), main_marts (star schema), main_seeds (reference CSVs), main_test_results (audit tables).
• Object Storage: Cloudflare R2 (or any S3-compatible store: SeaweedFS, MinIO, Hetzner S3) configured via S3_ENDPOINT_URL, S3_BUCKET_NAME, S3_ACCESS_KEY_ID, S3_SECRET_ACCESS_KEY, S3_REGION, S3_DB_KEY in .env.
• Authentication: Zero authentication on public Streamlit app; API/DB credentials stored in .env and Gitea Action secrets.

Commands

• Install dependencies: uv sync (Python) / npm --prefix dbt_nba/reports install (Evidence BI)
• Compile Python scripts: uv run python -m py_compile streamlit_app/app.py scripts/db_storage.py
• Validate dbt models: uv run dbt parse --project-dir dbt_nba --profiles-dir dbt_nba
• Install dbt dependencies: uv run dbt deps --project-dir dbt_nba --profiles-dir dbt_nba
• Run dbt tests: uv run dbt test --project-dir dbt_nba --profiles-dir dbt_nba
• Run full dbt build: uv run dbt build --project-dir dbt_nba --profiles-dir dbt_nba
• Check R2 storage status: uv run python scripts/db_storage.py status
• Download database from R2: uv run python scripts/db_storage.py download
• Upload database to R2: uv run python scripts/db_storage.py upload
• Run Streamlit app: ./scripts/run_app.sh (or uv run streamlit run streamlit_app/app.py)
• Run full ETL pipeline: ./scripts/pipeline.sh
• Build Docker image: docker build -t streamlit-nba .

Conventions

• Team Abbreviation Conforming: Historical franchise relocations and name changes handled via dbt_nba/seeds/team_maps.csv using start_year/end_year filtering in intermediate SQL models.
• dbt Testing & Severity: Schema tests configured with severity: warn and store_failures: true storing failures into main_test_results (dbt_nba/dbt_project.yml:45-50).
• Branch & Commit Strategy: Git branches main (production deployment) and dev (active development); conventional commit prefixes (feat:, fix:, ci:). Current working branch: update.
• Streamlit Caching: Data queries decorated with @st.cache_data(ttl=3600), database connection with @st.cache_resource (streamlit_app/app.py:70-83).

Testing

• dbt Schema & Data Tests: 38 data tests defined across models in dbt_nba/models/schema.yml and _int_models.yml (testing unique constraints, non-null values, referential integrity). Executed via uv run dbt test --project-dir dbt_nba --profiles-dir dbt_nba.
• CI Quality Gate: .gitea/workflows/test.yml:11-75 runs on every push and pull request to main and dev:
  1. Syntax & bytecode verification (python -m py_compile).
  2. dbt package and manifest validation (dbt parse).
  3. Cloudflare R2 storage reachability probe.
  4. Live production app endpoint healthcheck (https://streamlit.turbo-data.com/_stcore/health).

Open questions

• Postgres Source Availability for Local Dev: Local execution of scripts/pipeline.sh requires live POSTGRES_URL; developers without Postgres access rely on pre-built DuckDB downloads from Cloudflare R2 via db_storage.py download.
• Evidence BI Production Deployment: Evidence project in dbt_nba/reports has build scripts (npm run build) and queries local DuckDB, but whether it is independently hosted alongside Streamlit or used strictly for static export is not defined in Gitea workflows.

Starting points

• Adding or modifying a Streamlit chart/page: Start in streamlit_app/app.py around lines 150-1464; check available columns in main_marts or main_intermediate.
• Adding or updating a dbt metric / table: Start in dbt_nba/models/; update corresponding .yml test definitions in dbt_nba/models/schema.yml or _int_models.yml.
• Fixing team or franchise mappings: Edit dbt_nba/seeds/team_maps.csv or arena_maps.csv, then run dbt seed.
• Adjusting ETL pipeline or object storage: Inspect scripts/pipeline.sh and scripts/db_storage.py.
• Modifying CI/CD workflows: Inspect .gitea/workflows/pipeline.yaml (daily pipeline) and test.yml (health check).