Merge pull request 'pipeline' (#14) from update into main

Reviewed-on: #14

.gitea/workflows/pipeline.yaml

@@ -2,7 +2,7 @@ name: NBA Data Pipeline
 
 on:
   schedule:
-    - cron: '0 6 * * *'
+    - cron: '0 6 40 * *'
   workflow_dispatch:
 
 jobs:
@@ -22,12 +22,10 @@ jobs:
           echo "$HOME/.local/bin" >> $GITHUB_PATH
           echo "/root/.local/bin" >> $GITHUB_PATH
 
-      - name: Install DuckDB CLI
+      - name: Install Project Dependencies
         run: |
-          curl -fsSL https://github.com/duckdb/duckdb/releases/latest/download/duckdb_cli-linux-amd64.zip -o duckdb.zip
+          export PATH="$HOME/.local/bin:/root/.local/bin:$PATH"
-          unzip -o duckdb.zip
+          uv sync
-          mv duckdb /usr/local/bin/
-          rm -f duckdb.zip
 
       - name: Run ETL Pipeline & Upload to Cloudflare R2
         env:

.gitignore

@@ -197,3 +197,5 @@ AGENTS.md
 .claude
 
 .env.old
+
+.opencode

.opencode/context.md

@@ -0,0 +1,113 @@
+# Codebase Context — Streamlit_NBA
+
+Updated: 2026-10-28
+
+## Overview
+
+Streamlit_NBA is an end-to-end NBA analytics data warehouse and interactive visualization platform. It extracts raw game, team, and player box score and shot telemetry from PostgreSQL into DuckDB, transforms and models the data through a 3-layer medallion star schema (staging views → intermediate tables → dimension/incremental fact marts) using dbt-duckdb, and serves analytical insights via a 7-page Streamlit application and an Evidence BI report suite. Production DuckDB database artifacts are synced to and from Cloudflare R2 / S3 object storage with automated CI/CD and daily scheduled ETL builds orchestrated through Gitea Actions.
+
+## Stack
+
+- **Languages**: Python 3.12 (pinned in `pyproject.toml:4`, `Dockerfile:1`), SQL / Jinja (dbt SQL dialect for DuckDB), JavaScript / Markdown (Evidence BI)
+- **Data Warehouse & Transformation**: dbt (`dbt-duckdb`, dbt Core 1.12.5, dbt adapter 1.11.0 in `pyproject.toml:6`), DuckDB (`duckdb>=1.3.0` in `pyproject.toml:8`), `dbt-labs/dbt_utils` 1.3.1 (`dbt_nba/packages.yml:2-3`)
+- **Dashboards & Visualization**: Streamlit (`streamlit>=1.45.1` in `pyproject.toml:7`), Plotly (`plotly>=6.1.2` in `pyproject.toml:9`), `streamlit-echarts>=0.4.0` (`pyproject.toml:10`), Evidence BI (`@evidence-dev/evidence: ^40.1.8` in `dbt_nba/reports/package.json:20`)
+- **Storage & Infrastructure**: Cloudflare R2 / S3-compatible Object Storage via `boto3>=1.34.0` (`pyproject.toml:11`, `scripts/db_storage.py:33-53`), Docker (`Dockerfile:1-36`)
+- **Package & Runtime Management**: `uv` (`pyproject.toml`, `uv.lock`), npm (`dbt_nba/reports/package-lock.json`)
+- **CI/CD**: Gitea Actions (`.gitea/workflows/pipeline.yaml`, `.gitea/workflows/test.yml`)
+
+## Layout
+
+- `streamlit_app/`: Frontend application code (`streamlit_app/app.py:1-1464` containing all 7 analytical views and navigation).
+- `dbt_nba/`: Core dbt transformation project:
+  - `models/staging/`: 10 views cleaning raw extracts and deriving dynamic metric thresholds.
+  - `models/intermediate/`: 4 tables performing entity resolution, team abbreviation conforming, and performance joins.
+  - `models/marts/dimensions/`: 7 dimension tables (arenas, dates, player archetypes, players, seasons, shot zones, teams).
+  - `models/marts/facts/`: 6 incremental fact tables with 30-day idempotent refresh windows.
+  - `seeds/`: Reference CSVs for team and arena name/abbreviation history across seasons (`seeds/team_maps.csv`, `seeds/arena_maps.csv`).
+  - `reports/`: Evidence BI frontend project (Node.js/npm) with DuckDB connector.
+- `scripts/`: Operational automation:
+  - `pipeline.sh`: Full extraction from Postgres, dbt rebuild, and R2 artifact upload.
+  - `extract.sql`: DuckDB script attaching to Postgres and pulling source tables into `main.*`.
+  - `extract_data.py`: Python script orchestrating SQL extraction from Postgres into DuckDB.
+  - `db_storage.py`: CLI manager for R2 database upload, download, and hash/status checking.
+  - `run_app.sh`: Streamlit launcher with auto-download fallback from R2.
+- `data/`: Local storage directory for DuckDB database files (`data/DB/dbt_nba.duckdb`) and court assets (`shot_chart.png`, `shot_chart.svg`).
+- `.gitea/workflows/`: CI/CD definitions for scheduled daily ETL (`pipeline.yaml`) and quality checks (`test.yml`).
+- `docs/`: Historical documentation, prompts, and tournament records (`docs/init_prompt.md`, `docs/all_nba_cup_games_history.md`).
+
+## Architecture
+
+- **Raw Ingestion Layer**: DuckDB PostgreSQL extension (`scripts/extract.sql:5-26`, invoked via `scripts/extract_data.py`) attaches to the live PostgreSQL source (`POSTGRES_URL`) and copies 7 tables (`games`, `line_scores`, `player_game_basic_stats`, `player_game_adv_stats`, `player_shot_charts`, `team_game_basic_stats`, `team_game_adv_stats`) into `main` schema.
+- **Staging Layer (Views)**: `dbt_nba/dbt_project.yml:22-25` materializes models into `main_staging`. Includes percentile breakpoints (`dbt_nba/models/staging/stg_season_thresholds.sql`) and dynamic tiering (`dbt_nba/models/staging/stg_player_game_adv_stats_extended.sql:1-40`).
+- **Intermediate Layer (Tables)**: `dbt_nba/dbt_project.yml:27-30` materializes into `main_intermediate`. Handles temporal team mapping (`dbt_nba/models/intermediate/int_games_enriched.sql:15-35`) and unification of field goal shots with box-score free throws (`dbt_nba/models/intermediate/int_player_shots_enriched.sql:1-50`).
+- **Marts Layer (Star Schema)**: `dbt_nba/dbt_project.yml:32-44` materializes into `main_marts`:
+  - 7 Dimensions: `dim_teams`, `dim_players`, `dim_seasons`, `dim_dates`, `dim_arenas`, `dim_shot_zones`, `dim_player_game_archetypes`.
+  - 6 Facts (Incremental): Surrogate keys generated via `dbt_utils.generate_surrogate_key` (`dbt_nba/models/marts/facts/fct_player_game_stats.sql:10-25`), using 30-day lookback windows for idempotency.
+- **Application & Presentation**:
+  - `streamlit_app/app.py:70-83`: `get_connection()` establishes persistent read-only DuckDB connection using DuckDB's Python API, falling back between local dev path (`dbt_nba/reports/sources/nba/dbt_nba.duckdb`) and data storage target (`data/DB/dbt_nba.duckdb`).
+  - Evidence BI queries `sources/nba/dbt_nba.duckdb` directly through SQL source models.
+
+## Entrypoints & flows
+
+- **Daily Automated ETL Pipeline** (`.gitea/workflows/pipeline.yaml:26-38`, `scripts/pipeline.sh:1-44`):
+  1. Triggered on schedule (`cron: '0 6 * * *'`) or manual dispatch.
+  2. Runs `scripts/extract_data.py` executing `scripts/extract.sql` to pull raw Postgres tables into local `data/DB/dbt_nba.duckdb`.
+  3. Executes `uv run dbt deps` and `uv run dbt build --full-refresh` (`scripts/pipeline.sh:34`).
+  4. Runs 38 data and referential integrity tests in dbt.
+  5. Copies refreshed DuckDB file to Evidence BI path (`scripts/pipeline.sh:37`).
+  6. Uploads the finalized DuckDB database file to Cloudflare R2 bucket via `scripts/db_storage.py upload` (`scripts/pipeline.sh:41`).
+- **Local Application Startup**:
+  1. User/container launches `./scripts/run_app.sh` or Docker container entrypoint (`Dockerfile:36`).
+  2. Verifies local DuckDB file; if missing or if `--sync` flag is provided, downloads latest snapshot from Cloudflare R2 (`scripts/run_app.sh:23-38`).
+  3. Launches Streamlit application (`streamlit run streamlit_app/app.py`).
+
+## Data & integrations
+
+- **Source Database**: PostgreSQL database (`POSTGRES_URL` in `.env`), read-only extraction.
+- **Analytical Storage**: DuckDB (`dbt_nba.duckdb`) containing 6 schemas: `main` (raw tables), `main_staging` (views), `main_intermediate` (tables), `main_marts` (star schema), `main_seeds` (reference CSVs), `main_test_results` (audit tables).
+- **Object Storage**: Cloudflare R2 (or any S3-compatible store: SeaweedFS, MinIO, Hetzner S3) configured via `S3_ENDPOINT_URL`, `S3_BUCKET_NAME`, `S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY`, `S3_REGION`, `S3_DB_KEY` in `.env`.
+- **Authentication**: Zero authentication on public Streamlit app; API/DB credentials stored in `.env` and Gitea Action secrets.
+
+## Commands
+
+- **Install dependencies**: `uv sync` (Python) / `npm --prefix dbt_nba/reports install` (Evidence BI)
+- **Compile Python scripts**: `uv run python -m py_compile streamlit_app/app.py scripts/db_storage.py scripts/extract_data.py`
+- **Validate dbt models**: `uv run dbt parse --project-dir dbt_nba --profiles-dir dbt_nba`
+- **Install dbt dependencies**: `uv run dbt deps --project-dir dbt_nba --profiles-dir dbt_nba`
+- **Run dbt tests**: `uv run dbt test --project-dir dbt_nba --profiles-dir dbt_nba`
+- **Run full dbt build**: `uv run dbt build --project-dir dbt_nba --profiles-dir dbt_nba`
+- **Check R2 storage status**: `uv run python scripts/db_storage.py status`
+- **Download database from R2**: `uv run python scripts/db_storage.py download`
+- **Upload database to R2**: `uv run python scripts/db_storage.py upload`
+- **Run Streamlit app**: `./scripts/run_app.sh` (or `uv run streamlit run streamlit_app/app.py`)
+- **Run full ETL pipeline**: `./scripts/pipeline.sh`
+- **Build Docker image**: `docker build -t streamlit-nba .`
+
+## Conventions
+
+- **Team Abbreviation Conforming**: Historical franchise relocations and name changes handled via `dbt_nba/seeds/team_maps.csv` using `start_year`/`end_year` filtering in intermediate SQL models.
+- **dbt Testing & Severity**: Schema tests configured with `severity: warn` and `store_failures: true` storing failures into `main_test_results` (`dbt_nba/dbt_project.yml:45-50`).
+- **Branch & Commit Strategy**: Git branches `main` (production deployment), `dev` (active development), and working branch `update`; conventional commit prefixes (`feat:`, `fix:`, `ci:`).
+- **Streamlit Caching**: Data queries decorated with `@st.cache_data(ttl=3600)`, database connection with `@st.cache_resource` (`streamlit_app/app.py:70-83`).
+
+## Testing
+
+- **dbt Schema & Data Tests**: 38 data tests defined across models in `dbt_nba/models/schema.yml` and `_int_models.yml` (testing unique constraints, non-null values, referential integrity). Executed via `uv run dbt test --project-dir dbt_nba --profiles-dir dbt_nba`.
+- **CI Quality Gate**: `.gitea/workflows/test.yml:11-75` runs on every push and pull request to `main` and `dev`:
+  1. Syntax & bytecode verification (`python -m py_compile`).
+  2. dbt package and manifest validation (`dbt parse`).
+  3. Cloudflare R2 storage reachability probe.
+  4. Live production app endpoint healthcheck (`https://streamlit.turbo-data.com/_stcore/health`).
+
+## Open questions
+
+- **Postgres Source Availability for Local Dev**: Local execution of `scripts/pipeline.sh` requires live `POSTGRES_URL`; developers without Postgres access rely on pre-built DuckDB downloads from Cloudflare R2 via `db_storage.py download`.
+- **Evidence BI Production Deployment**: Evidence project in `dbt_nba/reports` has build scripts (`npm run build`) and queries local DuckDB, but whether it is independently hosted alongside Streamlit or used strictly for static export is not defined in Gitea workflows.
+
+## Starting points
+
+- **Adding or modifying a Streamlit chart/page**: Start in `streamlit_app/app.py` around lines 150-1464; check available columns in `main_marts` or `main_intermediate`.
+- **Adding or updating a dbt metric / table**: Start in `dbt_nba/models/`; update corresponding `.yml` test definitions in `dbt_nba/models/schema.yml` or `_int_models.yml`.
+- **Fixing team or franchise mappings**: Edit `dbt_nba/seeds/team_maps.csv` or `arena_maps.csv`, then run `dbt seed`.
+- **Adjusting ETL pipeline or object storage**: Inspect `scripts/pipeline.sh`, `scripts/extract_data.py`, and `scripts/db_storage.py`.
+- **Modifying CI/CD workflows**: Inspect `.gitea/workflows/pipeline.yaml` (daily pipeline) and `test.yml` (health check).

README.md

@@ -1,6 +1,6 @@
 # 🏀 NBA Analytics — Streamlit & dbt Data Warehouse
 
-[![NBA Data Pipeline](https://git.turbo-data.com/nprasad2077/Streamlit_NBA/actions/workflows/pipeline.yaml/badge.svg?branch=main)](https://git.turbo-data.com/nprasad2077/Streamlit_NBA/actions?workflow=pipeline.yaml)
+[![NBA Data Pipeline](https://git.turbo-data.com/nprasad2077/Streamlit_NBA/actions/workflows/pipeline.yaml/badge.svg?branch=main)](https://git.turbo-data.com/nprasad2077/Streamlit_NBA/actions?workflow=pipeline.yaml)[![CI Health Check & Quality Gate](https://git.turbo-data.com/nprasad2077/Streamlit_NBA/actions/workflows/test.yml/badge.svg?branch=main)](https://git.turbo-data.com/nprasad2077/Streamlit_NBA/actions?workflow=test.yml)
 [![Live Dashboard](https://img.shields.io/badge/Live%20Dashboard-streamlit.turbo--data.com-FF6B00?logo=streamlit)](https://streamlit.turbo-data.com)
 
 A modern NBA analytics data platform powered by **dbt**, **DuckDB**, and **Streamlit**. Transforms raw basketball event data into a dimensional star schema (32,000+ games, 480,000+ shots from 2002 to 2026), persisting artifacts to **Cloudflare R2 Object Storage** with automated **Gitea Actions** CI/CD and self-hosted **Coolify** deployment.

dbt_nba/seeds/arena_maps.csv

@@ -3,95 +3,95 @@ ARCO Arena (II),Sacramento,California,North Natomas,1988-11-08,2016-10-30
 AT&T Center,San Antonio,Texas,East Side,2006-01-01,2023-09-21
 AccorHotels Arena,Paris,France,Bercy (12th arrondissement),2015-10-01,2020-06-01
 Air Canada Centre,Toronto,Ontario,Downtown/Harbourfront,1999-02-20,2018-06-30
-Alamodome,San Antonio,Texas,Downtown,1993-05-15,2026-06-30
+Alamodome,San Antonio,Texas,Downtown,1993-05-15,2030-06-30
-Amalie Arena,Tampa,Florida,Channelside,1996-10-20,2026-06-30
+Amalie Arena,Tampa,Florida,Channelside,1996-10-20,2030-06-30
 America West Arena,Phoenix,Arizona,Downtown,1992-06-06,2006-09-30
-American Airlines Center,Dallas,Texas,Victory Park,2001-07-17,2026-06-30
+American Airlines Center,Dallas,Texas,Victory Park,2001-07-17,2030-06-30
 AmericanAirlines Arena,Miami,Florida,Downtown/Biscayne Bay,1999-12-31,2021-06-01
 Amway Arena,Orlando,Florida,Downtown,1989-01-29,2010-10-01
 Amway Center,Orlando,Florida,Downtown,2010-10-01,2023-12-19
 BMO Harris Bradley Center,Milwaukee,Wisconsin,Downtown,2012-05-21,2018-09-17
-Ball Arena,Denver,Colorado,Central Platte Valley/LoDo,2020-10-22,2026-06-30
+Ball Arena,Denver,Colorado,Central Platte Valley/LoDo,2020-10-22,2030-06-30
 Bankers Life Fieldhouse,Indianapolis,Indiana,Downtown,2011-12-01,2021-09-26
-Barclays Center,Brooklyn,New York,Downtown Brooklyn/Prospect Heights,2012-09-21,2026-06-30
+Barclays Center,Brooklyn,New York,Downtown Brooklyn/Prospect Heights,2012-09-21,2030-06-30
 Bradley Center,Milwaukee,Wisconsin,Downtown,1988-10-01,2012-05-20
-Capital One Arena,Washington,District of Columbia,Chinatown/Penn Quarter,2017-08-09,2026-06-30
+Capital One Arena,Washington,District of Columbia,Chinatown/Penn Quarter,2017-08-09,2030-06-30
 Charlotte Bobcats Arena,Charlotte,North Carolina,Uptown (First Ward),2005-10-21,2008-08-01
 Charlotte Coliseum,Charlotte,North Carolina,Tyvola Road,1988-08-11,2005-10-26
-Chase Center,San Francisco,California,Mission Bay,2019-09-06,2026-06-30
+Chase Center,San Francisco,California,Mission Bay,2019-09-06,2030-06-30
 Chesapeake Energy Arena,Oklahoma City,Oklahoma,Downtown,2011-01-01,2021-04-01
 Compaq Center,Houston,Texas,Greenway Plaza,1998-01-01,2003-10-01
 Conseco Fieldhouse,Indianapolis,Indiana,Downtown,1999-11-06,2011-11-30
 Continental Airlines Arena,East Rutherford,New Jersey,Meadowlands Sports Complex,1996-01-04,2007-06-30
-Crypto.com Arena,Los Angeles,California,South Park/Downtown,2021-12-25,2026-06-30
+Crypto.com Arena,Los Angeles,California,South Park/Downtown,2021-12-25,2030-06-30
-Delta Center,Salt Lake City,Utah,West Downtown,1991-10-04,2026-06-30
+Delta Center,Salt Lake City,Utah,West Downtown,1991-10-04,2030-06-30
 EnergySolutions Arena,Salt Lake City,Utah,West Downtown,2006-11-20,2015-10-25
 FTX Arena,Miami,Florida,Downtown/Biscayne Bay,2021-06-01,2023-01-01
-FedEx Forum,Memphis,Tennessee,Downtown (near Beale Street),2004-09-06,2026-06-30
+FedEx Forum,Memphis,Tennessee,Downtown (near Beale Street),2004-09-06,2030-06-30
-FedExForum,Memphis,Tennessee,Downtown (near Beale Street),2004-09-06,2026-06-30
+FedExForum,Memphis,Tennessee,Downtown (near Beale Street),2004-09-06,2030-06-30
 First Union Center,Philadelphia,Pennsylvania,South Philadelphia Sports Complex,1998-09-01,2003-06-30
-Fiserv Forum,Milwaukee,Wisconsin,Deer District/Downtown,2018-08-26,2026-06-30
+Fiserv Forum,Milwaukee,Wisconsin,Deer District/Downtown,2018-08-26,2030-06-30
 FleetCenter,Boston,Massachusetts,West End,1995-09-30,2005-06-30
 Footprint Center,Phoenix,Arizona,Downtown,2021-07-16,2025-02-17
 Ford Center,Oklahoma City,Oklahoma,Downtown,2002-06-08,2010-12-31
-Frost Bank Center,San Antonio,Texas,East Side,2023-09-22,2026-06-30
+Frost Bank Center,San Antonio,Texas,East Side,2023-09-22,2030-06-30
-Gainbridge Fieldhouse,Indianapolis,Indiana,Downtown,2021-09-27,2026-06-30
+Gainbridge Fieldhouse,Indianapolis,Indiana,Downtown,2021-09-27,2030-06-30
-Golden 1 Center,Sacramento,California,Downtown Commons (DoCo),2016-10-04,2026-06-30
+Golden 1 Center,Sacramento,California,Downtown Commons (DoCo),2016-10-04,2030-06-30
 Gund Arena,Cleveland,Ohio,Gateway District/Downtown,1994-10-01,2005-08-01
-Intuit Dome,Inglewood,California,Hollywood Park,2024-08-15,2026-06-30
+Intuit Dome,Inglewood,California,Hollywood Park,2024-08-15,2030-06-30
 Izod Center,East Rutherford,New Jersey,Meadowlands Sports Complex,2007-07-01,2015-01-15
-Kaseya Center,Miami,Florida,Downtown/Biscayne Bay,2023-04-01,2026-06-30
+Kaseya Center,Miami,Florida,Downtown/Biscayne Bay,2023-04-01,2030-06-30
 KeyArena at Seattle Center,Seattle,Washington,Seattle Center/Lower Queen Anne,1995-10-26,2008-04-13
-Kia Center,Orlando,Florida,Downtown,2023-12-20,2026-06-30
+Kia Center,Orlando,Florida,Downtown,2023-12-20,2030-06-30
-LLoyd Noble Center,Norman,Oklahoma,University of Oklahoma Campus,1975-11-01,2026-06-30
+LLoyd Noble Center,Norman,Oklahoma,University of Oklahoma Campus,1975-11-01,2030-06-30
-Little Caesars Arena,Detroit,Michigan,Midtown/District Detroit,2017-09-05,2026-06-30
+Little Caesars Arena,Detroit,Michigan,Midtown/District Detroit,2017-09-05,2030-06-30
 MCI Center,Washington,District of Columbia,Chinatown/Penn Quarter,1997-12-02,2006-02-28
-Madison Square Garden (IV),New York,New York,Midtown Manhattan (Penn Station),1968-02-11,2026-06-30
+Madison Square Garden (IV),New York,New York,Midtown Manhattan (Penn Station),1968-02-11,2030-06-30
-Mexico City Arena,Mexico City,Mexico,Azcapotzalco,2012-02-25,2026-06-30
+Mexico City Arena,Mexico City,Mexico,Azcapotzalco,2012-02-25,2030-06-30
-Moda Center,Portland,Oregon,Rose Quarter/Lloyd District,2013-08-13,2026-06-30
+Moda Center,Portland,Oregon,Rose Quarter/Lloyd District,2013-08-13,2030-06-30
-Moody Center,Austin,Texas,UT Campus,2022-04-20,2026-06-30
+Moody Center,Austin,Texas,UT Campus,2022-04-20,2030-06-30
-Mortgage Matchup Center,Phoenix,Arizona,Downtown,2025-10-02,2026-06-30
+Mortgage Matchup Center,Phoenix,Arizona,Downtown,2025-10-02,2030-06-30
 New Orleans Arena,New Orleans,Louisiana,Central Business District,1999-10-01,2014-02-04
-Oakland Arena,Oakland,California,Coliseum Industrial,2019-06-14,2026-06-30
+Oakland Arena,Oakland,California,Coliseum Industrial,2019-06-14,2030-06-30
 Oklahoma City Arena,Oklahoma City,Oklahoma,Downtown,2010-01-01,2010-12-31
 Oracle Arena,Oakland,California,Coliseum Industrial,2006-10-20,2019-06-13
-Paycom Center,Oklahoma City,Oklahoma,Downtown,2021-07-27,2026-06-30
+Paycom Center,Oklahoma City,Oklahoma,Downtown,2021-07-27,2030-06-30
 Pepsi Center,Denver,Colorado,Central Platte Valley/LoDo,1999-10-01,2020-10-21
-Pete Maravich Assembly Center,Baton Rouge,Louisiana,LSU Campus,1972-01-15,2026-06-30
+Pete Maravich Assembly Center,Baton Rouge,Louisiana,LSU Campus,1972-01-15,2030-06-30
 Philips Arena,Atlanta,Georgia,Centennial Olympic Park District,1999-09-01,2018-08-28
 Phoenix Suns Arena,Phoenix,Arizona,Downtown,2020-11-01,2021-07-15
 Power Balance Pavilion,Sacramento,California,North Natomas,2011-03-01,2012-12-31
-Prudential Center,Newark,New Jersey,Downtown Newark,2007-10-25,2026-06-30
+Prudential Center,Newark,New Jersey,Downtown Newark,2007-10-25,2030-06-30
 Pyramid Arena,Memphis,Tennessee,Downtown (Riverfront),1991-11-09,2004-12-31
 Quicken Loans Arena,Cleveland,Ohio,Gateway District/Downtown,2005-08-01,2019-03-31
-Rocket Arena,Cleveland,Ohio,Gateway District/Downtown,2025-02-18,2026-06-30
+Rocket Arena,Cleveland,Ohio,Gateway District/Downtown,2025-02-18,2030-06-30
 Rocket Mortgage Fieldhouse,Cleveland,Ohio,Gateway District/Downtown,2019-04-01,2025-02-17
 Rose Garden Arena,Portland,Oregon,Rose Quarter/Lloyd District,1995-10-12,2013-08-12
 SBC Center,San Antonio,Texas,East Side,2002-10-18,2005-12-31
 STAPLES Center,Los Angeles,California,South Park/Downtown,1999-10-17,2021-12-24
-Saitama Super Arena,Saitama,Japan,Chuo-ku/Shin-Toshin,2000-09-01,2026-06-30
+Saitama Super Arena,Saitama,Japan,Chuo-ku/Shin-Toshin,2000-09-01,2030-06-30
-Scotiabank Arena,Toronto,Ontario,Downtown/Harbourfront,2018-07-01,2026-06-30
+Scotiabank Arena,Toronto,Ontario,Downtown/Harbourfront,2018-07-01,2030-06-30
 Sleep Train Arena,Sacramento,California,North Natomas,2013-01-01,2016-10-30
-Smoothie King Center,New Orleans,Louisiana,Central Business District,2014-02-05,2026-06-30
+Smoothie King Center,New Orleans,Louisiana,Central Business District,2014-02-05,2030-06-30
-Spectrum Center,Charlotte,North Carolina,Uptown (First Ward),2016-08-17,2026-06-30
+Spectrum Center,Charlotte,North Carolina,Uptown (First Ward),2016-08-17,2030-06-30
-State Farm Arena,Atlanta,Georgia,Centennial Olympic Park District,2018-08-29,2026-06-30
+State Farm Arena,Atlanta,Georgia,Centennial Olympic Park District,2018-08-29,2030-06-30
-T-Mobile Arena,Las Vegas,Nevada,Las Vegas Strip,2016-04-06,2026-06-30
+T-Mobile Arena,Las Vegas,Nevada,Las Vegas Strip,2016-04-06,2030-06-30
 TD Banknorth Garden,Boston,Massachusetts,West End,2005-07-01,2009-07-15
-TD Garden,Boston,Massachusetts,West End,2009-07-16,2026-06-30
+TD Garden,Boston,Massachusetts,West End,2009-07-16,2030-06-30
 TD Waterhouse Centre,Orlando,Florida,Downtown,1999-10-01,2002-12-31
 Talking Stick Resort Arena,Phoenix,Arizona,Downtown,2015-09-01,2020-10-31
-Target Center,Minneapolis,Minnesota,Warehouse District/North Loop,1990-10-16,2026-06-30
+Target Center,Minneapolis,Minnesota,Warehouse District/North Loop,1990-10-16,2030-06-30
 The Arena in Oakland,Oakland,California,Coliseum Industrial,1997-10-01,2004-12-31
-The O2 Arena,London,United Kingdom,Greenwich Peninsula,2007-06-24,2026-06-30
+The O2 Arena,London,United Kingdom,Greenwich Peninsula,2007-06-24,2030-06-30
 The Palace of Auburn Hills,Auburn Hills,Michigan,Auburn Hills (I-75 corridor),1988-10-01,2017-10-12
 Time Warner Cable Arena,Charlotte,North Carolina,Uptown (First Ward),2008-08-01,2016-08-16
-Toyota Center,Houston,Texas,Downtown/East Downtown,2003-10-06,2026-06-30
+Toyota Center,Houston,Texas,Downtown/East Downtown,2003-10-06,2030-06-30
 US Airways Center,Phoenix,Arizona,Downtown,2006-10-01,2015-08-31
-UW–Milwaukee Panther Arena,Milwaukee,Wisconsin,Downtown,1950-08-04,2026-06-30
+UW–Milwaukee Panther Arena,Milwaukee,Wisconsin,Downtown,1950-08-04,2030-06-30
-United Center,Chicago,Illinois,Near West Side,1994-08-18,2026-06-30
+United Center,Chicago,Illinois,Near West Side,1994-08-18,2030-06-30
 Verizon Center,Washington,District of Columbia,Chinatown/Penn Quarter,2006-03-01,2017-08-08
 Vivint Arena,Salt Lake City,Utah,West Downtown,2020-01-01,2023-06-30
 Vivint Smart Home Arena,Salt Lake City,Utah,West Downtown,2015-10-26,2019-12-31
 Wachovia Center,Philadelphia,Pennsylvania,South Philadelphia Sports Complex,2003-07-01,2010-06-30
 Wells Fargo Center,Philadelphia,Pennsylvania,South Philadelphia Sports Complex,2010-07-01,2025-08-31
-Xfinity Mobile Arena,Philadelphia,Pennsylvania,South Philadelphia Sports Complex,2025-09-01,2026-06-30
+Xfinity Mobile Arena,Philadelphia,Pennsylvania,South Philadelphia Sports Complex,2025-09-01,2030-06-30

dbt_nba/seeds/team_maps.csv

@@ -1,39 +1,39 @@
 team_abbr,full_name,start_year,end_year,is_current,source,notes
-ATL,Atlanta Hawks,1968,2026,TRUE,all,No changes since 1968
+ATL,Atlanta Hawks,1968,2030,TRUE,all,No changes since 1968
-BOS,Boston Celtics,1946,2026,TRUE,all,No changes since 1946
+BOS,Boston Celtics,1946,2030,TRUE,all,No changes since 1946
-BRK,Brooklyn Nets,2012,2026,TRUE,bref,Basketball-Reference abbreviation for Brooklyn
+BRK,Brooklyn Nets,2012,2030,TRUE,bref,Basketball-Reference abbreviation for Brooklyn
 CHA,Charlotte Bobcats,2004,2014,FALSE,bref,Renamed to Hornets in 2014; ESPN/NBA.com also use CHA for current Hornets
 CHH,Charlotte Hornets,1988,2002,FALSE,bref,Original Hornets; relocated to New Orleans after 2001-02
-CHI,Chicago Bulls,1966,2026,TRUE,all,No changes since 1966
+CHI,Chicago Bulls,1966,2030,TRUE,all,No changes since 1966
-CHO,Charlotte Hornets,2014,2026,TRUE,bref,Basketball-Reference abbreviation; reclaimed original Hornets history
+CHO,Charlotte Hornets,2014,2030,TRUE,bref,Basketball-Reference abbreviation; reclaimed original Hornets history
-CLE,Cleveland Cavaliers,1970,2026,TRUE,all,No changes since 1970
+CLE,Cleveland Cavaliers,1970,2030,TRUE,all,No changes since 1970
-DAL,Dallas Mavericks,1980,2026,TRUE,all,Expansion team 1980-81 season
+DAL,Dallas Mavericks,1980,2030,TRUE,all,Expansion team 1980-81 season
-DEN,Denver Nuggets,1976,2026,TRUE,all,Joined NBA in 1976 from ABA
+DEN,Denver Nuggets,1976,2030,TRUE,all,Joined NBA in 1976 from ABA
-DET,Detroit Pistons,1957,2026,TRUE,all,In Detroit since 1957
+DET,Detroit Pistons,1957,2030,TRUE,all,In Detroit since 1957
-GSW,Golden State Warriors,1971,2026,TRUE,all,Name changed from San Francisco Warriors in 1971
+GSW,Golden State Warriors,1971,2030,TRUE,all,Name changed from San Francisco Warriors in 1971
-HOU,Houston Rockets,1971,2026,TRUE,all,Moved from San Diego in 1971
+HOU,Houston Rockets,1971,2030,TRUE,all,Moved from San Diego in 1971
-IND,Indiana Pacers,1976,2026,TRUE,all,Joined NBA in 1976 from ABA
+IND,Indiana Pacers,1976,2030,TRUE,all,Joined NBA in 1976 from ABA
-LAC,Los Angeles Clippers,1984,2026,TRUE,all,Relocated from San Diego
+LAC,Los Angeles Clippers,1984,2030,TRUE,all,Relocated from San Diego
-LAL,Los Angeles Lakers,1960,2026,TRUE,all,Moved from Minneapolis in 1960
+LAL,Los Angeles Lakers,1960,2030,TRUE,all,Moved from Minneapolis in 1960
-MEM,Memphis Grizzlies,2001,2026,TRUE,all,Relocated from Vancouver
+MEM,Memphis Grizzlies,2001,2030,TRUE,all,Relocated from Vancouver
-MIA,Miami Heat,1988,2026,TRUE,all,Expansion team 1988-89 season
+MIA,Miami Heat,1988,2030,TRUE,all,Expansion team 1988-89 season
-MIL,Milwaukee Bucks,1968,2026,TRUE,all,No changes since 1968
+MIL,Milwaukee Bucks,1968,2030,TRUE,all,No changes since 1968
-MIN,Minnesota Timberwolves,1989,2026,TRUE,all,Expansion team 1989-90 season
+MIN,Minnesota Timberwolves,1989,2030,TRUE,all,Expansion team 1989-90 season
 NJN,New Jersey Nets,1977,2012,FALSE,all,Moved to Brooklyn after 2011-12 season
 NOH,New Orleans Hornets,2002,2005,FALSE,bref,Relocated from Charlotte; excludes NOK years; renamed Pelicans in 2013
 NOK,New Orleans/Oklahoma City Hornets,2005,2007,FALSE,bref,Hurricane Katrina temporary relocation; split home games
-NOP,New Orleans Pelicans,2013,2026,TRUE,all,Renamed from Hornets
+NOP,New Orleans Pelicans,2013,2030,TRUE,all,Renamed from Hornets
-NYK,New York Knicks,1946,2026,TRUE,all,No changes since 1946
+NYK,New York Knicks,1946,2030,TRUE,all,No changes since 1946
-OKC,Oklahoma City Thunder,2008,2026,TRUE,all,Relocated from Seattle
+OKC,Oklahoma City Thunder,2008,2030,TRUE,all,Relocated from Seattle
-ORL,Orlando Magic,1989,2026,TRUE,all,Expansion team 1989-90 season
+ORL,Orlando Magic,1989,2030,TRUE,all,Expansion team 1989-90 season
-PHI,Philadelphia 76ers,1963,2026,TRUE,all,Moved from Syracuse in 1963
+PHI,Philadelphia 76ers,1963,2030,TRUE,all,Moved from Syracuse in 1963
-PHO,Phoenix Suns,1968,2026,TRUE,bref,Basketball-Reference abbreviation
+PHO,Phoenix Suns,1968,2030,TRUE,bref,Basketball-Reference abbreviation
-POR,Portland Trail Blazers,1970,2026,TRUE,all,No changes since 1970
+POR,Portland Trail Blazers,1970,2030,TRUE,all,No changes since 1970
-SAC,Sacramento Kings,1985,2026,TRUE,all,Relocated from Kansas City
+SAC,Sacramento Kings,1985,2030,TRUE,all,Relocated from Kansas City
-SAS,San Antonio Spurs,1976,2026,TRUE,all,Joined NBA in 1976 from ABA
+SAS,San Antonio Spurs,1976,2030,TRUE,all,Joined NBA in 1976 from ABA
 SDC,San Diego Clippers,1978,1984,FALSE,bref,Moved to Los Angeles after 1983-84 season
 SEA,Seattle SuperSonics,1967,2008,FALSE,all,Relocated to Oklahoma City after 2007-08 season
-TOR,Toronto Raptors,1995,2026,TRUE,all,Expansion team 1995-96 season
+TOR,Toronto Raptors,1995,2030,TRUE,all,Expansion team 1995-96 season
-UTA,Utah Jazz,1979,2026,TRUE,all,Moved from New Orleans in 1979
+UTA,Utah Jazz,1979,2030,TRUE,all,Moved from New Orleans in 1979
 VAN,Vancouver Grizzlies,1995,2001,FALSE,all,Expansion team 1995-96; relocated to Memphis after 2000-01
-WAS,Washington Wizards,1997,2026,TRUE,all,Renamed from Bullets
+WAS,Washington Wizards,1997,2030,TRUE,all,Renamed from Bullets

scripts/extract_data.py

@@ -0,0 +1,42 @@
+#!/usr/bin/env python3
+"""
+Extract source tables from Postgres into DuckDB using the DuckDB Python library.
+Reads SQL commands from scripts/extract.sql with environment variable interpolation.
+"""
+import os
+import sys
+from pathlib import Path
+import duckdb
+
+def main():
+    repo_root = Path(__file__).resolve().parent.parent
+    db_path = repo_root / "data" / "DB" / "dbt_nba.duckdb"
+    sql_path = repo_root / "scripts" / "extract.sql"
+    
+    postgres_url = os.getenv("POSTGRES_URL")
+    if not postgres_url:
+        print("ERROR: POSTGRES_URL environment variable is not set", file=sys.stderr)
+        print(f"Please export POSTGRES_URL or configure it in {repo_root}/.env", file=sys.stderr)
+        sys.exit(1)
+        
+    db_path.parent.mkdir(parents=True, exist_ok=True)
+    if db_path.exists():
+        db_path.unlink()
+        
+    with open(sql_path, "r", encoding="utf-8") as f:
+        sql = f.read()
+        
+    # Replace variable
+    sql = sql.replace("${POSTGRES_URL}", postgres_url)
+    
+    print(f"==> Connecting to DuckDB at {db_path}...")
+    con = duckdb.connect(str(db_path))
+    try:
+        print("==> Executing extraction queries...")
+        con.execute(sql)
+        print("==> Extraction complete successfully!")
+    finally:
+        con.close()
+
+if __name__ == "__main__":
+    main()

scripts/pipeline.sh

@@ -23,7 +23,7 @@ echo "==> [1/4] Extracting from Postgres into DuckDB..."
 mkdir -p "$(dirname "$DB_PATH")"
 mkdir -p "$(dirname "$REPORTS_DB")"
 rm -f "$DB_PATH"
-envsubst < "$REPO_ROOT/scripts/extract.sql" | duckdb "$DB_PATH"
+uv run --project "$REPO_ROOT" python "$REPO_ROOT/scripts/extract_data.py"
 
 echo "==> [2/4] Running dbt build --full-refresh..."
 export DBT_PROJECT_DIR="$REPO_ROOT/dbt_nba"