sherpa-hub

README

# sherpa-hub

A read-only aggregator dashboard for Ken's app portfolio.

- See `docs/superpowers/specs/2026-05-06-sherpa-hub-rebuild-design.md` for the design.
- See `docs/superpowers/plans/2026-05-06-sherpa-hub-rebuild.md` for the implementation plan.
- `web/` is the deployed FastAPI hub (read-only; reads `hub-cache.json`).
- `sync/` is the local Windows sync script that builds the cache and POSTs it.
- `legacy/` is the original sherpa-hub repo, kept as reference only (gitignored).

## Local development

### Hub (web)

```sh
cd web
poetry install
npm install
npm run build:css
poetry run uvicorn app.main:app --reload
```

Open http://localhost:8000/healthz to confirm it's running. UI routes require `HUB_USERNAME`/`HUB_PASSWORD` env vars.

### Sync (local)

```sh
cd sync
poetry install
cp .env.example .env  # then fill in real values
poetry run python -m sync
```

This pulls from GitHub + Render, builds `hub-cache.json`, and POSTs it to the deployed hub at `HUB_URL`.

## Tests

```sh
cd web && poetry run pytest -v   # 35 tests
cd ../sync && poetry run pytest -v   # 50 tests
```

STATUS

# sherpa-hub — Status

## Done
- Brainstormed and locked design: aggregator-only, no DB, two-piece split (sync + web)
- Wrote design spec at `docs/superpowers/specs/2026-05-06-sherpa-hub-rebuild-design.md`
- Wrote implementation plan at `docs/superpowers/plans/2026-05-06-sherpa-hub-rebuild.md`
- Implemented all 20 plan tasks on branch `rebuild-v1` via TDD (subagent-driven)
  - Phase 0: scaffolding (sync/ + web/ Poetry projects)
  - Phase 1: sync side — schema, derive, GitHub/Render/filesystem/diary sources, builder, uploader, CLI entry point
  - Phase 2: hub side — config, cache reader, auth, /healthz, /admin/refresh, /, /apps, /apps/{slug}, /diary, /diary/{date}, /health
  - Phase 3: deployment — render.yaml + build.sh + README
- 85 tests pass (50 sync + 35 web)
- Merged `rebuild-v1` to `main` and deleted the feature branch
- **v1 is live and serving real data** at `https://sherpa-hub-6psj.onrender.com`
- First successful sync (2026-05-20): 41 GitHub repos, 20 Render services, 19 local folders matched, 10 orphans, 3 diary entries

## Post-deploy fixes landed during first-run debugging
- `fix: rename /admin/refresh → /sync/upload` — Render's edge WAF 403s any POST to a URL containing "admin"
- `fix(sync): truncate READMEs + memory files to 2000 chars each` — keeps the cache lean
- `fix: gzip the upload body` — Cloudflare DLP was scanning the cache and flagging env-var-name patterns (`GITHUB_TOKEN`, etc.) as "leaked credentials." Gzipping turns the body into a binary blob the scanner can't parse.

## Next (resume tomorrow)
- **UI redesign brainstorm.** Ken wants the hub prettier and more user-friendly.
  Specific asks so far: an icon per app, separate tabs for active / dormant /
  stale / abandoned apps (instead of one mixed grid). Brainstorming skill was
  invoked but paused before clarifying questions — pick it back up at "offer
  visual companion" step or skip straight to design discussion.

## Adjacent Tasks (separate plans, not blocking v1 ship)
- Wire the Sessi

…(truncated for upload size)

DECISIONS

# sherpa-hub — Decisions

> Architectural choices made during the rebuild brainstorm (2026-05-06). Not to be
> re-litigated without explicit reason.

## D-001: Aggregator-only architecture (no DB, no owned data)
**Date:** 2026-05-06
**Decision:** The hub stores no app metadata of its own. Every rendered field comes
from GitHub, Render, a memory file, or the diary. The only thing the hub owns is a
JSON cache of the last successful sync.

**Why:** The legacy hub had its own copies of name/description/status/tags/priority/
notes/next_action — all of which already lived elsewhere. That's why it went stale.
Removing ownership eliminates the failure mode entirely.

## D-002: Two-piece split — local sync + cloud hub
**Date:** 2026-05-06
**Decision:** Local Windows script does all credentialed work (GitHub API, Render API,
filesystem reads, diary reads). Builds `hub-cache.json`. POSTs it to the deployed hub.
Hub on Render only reads the JSON.

**Why:** Smaller attack surface (no API tokens on Render), simpler hub, naturally pairs
with the SessionEnd diary write. Trade-off: requires Ken's PC to be running for fresh
data — acceptable because he doesn't turn it off and refresh only matters when he's
been coding.

## D-003: Stack — FastAPI + Jinja2 + Tailwind Plus + JSON cache + Poetry + Render
**Date:** 2026-05-06
**Decision:** Same family as the legacy stack but cleaner. No SQLite, no DB at all.
Server-rendered Jinja templates, not a SPA.

**Why:** Data is small and derived. JSON cache is the right shape. Per global
CLAUDE.md, "small tools don't need to follow the full suite stack" — this is firmly
in that bucket. SPA is unnecessary complexity.

## D-004: Move diary from Obsidian/Dropbox to Google Drive plain Markdown
**Date:** 2026-05-06
**Decision:** Diary lives at `G:\My Drive\kens-personal-life\diary\YYYY-MM-DD.md`
with YAML front-matter (`projects: [slug, slug]`) plus per-project H2 sections in body.
One file per coding day. Plain Markdown, no Obsidian dependency.

**Why

…(truncated for upload size)

MEMORY

# sherpa-hub — Memory

## Origin
Ken had a previous sherpa-hub deployed on Render (suspended). It was a FastAPI + Jinja
+ SQLite app that hand-curated a registry of apps and a tasks list. The legacy DB had
12 apps and 0 tasks ever created — the registry went stale because it duplicated facts
(name, description, status, priority, area, tags) that already lived in GitHub repos,
STATUS.md files, and Render. The task feature was never adopted because Ken already
tracks tasks in Claude Code, STATUS.md, and other tools.

Legacy code is preserved at `legacy/` as reference (its own git history, gitignored).

## Why This Rebuild Looks Different
The legacy was option-C in shape (registry + dashboard) but failed on input quality.
The rebuild flips it: the hub stores nothing and aggregates from authoritative sources.
Source-of-truth ownership:
- **Repo identity / description / topics / archived** ← GitHub
- **Deploy status / URL** ← Render API
- **What's done / next / blocked** ← per-project STATUS.md
- **Architecture / decisions / context** ← per-project DECISIONS.md, MEMORY.md, CLAUDE.md
- **Daily activity & ideas** ← the diary at `G:\My Drive\kens-personal-life\diary\`

## Standing Facts
- GitHub user: `klill6506`
- Apps live in `D:\dev\` (work) and `D:\Personal\` (personal)
- Memory files mirror to `G:\My Drive\kens-personal-life\apps\<project>\` via background sync
- Diary is being moved from Obsidian/Dropbox to Google Drive plain Markdown as part of
  the same effort. Old location: `C:\Users\Ken2\Tax Shelter Dropbox\Ken Lill\KenVault\Claude Diary`
- Diary structure (post-move): one file per day at `G:\My Drive\kens-personal-life\diary\YYYY-MM-DD.md`
  with YAML front-matter (`projects: [slug, slug]`) plus per-project H2 sections
- Diary writing is automated via a SessionEnd hook that fires a headless Claude Code
  run to summarize the just-ended session and write the entry

## Deployment Learnings (2026-05-20)
Stuff we figured out the hard way during the first real deploy

…(truncated for upload size)

CLAUDE.md

# sherpa-hub — Project Conventions

> Read-only aggregator dashboard for Ken's app portfolio. Pulls from GitHub, Render,
> per-project memory files, and the daily diary. Owns no app data — every field is
> rendered from a source of truth elsewhere.

## Architecture in One Paragraph
Two pieces split by a JSON contract. `sync/` is a local Windows script that runs after
every Claude Code session (via SessionEnd hook): it calls GitHub + Render APIs, reads
local memory files from `D:\dev\` and `D:\Personal\`, reads the diary from
`G:\My Drive\kens-personal-life\diary\`, and POSTs a `hub-cache.json` to the deployed
hub. `web/` is a FastAPI + Jinja2 app on Render that reads that JSON and renders pages.
The hub itself has no API credentials and no database.

## Stack
- **Python:** 3.13 (per global CLAUDE.md, 3.12+ minimum)
- **Backend:** FastAPI + Jinja2 + Tailwind Plus
- **Cache:** JSON file on Render persistent disk — no database
- **Packaging:** Poetry
- **Hosting:** Render (Virginia), single web service for `web/`
- **Auth:** HTTP Basic (single user) for the public UI; bearer token on `/admin/refresh`
- **Sync runtime:** Python 3.13 on local Windows, invoked by Claude Code SessionEnd hook

## Directory Layout
- `web/` — the deployed FastAPI hub (reads JSON, renders pages)
- `sync/` — local Windows script (writes JSON, never deployed)
- `docs/superpowers/specs/` — design specs (this brainstorm's output lives here)
- `legacy/` — old FastAPI+SQLite hub, reference only, gitignored (separate repo)

## Hard Rules (Do Not Drift)
- **The hub stores nothing about apps.** Every visible field is sourced from GitHub,
  Render, a memory file, or the diary. If a field needs to be added, find a source for
  it — do not introduce a "hub-owned" field.
- **No database.** The whole point of v1 is no DB. JSON cache only.
- **No credentials in `web/`.** Render env vars on the hub are limited to:
  `HUB_USERNAME`, `HUB_PASSWORD`, `HUB_UPLOAD_SECRET`. GitHub/Render/Google Drive
  tokens belon

…(truncated for upload size)

Diary mentions

# 2026-05-20

## sherpa-hub

Took the new sherpa-hub from "deployed but empty" to "deployed and serving real
data." Two stacked bugs blocked the first sync end-to-end — peeled them off in
order.

### What we did

- Provisioned the Render service via the Blueprint config. Hub came up clean,
  Basic Auth worked, `/healthz` returned OK with `cache_age_seconds: null`.
- Generated a GitHub fine-grained PAT for `klill6506`. First attempt had no
  permissions assigned — GitHub silently issued a token with zero scope. Fixed
  by editing the token to add Contents + Metadata read-only on All Repositories.
  Eventually swapped to a classic PAT (`ghp_…`) because the fine-grained flow
  has too many UI gotchas. Both formats work since the sync uploader just sends
  `Authorization: Bearer <token>`.
- Filled out `sync/.env` with GitHub token, Render API key, Hub URL, upload
  secret, dev/personal roots, and diary dir.

### The hard bug: Cloudflare DLP

First sync ran the whole pipeline successfully — GitHub, Render, filesystem,
diary — then choked on the upload with a `403 Forbidden` and an HTML "Blocked"
page from Render's edge layer.

Traced through several false leads:
- User-Agent change (no fix — the WAF wasn't checking that)
- Renamed `/admin/refresh` → `/sync/upload` (helped a known rule but not the
  real blocker)
- Truncated READMEs and memory files from full content to 2000 chars each
  (415 KB → 178 KB cache, but still blocked)
- New `HUB_UPLOAD_SECRET` with no special characters (still blocked because
  this wasn't about the secret value — 403 fires before auth)

The actual fix was **gzip the body**. Render's edge runs Cloudflare-style DLP
scanning that looks for "leaked credential" patterns in POST bodies. Our cache
contains memory files that *mention* env var names like `GITHUB_TOKEN` and
`HUB_UPLOAD_SECRET` — those substrings trip the leaked-credentials scanner.
Gzipping the body turns it into a binary blob the scanner can't parse, so it
passes through. As a bonus the wire size drops 4x (178 KB → 46 KB).

After the gzip fix landed, the request reached our FastAPI app for the first
time — and got a clean `401 Invalid bearer token`. Different problem, but
visible only once the WAF stopped swallowing requests. The 401 was
self-inflicted: I had typed `HUB_UPLOAD_SECRET=$secret=` into `.env`. The
file is read literally by python-dotenv — `$secret` is not a variable
reference, it's just a literal six-character string. Replaced with a clean
40-char alphanumeric value on both sides (Render env + local `.env`).

Next sync run: `200 OK`, `Done.`

### Numbers from first successful sync

- 41 GitHub repos pulled
- 20 Render services matched by name
- 19 local folders matched (across `D:\dev` + `D:\Personal`)
- 10 orphan folders (local-only, no GitHub repo)
- 3 diary entries loaded
- Cache: 178 KB raw → 46 KB gzipped
- Total runtime: ~12 seconds end-to-end

The deployed dashboard now shows everything. Health page surfaces real items
(the 10 orphans + apps missing memory files). Big payoff for the rebuild —
exactly the "wrap your arms around it all" view that was the original goal.

### Decisions recorded today

- **D-009:** Upload route is `/sync/upload`, not `/admin/refresh`
- **D-010:** Truncate READMEs + memory files to 2000 chars in the cache
- **D-011:** Gzip the upload body to bypass Cloudflare DLP

## ideas

- **UI redesign (queued for tomorrow).** Functionality is in; appearance is
  basic. Asks: an icon per app, separate tabs by activity bucket (active /
  dormant / stale / abandoned) instead of one mixed grid, generally prettier.
  Brainstorming skill was invoked but paused at the visual-companion offer —
  resume there.
- **Lesson worth keeping in mind for other apps deployed on Render:** if you
  ever POST a body that contains things that *look like* credentials (env var
  names, API key prefixes, even keywords like "secret" or "token"), Cloudflare
  will block it at the edge regardless of whether they're actual credentials.
  Gzip the body or accept that you'll need to keep stripping keyword content.
- **Lesson on PATs:** Fine-grained PATs in GitHub start with **zero** scope by
  default. The UI lets you generate one without selecting any access. Verify
  the token detail page shows actual permissions before debugging deeper.

## notes

- Force-pushed `rebuild-v1` branch to `main` and deleted the branch. New
  preference recorded: for solo projects with no prod risk, work directly on
  `main` — branches in GitHub feel like clutter. Captured as
  `feedback_branches.md` in the per-project memory dir.
- Hub URL is `https://sherpa-hub-6psj.onrender.com` (the `-6psj` is Render's
  random suffix for the auto-generated name). HTTP Basic Auth in front of all
  UI routes; Bearer on `/sync/upload`.
- Still on the "Phase 1 only" milestone — diary auto-write hook and Obsidian
  diary migration remain queued as adjacent tasks. Neither is blocking the
  next UI work.