python migration
This commit is contained in:
+77
-60
@@ -1,5 +1,7 @@
|
||||
# Implementation Plan — Test Dashboard
|
||||
|
||||
> **Status as of May 2026**: fully implemented and running. Backend migrated from the original Node.js plan to Python (Flask). See current tech stack and structure below.
|
||||
|
||||
## 1. Overview
|
||||
|
||||
A web dashboard that monitors test execution progress by comparing a **target tests directory** (what should run) against a **results directory** (what has run). The backend runs on one machine with access to both directories; the frontend is accessible from any device on the network.
|
||||
@@ -8,55 +10,59 @@ A web dashboard that monitors test execution progress by comparing a **target te
|
||||
|
||||
## 2. Tech Stack
|
||||
|
||||
| Layer | Technology | Rationale |
|
||||
| Layer | Technology | Notes |
|
||||
|---|---|---|
|
||||
| Frontend | React + Vite (existing) | Already scaffolded |
|
||||
| Backend | Node.js + Express | Lightweight API + static file serving |
|
||||
| Real-time | Server-Sent Events (SSE) | Simpler than WebSocket for one-way push |
|
||||
| Directory watching | chokidar | Cross-platform file system watcher |
|
||||
| Database | SQLite (via `better-sqlite3`) | Embedded, no separate process, fast reads |
|
||||
| Frontend state | React Query (TanStack Query) | Cache, refetch, and SSE invalidation |
|
||||
| UI | TailwindCSS + shadcn/ui | Rapid, consistent component styling |
|
||||
| Frontend | React + Vite | Dev server proxies `/api` → `localhost:3001` |
|
||||
| Backend | Python 3 + Flask | Replaced original Node.js/Express plan |
|
||||
| Real-time | Server-Sent Events (SSE) | One-way push from backend to browser |
|
||||
| Directory watching | watchdog | Python filesystem watcher (Windows-compatible) |
|
||||
| Database | SQLite (via `sqlite3` stdlib) | WAL mode; single persistent connection with RLock |
|
||||
| Frontend state | TanStack React Query | Cache + SSE-driven invalidation |
|
||||
| UI | TailwindCSS v4 | Via `@tailwindcss/vite` plugin |
|
||||
|
||||
---
|
||||
|
||||
## 3. Project Structure
|
||||
|
||||
```
|
||||
Projects/
|
||||
├── dashboard/ ← React frontend (existing)
|
||||
│ ├── src/
|
||||
│ │ ├── components/
|
||||
│ │ │ ├── StatCard.jsx – metric display card
|
||||
│ │ │ ├── CompletionBar.jsx – progress bar with percentage
|
||||
│ │ │ ├── TestTable.jsx – filterable test list
|
||||
│ │ │ ├── FilterPanel.jsx – tag filter controls
|
||||
│ │ │ └── TimeDisplay.jsx – elapsed / estimated time
|
||||
│ │ ├── hooks/
|
||||
│ │ │ ├── useTests.js – fetch + SSE subscription
|
||||
│ │ │ └── useStats.js – derived stats from test data
|
||||
│ │ ├── lib/
|
||||
│ │ │ └── api.js – axios/fetch base client
|
||||
│ │ ├── App.jsx
|
||||
│ │ └── main.jsx
|
||||
│ ├── package.json
|
||||
│ └── vite.config.js – proxy /api → backend port
|
||||
test_house_dashboard/
|
||||
├── DESIGNPLAN.md
|
||||
├── IMPLEMENTATION_PLAN.md
|
||||
├── CLAUDE.md ← project context for AI assistants
|
||||
│
|
||||
└── server/ ← NEW: Node.js backend
|
||||
├── index.js – Express app entry point
|
||||
├── db.js – SQLite schema + query helpers
|
||||
├── watcher.js – chokidar setup + change handlers
|
||||
├── parser.js – filename/foldername → tags
|
||||
├── scanner.js – full directory scan on startup
|
||||
├── sse.js – SSE client registry + broadcast
|
||||
├── routes/
|
||||
│ ├── tests.js – GET /api/tests, GET /api/tests/:id
|
||||
│ ├── stats.js – GET /api/stats
|
||||
│ ├── config.js – GET/POST /api/config (directory paths)
|
||||
│ ├── browse.js – GET /api/browse (server-side directory browser)
|
||||
│ └── events.js – GET /api/events (SSE stream)
|
||||
├── package.json
|
||||
└── .env – TARGET_DIR, RESULTS_DIR, PORT=3001
|
||||
├── dashboard/ ← React frontend
|
||||
│ ├── vite.config.js – proxy /api → localhost:3001
|
||||
│ ├── index.html
|
||||
│ ├── package.json
|
||||
│ └── src/
|
||||
│ ├── App.jsx – root layout, SSE wiring via useStats
|
||||
│ ├── main.jsx
|
||||
│ ├── components/
|
||||
│ │ ├── StatCard.jsx – metric card (label / value / sub)
|
||||
│ │ ├── CompletionBar.jsx – progress bar
|
||||
│ │ ├── TestTable.jsx – filterable test list table
|
||||
│ │ ├── FilterPanel.jsx – dropdown tag filters
|
||||
│ │ ├── TimeDisplay.jsx – elapsed / estimated time display
|
||||
│ │ ├── StatusBadge.jsx – completed / pending indicator
|
||||
│ │ ├── ConfigModal.jsx – settings modal (dirs + avg time overrides)
|
||||
│ │ └── DirectoryBrowser.jsx – server-side folder picker (uses /api/browse)
|
||||
│ ├── hooks/
|
||||
│ │ ├── useStats.js – fetches stats; owns the SSE EventSource
|
||||
│ │ ├── useTests.js – fetches test list
|
||||
│ │ └── useConfig.js – fetches/saves config
|
||||
│ └── lib/
|
||||
│ └── api.js – thin fetch wrapper; BASE = '/api'
|
||||
│
|
||||
└── server/ ← Python backend
|
||||
├── app.py – Flask app, all routes, bootstrap(), __main__
|
||||
├── db_py.py – SQLite schema, query helpers, config CRUD
|
||||
├── scanner.py – full_scan(), scan_targets(), scan_results(), process_result_dir()
|
||||
├── parser.py – parse_target_filename(), parse_result_filename(), etc.
|
||||
├── watcher.py – watchdog observers for target + results dirs
|
||||
├── sse_py.py – SSE queue registry + broadcast()
|
||||
├── requirements.txt
|
||||
├── dashboard.db – SQLite database (auto-created)
|
||||
└── .venv/ – Python virtual environment
|
||||
```
|
||||
|
||||
---
|
||||
@@ -299,12 +305,16 @@ Allows the frontend to navigate the server's local filesystem so users can pick
|
||||
|
||||
## 7. Real-time Updates
|
||||
|
||||
- The backend registers an SSE endpoint at `GET /api/events`
|
||||
- chokidar watches both the target and results directories for `add`, `unlink`, and `change` events
|
||||
- On any change, the backend re-scans the affected path, updates SQLite, and broadcasts an SSE event: `data: {"type":"update"}`
|
||||
- The React frontend subscribes to the SSE stream; on receiving an `update` event it invalidates and refetches stats and test list via React Query
|
||||
- Backend registers an SSE endpoint at `GET /api/events` (`sse_py.py`)
|
||||
- `watchdog` watches both target and results directories via two separate `Observer` instances (`watcher.py`)
|
||||
- On any change, the backend updates SQLite and calls `broadcast({"type": "update"})`, which pushes `data: {"type":"update"}\n\n` to all connected SSE clients
|
||||
- `useStats.js` owns the `EventSource('/api/events')` connection; on `update` it calls `queryClient.invalidateQueries` for both `['stats']` and `['tests']`
|
||||
|
||||
**`unlink` handling for result directories**: when chokidar detects that a result directory has been deleted, the corresponding test row is reset — `completed = 0`, `completed_at = NULL`, `duration_seconds = NULL`. This ensures the dashboard always reflects the actual state of the filesystem.
|
||||
**Windows watchdog quirk — `is_directory` unreliable on delete**: when a directory is deleted, watchdog calls `os.path.isdir()` to set `event.is_directory`, but the directory is already gone by then, so it returns `False`. All delete/move handlers therefore check `_is_direct_child_dir(path)` directly instead of relying on `event.is_directory`.
|
||||
|
||||
**Result directory deleted**: `reset_by_file_id_and_device(test_id, device)` resets `completed = 0`, `completed_at = NULL`, `duration_seconds = NULL`. broadcast fires unconditionally regardless of whether the folder name was parseable.
|
||||
|
||||
**Recycle Bin delete on Windows**: fires a `MovedEvent` (`src` = result dir, `dest` = `$RECYCLE.BIN\...`). Handled by `on_moved` via `_is_direct_child_dir` on the source path.
|
||||
|
||||
---
|
||||
|
||||
@@ -370,34 +380,41 @@ All config values are POSTed to `POST /api/config` as key/value pairs and persis
|
||||
|
||||
## 9. Vite Proxy Configuration
|
||||
|
||||
`vite.config.js` is updated to proxy all `/api` requests to the backend, so the React dev server and production build do not need CORS configuration:
|
||||
`vite.config.js` proxies all `/api` requests to the Python backend:
|
||||
|
||||
```js
|
||||
// vite.config.js
|
||||
server: {
|
||||
host: true, // expose on LAN so other devices can reach the dev server
|
||||
proxy: {
|
||||
'/api': 'http://localhost:3001'
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
In production, Express serves the built React `dist/` as static files.
|
||||
In production, Flask serves the built React `dist/` as static files via `send_from_directory`. Build with `npm run build` inside `dashboard/`.
|
||||
|
||||
---
|
||||
|
||||
## 10. Implementation Phases
|
||||
## 10. Implementation Status
|
||||
|
||||
### Phase 1 — Backend Foundation
|
||||
1. Initialize `server/package.json`; install `express`, `better-sqlite3`, `chokidar`, `dotenv`
|
||||
2. Implement `db.js` — create schema, upsert helpers
|
||||
3. Implement `parser.js` — regex-based tag extraction from filenames and folder names
|
||||
4. Implement `scanner.js` — walk target dir to build test list; walk results dir to mark completions
|
||||
5. Implement `watcher.js` — chokidar watchers for both directories; call scanner on change
|
||||
6. Implement `sse.js` — maintain SSE client set; broadcast on update
|
||||
7. Wire up Express routes and start server
|
||||
### Completed
|
||||
- [x] Python Flask backend (`app.py`) with all API routes
|
||||
- [x] SQLite schema, WAL mode, thread-safe helpers (`db_py.py`)
|
||||
- [x] Tag parsing from target filenames and parent folder names (`parser.py`)
|
||||
- [x] Full directory scan on startup / config change (`scanner.py`)
|
||||
- [x] Watchdog filesystem watchers for both target and results dirs (`watcher.py`)
|
||||
- [x] Windows `is_directory` timing bug fixed (check path directly)
|
||||
- [x] Recycle Bin delete handled via `on_moved`
|
||||
- [x] SSE broadcast with correct `\n\n` terminators (`sse_py.py`)
|
||||
- [x] All frontend components and hooks
|
||||
- [x] Config modal: directories always editable (lock removed)
|
||||
- [x] Vite dev proxy + Flask static serving for production
|
||||
- [x] Config migrated from `config.json` → SQLite on first run
|
||||
|
||||
### Phase 2 — Frontend Core
|
||||
1. Update `vite.config.js` with API proxy
|
||||
### Remaining / Future
|
||||
- [ ] SSO / authentication (noted in design plan)
|
||||
- [ ] Test detail view (click a row to see tput/RSSI breakdown)
|
||||
- [ ] Production deployment docs (systemd / Task Scheduler service)
|
||||
2. Install `@tanstack/react-query`, `axios`, `tailwindcss`, `lucide-react`
|
||||
3. Build `api.js` — base fetch helpers + SSE subscription hook
|
||||
4. Build `useTests` and `useStats` hooks
|
||||
|
||||
Reference in New Issue
Block a user