Optimize performance across server, web and iOS #24
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "t3code/perf-optimize-all"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Performance pass across the server, web frontend and iOS app. Every item was measured, independently verified, implemented as its own commit (49, bisectable, each typechecks), and code-reviewed.
Results (seeded steady-state DB: 225k observations, 6k listings; back-to-back on the same host)
/api/dashboard/api/listings(500 rows)/api/exportpeak memoryServer
watch_listingswith indexed observation seeks (watch stats, baselines, analytics, watch analytics, listing-detail history); each has an old-SQL oracle test.migrations/030_perf_indexes.sql:observations(listing_id),connector_runs_recent, partialscans_unfinished; drops three unread indexes./api/*(SSE and/mcpexcluded); precompressed static assets; immutable caching for hashed assets./api/dashboard?top=Ncompact response for widgets (default response unchanged).scout_dashboardreturns 20 listings pluslistingsTruncated.Web
iOS
?top=12(463 KB → 10 KB), reload only when drawn content changes (176 → 10 reloads in a 2 h simulation), reuse the app's fresh snapshot, download only drawn photos.Before merging
workflow_dispatchon the GitHub mirror) or build in Xcode./api/debug/query); the rewritten queries rely on it returning 0:SELECT COUNT(*) FROM observations o LEFT JOIN watch_listings wl ON wl.id = o.watch_listing_id WHERE wl.id IS NULL OR wl.watch_id <> o.watch_id OR wl.listing_id <> o.listing_ida9c5a85; #18/#19 don't touch the measured paths.Checks on the tip: typecheck ✓ ·
npm test321/321 ✓ · build ✓ ·npm audit --omit=dev0 ✓ · ScoutKit 69/69 ✓ · smoke tests with auth off and on ✓.🤖 Generated with Claude Code
parseOlxOffer dropped offer.description and all but one photo, so every high-priority OLX verification fetched the www.olx.pl HTML offer page (0.3-1 MB, or a Chromium render when fenced) only to extract the same text. The parser now records { description, imageUrls } in a module-level WeakMap<NormalizedListing, OlxDetailHint> exposed through olxDetailHint(). The description goes through the new listingDescriptionText(), the same normalization parseListingDescription applies to a page candidate (text content, 8,000-character cap, boilerplate filter); photos are the first 12 gallery URLs at 1000x750, filtered like parseListingImageUrls. Nothing is added to NormalizedListing, so the text never reaches notification payloads or API JSON; a cloned listing has no hint and falls back to the page fetch. verifyHighPriorityDealOnce uses the hint when it carries a description (the Jev gallery takes its first 3 photos) and otherwise keeps today's page fetch, including the no-description 'unknown' semantics. The S2 freshness skip in processDealCandidates is unchanged and still runs first; the hint only removes the network fetch from verifications that do run, which covers the rejected/no-description alertable standing deals S2 left refetching. Behavior change (accepted by the item): OLX verification text is now the API description rather than the first HTML candidate, so a standing OLX deal whose normalized text differs gets a new snapshot state and one re-verification after deploy. Measured (bench-verify-refetch with the description the real API carries, 3 standing OLX deals, 5 scans, 500 KB detail page): passing deals 3 page fetches (1.4 MB) -> 0; rejected deals with a webhook 15 fetches (7.2 MB) -> 0. AI calls unchanged (3). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>scout_dashboard now returns { ...dashboard, listings: first 20, listingsTruncated: true when the feed held more }. Its description tells agents to page the full feed with scout_listings. The user approved this contract change. Stats, watches, connectors and lastScan are unchanged, and the REST /api/dashboard still returns the full 500-row feed. The new MCP test covers the exact-20 (not truncated) and 27-row (truncated) cases against service.dashboard(). scout_dashboard tool text on baseline.db: 461,998 -> 27,686 chars (JSON-RPC string 514.3 -> 31.1 KB). Against the pretty-printed original (595,557 chars), the change is -95%, roughly 150k -> 8k tokens per call. The server time is the same dashboard() call (26-32 ms in-process). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>F1: /api/export opens its read-only reader, runs BEGIN and reads settings synchronously in exportStream(), before the route sets any header, so an open failure throws in the handler and becomes a plain 500 instead of a JSON error body labelled Content-Encoding: br/gzip. The snapshot release is idempotent and also runs on stream close, so a stream destroyed before its first read still closes the reader. The compression hook also remembers the replies whose stream it is compressing and, if that stream fails before headers are sent, drops content-encoding (on the reply and the raw response) before handling the error payload. F2: mapPool stops taking new items once any call rejects; calls already in flight finish and the pool rejects with the first error, so a failed relevance or variant pass no longer keeps spending AI calls. F3: onClose clears the scheduler and other intervals first, then calls closeBrowser({ final: true }), which refuses later Chromium launches and waits at most 5 s for the launch/close, so the WAL checkpoint and db.close() always run even with a wedged browser. F4: the server SSR build emits stable chunk names (index.js, mcp.js, service.js) via a vite config branch on isSsrBuild, so a running server's lazy import('./mcp.js') still resolves after an in-place rebuild that fails before the restart. The MCP SDK stays lazily loaded. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>