fix(convex): scale the library scanner and search to large libraries #102

Closed
joshuafhiggins wants to merge 2 commits from t3/scale-scanner-and-search into main
pull from: t3/scale-scanner-and-search
Owner

The library scanner and search fall over on large libraries. The scanner loaded every artist, album and song through runQuery about 1 + 3×codecs times per run, stat'd every song file, and used O(n²) lookups, so past Convex's per-query read limits the 15-minute cron failed every run. Untitled albums were created empty, deleted next run and recreated with a new ID (dropping their favorites). The admin button and the cron could run at the same time and delete each other's fresh rows. Search, browseSongs and the library play queue collected the whole library on every call.

Scanner (actions.ts)

  • Reads artists/albums/songs once via paginated internal queries and walks the disk once (one readdir per directory, no per-song existsSync). Everything else is Map/Set lookups.
  • Parses only untracked files, with a 6 minute parse budget; anything left over is picked up next run.
  • Writes only what changed, in batches. A re-scan of an unchanged library makes no library writes.
  • importAlbums creates the artist, album and songs together, so an album never exists without songs. Untitled files fall back to the filename.
  • A renamed or moved file inside an album updates the existing song (same ID, favorites kept) instead of delete + recreate.
  • Same-titled sibling files no longer steal a tracked path, and covers keep the stored file while it exists instead of ping-ponging between codecs. Missing years are stored as 0 / "" (same as #91, which this supersedes for actions.ts).
  • Refuses to run if no audio files are found at all, so an unmounted volume can't empty the library.
  • A lease lock (libraryScanLock) means only one scan runs at a time. The admin button now errors with "A library scan is already running" instead of racing the cron. Empty album/artist deletes re-check emptiness inside the mutation.

Search and browse (library.ts)

  • search_name search indexes on artists, albums and songs. Search takes capped hits (20/40/100) plus the albums/songs of the top 3 matched artists/albums. Full-text hits only count when every query word prefixes a word of the name.
  • browseSongs is capped at 500.
  • browseSortedSongIds / playFromLibrarySong return up to 1000 IDs starting at the tapped song, in the same order as the paginated list. Web and mobile pass fromSongId.

Unknown years

Untagged years are stored as 0 and dates as "". The album detail query returns year: null and the Siri catalog sends an empty year in that case, so web, mobile and iOS intents all hide them. Rows already saved with the old 2026 placeholder are not backfilled.

Known gap: track artists (the artist field, string | string[]) aren't search indexed, so a query that only matches a song's track artist (not its title, album or album artist) won't find it.

Testing

Self-hosted Convex backend with a synthetic library of 22,000 MP3s (500 artists × 4 albums × 10 songs, 10% also in lossless/, one untitled album):

  • Initial import: 20,000 songs in 66s (lossless copies merged onto the same songs).
  • Unchanged re-scan: 3.3s, exported songs/albums/artists byte-identical before and after.
  • Removing a codec copy, deleting an artist, renaming a file, adding an album and removing a cover: each produced exactly the expected writes. The renamed song kept its ID.
  • A same-titled sibling file: no writes across repeated scans.
  • A manual scan while the cron was running was refused by the lock.
  • Search, browseSongs and the sorted queue window checked against the same data.

Changes made by Claude Opus 5.5 running in Claude Code (T3 Code).

🤖 Generated with Claude Code

The library scanner and search fall over on large libraries. The scanner loaded every artist, album and song through `runQuery` about 1 + 3×codecs times per run, stat'd every song file, and used O(n²) lookups, so past Convex's per-query read limits the 15-minute cron failed every run. Untitled albums were created empty, deleted next run and recreated with a new ID (dropping their favorites). The admin button and the cron could run at the same time and delete each other's fresh rows. Search, `browseSongs` and the library play queue collected the whole library on every call. ## Scanner (`actions.ts`) - Reads artists/albums/songs once via paginated internal queries and walks the disk once (one `readdir` per directory, no per-song `existsSync`). Everything else is Map/Set lookups. - Parses only untracked files, with a 6 minute parse budget; anything left over is picked up next run. - Writes only what changed, in batches. **A re-scan of an unchanged library makes no library writes.** - `importAlbums` creates the artist, album and songs together, so an album never exists without songs. Untitled files fall back to the filename. - A renamed or moved file inside an album updates the existing song (same ID, favorites kept) instead of delete + recreate. - Same-titled sibling files no longer steal a tracked path, and covers keep the stored file while it exists instead of ping-ponging between codecs. Missing years are stored as `0` / `""` (same as #91, which this supersedes for `actions.ts`). - Refuses to run if no audio files are found at all, so an unmounted volume can't empty the library. - A lease lock (`libraryScanLock`) means only one scan runs at a time. The admin button now errors with "A library scan is already running" instead of racing the cron. Empty album/artist deletes re-check emptiness inside the mutation. ## Search and browse (`library.ts`) - `search_name` search indexes on artists, albums and songs. Search takes capped hits (20/40/100) plus the albums/songs of the top 3 matched artists/albums. Full-text hits only count when every query word prefixes a word of the name. - `browseSongs` is capped at 500. - `browseSortedSongIds` / `playFromLibrarySong` return up to 1000 IDs starting at the tapped song, in the same order as the paginated list. Web and mobile pass `fromSongId`. ## Unknown years Untagged years are stored as `0` and dates as `""`. The album detail query returns `year: null` and the Siri catalog sends an empty year in that case, so web, mobile and iOS intents all hide them. Rows already saved with the old 2026 placeholder are not backfilled. Known gap: track artists (the `artist` field, `string | string[]`) aren't search indexed, so a query that only matches a song's track artist (not its title, album or album artist) won't find it. ## Testing Self-hosted Convex backend with a synthetic library of 22,000 MP3s (500 artists × 4 albums × 10 songs, 10% also in `lossless/`, one untitled album): - Initial import: 20,000 songs in 66s (lossless copies merged onto the same songs). - Unchanged re-scan: 3.3s, exported songs/albums/artists byte-identical before and after. - Removing a codec copy, deleting an artist, renaming a file, adding an album and removing a cover: each produced exactly the expected writes. The renamed song kept its ID. - A same-titled sibling file: no writes across repeated scans. - A manual scan while the cron was running was refused by the lock. - Search, `browseSongs` and the sorted queue window checked against the same data. --- Changes made by Claude Opus 5.5 running in Claude Code (T3 Code). 🤖 Generated with [Claude Code](https://claude.com/claude-code)
joshuafhiggins added 1 commit 2026-11-16 14:19:08 -05:00
The scanner loaded every artist, album and song several times per run with
full collects, so large libraries hit Convex read limits and every cron run
failed. Search and browse also collected the whole library per call.

- Scanner reads the library once in pages, walks the disk once, parses only
  untracked files and writes only what changed. A re-scan of an unchanged
  library makes no library writes.
- Albums are created together with their songs; untitled files fall back to
  the filename so their album is never left empty and recreated.
- A lease lock keeps the cron and the admin button from running concurrently;
  empty album/artist deletes re-check inside the mutation.
- Search uses name search indexes with capped results; browseSongs and the
  library play queue are bounded.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
joshuafhiggins added 1 commit 2026-11-16 15:31:51 -05:00
Unknown years are stored as 0 and dates as "". Album detail now returns a
null year and the Siri catalog sends an empty year instead of "0", so no
client has to special-case 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
joshuafhiggins closed this pull request 2026-11-16 20:21:16 -05:00

Pull request closed

Please reopen this pull request to perform a merge.
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: joshuafhiggins/otunes#102