fix(convex): scale the library scanner and search to large libraries #102
Closed
joshuafhiggins
wants to merge 2 commits from
t3/scale-scanner-and-search into main
pull from: t3/scale-scanner-and-search
merge into: joshuafhiggins:main
joshuafhiggins:main
joshuafhiggins:t3/mobile-lyrics-fill
joshuafhiggins:t3/remove-player-color-separator
joshuafhiggins:t3/block-android-dolby-atmos-downloads
joshuafhiggins:t3/low-quality-downloads
joshuafhiggins:t3/mobile-build-workflows
joshuafhiggins:t3/mobile-settings-redesign
joshuafhiggins:t3/ios-background-downloads
joshuafhiggins:t3/fix-medium-download-transcode
joshuafhiggins:t3/fuzzy-song-lyrics-search-1
joshuafhiggins:t3/match-mobile-button-styling
joshuafhiggins:t3/fix-library-album-navigation
joshuafhiggins:t3/lyrics-line-sweep
joshuafhiggins:t3/lyrics-fill-timing
joshuafhiggins:t3/revert-ios-inline-titles
joshuafhiggins:t3/lyrics-blur-fill
joshuafhiggins:t3/ios-inline-profile-button-main
joshuafhiggins:t3/ios-header-title-style
joshuafhiggins:t3/ios-inline-profile-button
joshuafhiggins:t3/fix-unfavorite-menu
joshuafhiggins:t3/queue-history-current-menu
joshuafhiggins:t3/electron-eac3-atmos-support
joshuafhiggins:t3/queue-context-menu
joshuafhiggins:t3/mobile-playlist-buttons
joshuafhiggins:t3/fix-mobile-profile-photos
joshuafhiggins:t3/lyrics-search-display
joshuafhiggins:t3/reposition-mobile-download-icon
joshuafhiggins:t3/move-downloads-under-songs
joshuafhiggins:t3/popup-screen
joshuafhiggins:t3/remove-action-menu-counts
joshuafhiggins:t3/match-mobile-queue-sidebar
joshuafhiggins:t3/offline-connection-screen
joshuafhiggins:t3/fix-uitab-availability
joshuafhiggins:t3/playlist-search-top-right
joshuafhiggins:t3/music-download-quality
joshuafhiggins:t3/playlist-cover-artwork
joshuafhiggins:fix/docker-pnpm-workspace
joshuafhiggins:t3/static-android-tab-ripple
joshuafhiggins:t3/fix-catalyst-handoff
joshuafhiggins:t3/favorites-off-users-document
joshuafhiggins:fix/duplicate-ionicons-import
joshuafhiggins:t3/simplify-android-transitions
joshuafhiggins:t3/review-prs-102-91
joshuafhiggins:t3/ios-playlist-song-removal
joshuafhiggins:t3/reduce-player-rerenders
joshuafhiggins:t3/share-duplicated-logic
joshuafhiggins:t3/incremental-siri-catalog
joshuafhiggins:t3/split-users-document
joshuafhiggins:t3/auth-song-queries
joshuafhiggins:t3/virtualize-web-song-lists
joshuafhiggins:t3/consolidate-web-track-rows
joshuafhiggins:t3/split-mobile-playback
joshuafhiggins:t3/general-cleanup
joshuafhiggins:t3/system-color-menu-button
joshuafhiggins:t3/clean-up-loading-skeletons
joshuafhiggins:t3/activate-audio-on-play
joshuafhiggins:t3/fix-streaming-leaks
joshuafhiggins:t3/harden-docker
joshuafhiggins:t3/fix-scanner-file-paths
joshuafhiggins:t3/song-swipe-queue-action
joshuafhiggins:t3/pause-after-queue-end
joshuafhiggins:t3/align-player-context-action-order
joshuafhiggins:t3/ios-context-menu-previews
joshuafhiggins:t3/mobile-player-artist-menu
joshuafhiggins:t3/prevent-swiped-song-play
joshuafhiggins:t3/fix-song-sharing-interactions
joshuafhiggins:t3/ipad-sidebar-layout
joshuafhiggins:t3/match-album-color-algorithm
joshuafhiggins:t3/local-settings-toggle
joshuafhiggins:t3/add-macos-catalyst-support
joshuafhiggins:feat/native-apple-handoff
joshuafhiggins:t3/ios-zoom-album-playlist
joshuafhiggins:t3/fix-android-page-transitions
joshuafhiggins:t3/restart-collection-playback
joshuafhiggins:t3/fix-android-tab-tap-flash
joshuafhiggins:t3/fix-android-separator-width
joshuafhiggins:fix/mobile-icon-build-names
joshuafhiggins:t3/compact-web-player
joshuafhiggins:t3/web-playlist-search
joshuafhiggins:t3/open-app-mobile-only
joshuafhiggins:t3/mobile-playlist-search
joshuafhiggins:t3/mobile-deep-links-share-art
joshuafhiggins:t3/fix-mobile-skeleton-bounds
joshuafhiggins:t3/shuffle-album-playlist-selection
joshuafhiggins:t3/web-player-light-mode
joshuafhiggins:t3/preserve-scroll-back-navigation
joshuafhiggins:t3/album-audio-format-icons
joshuafhiggins:t3/remove-mobile-pagination
joshuafhiggins:t3/disable-android-atmos
joshuafhiggins:t3/mobile-sidebar-toggle
joshuafhiggins:t3/fix-context-menu-system-colors
joshuafhiggins:t3/restore-paused-audio-session
joshuafhiggins:t3/fix-airplay-picker-taps
joshuafhiggins:t3/fix-android-native-player-scrubbing
joshuafhiggins:t3/fix-player-menu-navigation
joshuafhiggins:t3/profile-app-identity
joshuafhiggins:t3/keep-airplay-button-style
joshuafhiggins:t3/mobile-admin-account-creation
joshuafhiggins:t3/fix-player-swipe-black-screen
joshuafhiggins:t3/animate-volume-controls
joshuafhiggins:t3/ios-alac-android-atmos
joshuafhiggins:t3/mobile-system-volume-control
joshuafhiggins:t3/android-layout-and-colors
joshuafhiggins:t3/mobile-native-airplay-cast-ui
joshuafhiggins:t3/create-station-context-menu
joshuafhiggins:t3/remove-mobile-splash-fix-server-navigation
joshuafhiggins:t3/fix-mobile-content-disappearing
joshuafhiggins:t3/fix-expo-audio-docker-builds
joshuafhiggins:t3/update-expo-packages-audio-patches
joshuafhiggins:t3/remove-mobile-toasts
joshuafhiggins:t3/apple-music-now-playing-view
joshuafhiggins:t3/center-wrapped-artists
joshuafhiggins:t3/fix-screen-padding-1
joshuafhiggins:t3/hide-profile-subviews
joshuafhiggins:t3/mobile-track-layout-discs
joshuafhiggins:t3/ios-native-swipe-remove
joshuafhiggins:t3code/apple-music-cover-colors
joshuafhiggins:t3code/fix-carousel-buttons
joshuafhiggins:t3code/admin-create-users
joshuafhiggins:t3code/reduce-docker-image-size
joshuafhiggins:t3code/client-first-playback-caching
joshuafhiggins:t3code/fix-auth-pages-convex-url
joshuafhiggins:t3code/docker-self-hosted-convex
joshuafhiggins:t3code/server-side-auth-pages
joshuafhiggins:t3code/hide-profile-nonprimary-pages
Reference in new issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
The library scanner and search fall over on large libraries. The scanner loaded every artist, album and song through
runQueryabout 1 + 3×codecs times per run, stat'd every song file, and used O(n²) lookups, so past Convex's per-query read limits the 15-minute cron failed every run. Untitled albums were created empty, deleted next run and recreated with a new ID (dropping their favorites). The admin button and the cron could run at the same time and delete each other's fresh rows. Search,browseSongsand the library play queue collected the whole library on every call.Scanner (
actions.ts)readdirper directory, no per-songexistsSync). Everything else is Map/Set lookups.importAlbumscreates the artist, album and songs together, so an album never exists without songs. Untitled files fall back to the filename.0/""(same as #91, which this supersedes foractions.ts).libraryScanLock) means only one scan runs at a time. The admin button now errors with "A library scan is already running" instead of racing the cron. Empty album/artist deletes re-check emptiness inside the mutation.Search and browse (
library.ts)search_namesearch indexes on artists, albums and songs. Search takes capped hits (20/40/100) plus the albums/songs of the top 3 matched artists/albums. Full-text hits only count when every query word prefixes a word of the name.browseSongsis capped at 500.browseSortedSongIds/playFromLibrarySongreturn up to 1000 IDs starting at the tapped song, in the same order as the paginated list. Web and mobile passfromSongId.Unknown years
Untagged years are stored as
0and dates as"". The album detail query returnsyear: nulland the Siri catalog sends an empty year in that case, so web, mobile and iOS intents all hide them. Rows already saved with the old 2026 placeholder are not backfilled.Known gap: track artists (the
artistfield,string | string[]) aren't search indexed, so a query that only matches a song's track artist (not its title, album or album artist) won't find it.Testing
Self-hosted Convex backend with a synthetic library of 22,000 MP3s (500 artists × 4 albums × 10 songs, 10% also in
lossless/, one untitled album):browseSongsand the sorted queue window checked against the same data.Changes made by Claude Opus 5.5 running in Claude Code (T3 Code).
🤖 Generated with Claude Code
Pull request closed