fix(convex): move favorites off the users document #113

Merged
joshuafhiggins merged 2 commits from t3/favorites-off-users-document into main 2026-11-17 11:14:49 -05:00
Owner

Favorites were still three arrays on users (left behind by #100). Every toggle rewrote the whole user doc and reran every query that reads users, and big libraries could hit Convex's 8,192-value array limit. Moving favorites into a table also showed that several queries, and the favorites writes themselves, break on big libraries: they hit the same array limit or Convex's 4,096 reads per function.

Favorites table

  • New favorites table, one row per favorite: { userId, kind, entityId }, typed as a union on kind so entityId is always the right Id<>. Indexes:
    • by_user_kind: a user's songs/albums/artists in favorited order (_creationTime). This keeps the Favorites playlist (otunes:favorites) order.
    • by_user_entity: "is this favorited" lookups for small toggles. kind isn't needed since the id already names its table.
    • by_entity: the scanner purge helpers delete rows for deleted media via the index instead of scanning every user.
      I picked one table over three per-kind tables because every reader wants all three lists together anyway, and the union still gives typed ids.
  • convex/favoriteStore.ts holds the read/write helpers, like playbackState.ts did for #100.
  • Migration: migrations.migrateUsers schedules migrations.migrateFavorites per user with legacy arrays. It copies rows in array order, dedupes, then clears the legacy fields. Every favorites mutation migrates the caller first. Reads fall back to the legacy arrays only when a user has no rows at all, so migrated users' favorite queries don't read users. Legacy fields stay optional in the schema with a comment.
  • Account deletion deletes the user's favorite rows.

Large libraries

Array limit. Queries that return a whole library or favorites list now return Chunked<T> (T[][], each chunk at most 8,192, from chunk() in limits.ts). Clients call .flat(), which is also a no-op on old flat values still in a persisted cache:

  • favorites.getUserFavorites (each list) and favorites.browseFavoritesPlaylistDetail (songs, now BrowseFavoritesDetail)
  • library.browseSortedSongIds, browseAlbums, browseArtists, browseRecentlyAdded
  • playlists.getArtistSongIds, getOrderedArtistSongIds (a big "Various Artists" can pass 8k)

library.searchLibrary keeps the top 500 results per section instead of chunking. A query like "a" matches most of the library and nobody scrolls that far.

4,096 reads per function. Each db.get, index lookup and db.delete counts as one read, and the first version of this PR did one per id:

  • Favorite writes and cascade checks now use one helper. It does index lookups for up to 64 ids, and one scan of the user's list above that.
  • browseSongsFromIds scans the songs/albums/artists tables once when asked for more than 1,000 songs, as search and the sorted-ids query already do. This also fixes regular playlists over about 4k songs.
  • Unfavoriting deletes up to 1,000 rows inline and schedules the rest (favoriteStore.deleteRows) in batches of 1,000.

Shapes changed: the queries listed under the array limit above. Web and mobile are updated; older builds will break against these queries.

Known gap: after a large unfavorite (more than 1,000 rows), the scheduled deletes take about a second. If the user re-favorites the same items before those deletes finish, they still remove the old rows, and those favorites are lost.

Testing

Against a local self-hosted backend (same image as the Dockerfile):

  • Migration: deployed main and seeded legacy favorites, then deployed this branch. Checked the read fallback, and that migrateUsers keeps order, drops duplicates, clears legacy fields and is a no-op on rerun.
  • Favorite behavior:
    • song→album→artist promotion and the unfavorite cascades;
    • library:deleteSongs purging rows;
    • a toggle by an unmigrated user migrating them in the same mutation.
  • Large library: seeded 9,000 songs under one artist. Favoriting the artist gives getUserFavorites and the favorites detail chunks of [8192, 808]. browseSortedSongIds and getOrderedArtistSongIds return the same. Search for "song" returns 500. Unfavoriting the album or artist clears all 9,000 rows within a couple of seconds.

pnpm -r typecheck, web lint (no new warnings) and the convex node tests pass.


Changes made by Claude Opus 5.5 (claude-opus-5-5) in Claude Code, run through T3 Code.

🤖 Generated with Claude Code

Favorites were still three arrays on `users` (left behind by #100). Every toggle rewrote the whole user doc and reran every query that reads `users`, and big libraries could hit Convex's 8,192-value array limit. Moving favorites into a table also showed that several queries, and the favorites writes themselves, break on big libraries: they hit the same array limit or Convex's 4,096 reads per function. ## Favorites table - New `favorites` table, one row per favorite: `{ userId, kind, entityId }`, typed as a union on `kind` so `entityId` is always the right `Id<>`. Indexes: - `by_user_kind`: a user's songs/albums/artists in favorited order (`_creationTime`). This keeps the Favorites playlist (`otunes:favorites`) order. - `by_user_entity`: "is this favorited" lookups for small toggles. `kind` isn't needed since the id already names its table. - `by_entity`: the scanner purge helpers delete rows for deleted media via the index instead of scanning every user. I picked one table over three per-kind tables because every reader wants all three lists together anyway, and the union still gives typed ids. - `convex/favoriteStore.ts` holds the read/write helpers, like `playbackState.ts` did for #100. - Migration: `migrations.migrateUsers` schedules `migrations.migrateFavorites` per user with legacy arrays. It copies rows in array order, dedupes, then clears the legacy fields. Every favorites mutation migrates the caller first. Reads fall back to the legacy arrays only when a user has no rows at all, so migrated users' favorite queries don't read `users`. Legacy fields stay optional in the schema with a comment. - Account deletion deletes the user's favorite rows. ## Large libraries **Array limit.** Queries that return a whole library or favorites list now return `Chunked<T>` (`T[][]`, each chunk at most 8,192, from `chunk()` in `limits.ts`). Clients call `.flat()`, which is also a no-op on old flat values still in a persisted cache: - `favorites.getUserFavorites` (each list) and `favorites.browseFavoritesPlaylistDetail` (`songs`, now `BrowseFavoritesDetail`) - `library.browseSortedSongIds`, `browseAlbums`, `browseArtists`, `browseRecentlyAdded` - `playlists.getArtistSongIds`, `getOrderedArtistSongIds` (a big "Various Artists" can pass 8k) `library.searchLibrary` keeps the top 500 results per section instead of chunking. A query like "a" matches most of the library and nobody scrolls that far. **4,096 reads per function.** Each `db.get`, index lookup and `db.delete` counts as one read, and the first version of this PR did one per id: - Favorite writes and cascade checks now use one helper. It does index lookups for up to 64 ids, and one scan of the user's list above that. - `browseSongsFromIds` scans the songs/albums/artists tables once when asked for more than 1,000 songs, as search and the sorted-ids query already do. This also fixes regular playlists over about 4k songs. - Unfavoriting deletes up to 1,000 rows inline and schedules the rest (`favoriteStore.deleteRows`) in batches of 1,000. **Shapes changed:** the queries listed under the array limit above. Web and mobile are updated; older builds will break against these queries. **Known gap:** after a large unfavorite (more than 1,000 rows), the scheduled deletes take about a second. If the user re-favorites the same items before those deletes finish, they still remove the old rows, and those favorites are lost. ## Testing Against a local self-hosted backend (same image as the Dockerfile): - **Migration:** deployed `main` and seeded legacy favorites, then deployed this branch. Checked the read fallback, and that `migrateUsers` keeps order, drops duplicates, clears legacy fields and is a no-op on rerun. - **Favorite behavior:** - song→album→artist promotion and the unfavorite cascades; - `library:deleteSongs` purging rows; - a toggle by an unmigrated user migrating them in the same mutation. - **Large library:** seeded 9,000 songs under one artist. Favoriting the artist gives `getUserFavorites` and the favorites detail chunks of `[8192, 808]`. `browseSortedSongIds` and `getOrderedArtistSongIds` return the same. Search for "song" returns 500. Unfavoriting the album or artist clears all 9,000 rows within a couple of seconds. `pnpm -r typecheck`, web lint (no new warnings) and the convex node tests pass. --- Changes made by Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code, run through T3 Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
joshuafhiggins added 1 commit 2026-11-16 22:19:38 -05:00
Favorites now live in a `favorites` table, one row per favorite.
`migrations.migrateUsers` schedules a per-user copy of the legacy arrays,
and writes migrate the caller first. Public query and mutation shapes are
unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
joshuafhiggins force-pushed t3/favorites-off-users-document from 0fcae7ce72 to b4114db046 2026-11-16 22:19:38 -05:00 Compare
joshuafhiggins added 1 commit 2026-11-17 10:44:20 -05:00
Whole-library and favorites queries return lists split into chunks of at
most 8,192 values, and search keeps the top 500 per section. Favorite
writes and song detail lookups read whole ranges instead of one id at a
time, and large unfavorites finish in scheduled batches, so they stay
under Convex's 4,096 reads per function.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
joshuafhiggins merged commit 5dcbe6b9c5 into main 2026-11-17 11:14:49 -05:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: joshuafhiggins/otunes#113