fix(mobile): sync Siri catalog incrementally #101

Merged
joshuafhiggins merged 2 commits from t3/incremental-siri-catalog into main 2026-11-16 21:57:54 -05:00
Owner

The Siri/App Intents catalog re-downloaded the entire library on mount, every foreground, every 5 minutes, and on every favorite change. Each page also did per-row gets and read the user document, so a like invalidated every cached page.

How

Server (convex/intentCatalog.ts)

  • head: the newest insert, rename or deletion per kind (song, album, artist). Clients subscribe to it.
  • page: rows created after since (by_creation_time), plus renames and deletions from a new intentCatalogChanges log. Album and artist lookups are batched and deduplicated. Returns reset if the cursor is older than the log's 30-day retention or has more than 2,000 changes behind it.
  • playlists: own playlists via by_owner, plus subscribed ones. No more paging every user's playlists.
  • Library deletes and album renames write to the change log; a daily cron prunes it.
  • Catalog records carry no per-user state.

Mobile

  • components/app-intents/catalog-sync.ts fetches a kind only when head passes the cursor stored in AsyncStorage, merges into the disk catalog, and stamps metadata.favorite locally. The 5-minute interval is gone. Foreground only retries, with no queries when the catalog is current. Cursors are cleared on sign-out.
  • app-intent-bridge.tsx (822 lines) split into catalog-sync.ts, now-playing.ts and dispatch.ts.
  • Siri like/unlike and add-to-Favorites now go through useFavorites() (new throwing setFavorite/addFavoriteSongs), so they get the same optimistic updates. The bridge moved inside FavoritesProvider.

Measurements

Local self-hosted backend, 5,005 songs / 501 albums / 51 artists:

queries payload
main, every trigger (mount, foreground, 5 min, any like) 31 1,808 KiB
branch, first sync on a fresh device 16 1,714 KiB
branch, nothing changed 0 new (2 live subscriptions) ~0
branch, favorite change 0 0
branch, one album renamed (1 album + 10 songs) 4 4 KiB

Deletion reaches clients via removed, and a stale cursor returns reset. Both checked against the local backend.

Simulator verification (iPhone 18 Pro, iOS 27.2, dev build, local backend)

  • Fresh sign-in synced 51 artists, 501 albums, 5,005 songs and playlists to disk, and saved cursors. A relaunch reused the artist cursor and did not refetch it.
  • An album rename, a deletion and a re-added file all matched the server on device.
  • Liking a song in the app flipped metadata.favorite in song.json within a second, with no catalog fetch.
  • Shortcuts: entity suggestions showed the favorited song first and the renamed album. Search in the Artist picker found "Siri Test Band".
  • Play Music (song) ran through dispatch and reported "Playing Morning Light."
  • Love Song and Unlike Song on the current song updated both server favorites and the on-device catalog.

Note: Spotlight indexing of the album suggestions takes about 2 minutes on the simulator, and the rest of the first sync waits behind it. That cost is unchanged from main, but now it is paid once instead of on every trigger.

Checks: pnpm --filter @otunes/mobile typecheck, pnpm --filter @otunes/convex typecheck, and the existing node --test intent tests pass.


Changes made by Claude Opus 5.5 (claude-opus-5-5) in Claude Code via T3 Code.

🤖 Generated with Claude Code

The Siri/App Intents catalog re-downloaded the entire library on mount, every foreground, every 5 minutes, and on every favorite change. Each page also did per-row `get`s and read the user document, so a like invalidated every cached page. ## How **Server** (`convex/intentCatalog.ts`) - `head`: the newest insert, rename or deletion per kind (song, album, artist). Clients subscribe to it. - `page`: rows created after `since` (`by_creation_time`), plus renames and deletions from a new `intentCatalogChanges` log. Album and artist lookups are batched and deduplicated. Returns `reset` if the cursor is older than the log's 30-day retention or has more than 2,000 changes behind it. - `playlists`: own playlists via `by_owner`, plus subscribed ones. No more paging every user's playlists. - Library deletes and album renames write to the change log; a daily cron prunes it. - Catalog records carry no per-user state. **Mobile** - `components/app-intents/catalog-sync.ts` fetches a kind only when `head` passes the cursor stored in AsyncStorage, merges into the disk catalog, and stamps `metadata.favorite` locally. The 5-minute interval is gone. Foreground only retries, with no queries when the catalog is current. Cursors are cleared on sign-out. - `app-intent-bridge.tsx` (822 lines) split into `catalog-sync.ts`, `now-playing.ts` and `dispatch.ts`. - Siri like/unlike and add-to-Favorites now go through `useFavorites()` (new throwing `setFavorite`/`addFavoriteSongs`), so they get the same optimistic updates. The bridge moved inside `FavoritesProvider`. ## Measurements Local self-hosted backend, 5,005 songs / 501 albums / 51 artists: | | queries | payload | |---|---|---| | main, every trigger (mount, foreground, 5 min, any like) | 31 | 1,808 KiB | | branch, first sync on a fresh device | 16 | 1,714 KiB | | branch, nothing changed | 0 new (2 live subscriptions) | ~0 | | branch, favorite change | 0 | 0 | | branch, one album renamed (1 album + 10 songs) | 4 | 4 KiB | Deletion reaches clients via `removed`, and a stale cursor returns `reset`. Both checked against the local backend. ## Simulator verification (iPhone 18 Pro, iOS 27.2, dev build, local backend) - Fresh sign-in synced 51 artists, 501 albums, 5,005 songs and playlists to disk, and saved cursors. A relaunch reused the artist cursor and did not refetch it. - An album rename, a deletion and a re-added file all matched the server on device. - Liking a song in the app flipped `metadata.favorite` in `song.json` within a second, with no catalog fetch. - Shortcuts: entity suggestions showed the favorited song first and the renamed album. Search in the Artist picker found "Siri Test Band". - Play Music (song) ran through dispatch and reported "Playing Morning Light." - Love Song and Unlike Song on the current song updated both server favorites and the on-device catalog. Note: Spotlight indexing of the album suggestions takes about 2 minutes on the simulator, and the rest of the first sync waits behind it. That cost is unchanged from main, but now it is paid once instead of on every trigger. Checks: `pnpm --filter @otunes/mobile typecheck`, `pnpm --filter @otunes/convex typecheck`, and the existing `node --test` intent tests pass. --- Changes made by Claude Opus 5.5 (`claude-opus-5-5`) in Claude Code via T3 Code. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
joshuafhiggins added 1 commit 2026-11-16 14:16:51 -05:00
The App Intents catalog re-downloaded the whole library on mount, every
foreground, every 5 minutes and on every favorite change.

- New `intentCatalog` module: a live `head` per kind, change-since pages
  backed by an `intentCatalogChanges` log for renames and deletions, and an
  indexed playlists query. Lookups are batched and deduplicated.
- The client fetches a kind only when its head passes the stored cursor and
  stamps favorites locally, so likes cost no catalog queries.
- Split app-intent-bridge into catalog sync, now playing and dispatch.
- Siri favorites go through the optimistic favorites context.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
joshuafhiggins added 1 commit 2026-11-16 21:57:48 -05:00
# Conflicts:
#	packages/convex/convex/library.ts
joshuafhiggins merged commit 69c9ffca1f into main 2026-11-16 21:57:54 -05:00
Sign in to join this conversation.
No Reviewers
No labels
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: joshuafhiggins/otunes#101