RinwaProduct & architecture Back to documentation

Technical reference

Story clustering (rev-story-clustering) — design

The deterministic local clustering design behind duplicate-story collapse and the most-reported view.

Generated from the product repository Source updated 2026-07-30 View source on GitHub ↗

Story clustering (rev-story-clustering) — design#

Status: approved (design) Date: 2026-07-30 Owner area: apps/api (clustering engine, cluster store, sync), apps/web (inline collapsed rows, hot-topics panel)

Goal#

When several sources report the same event, collapse them into one row in the normal reading list instead of repeating the story N times, and expose a "most-reported" panel so the user can see which stories got the widest coverage today.

Two distinct jobs:

  1. Inline collapse — noise reduction. The everyday list shows one row per story with a "+N sources" affordance.
  2. Hot topics panel — discovery. The same cluster data, sorted by member count: "what was reported by the most sources". Nearly free once clustering exists.

Clustering is deterministic and local: embedding cosine similarity above a threshold, within a time window. No training, no server, no opaque ranking.

Product role#

Story clustering is one half of Rinwa's semantic reading pillar, alongside hybrid semantic search. Search answers “where have I read this idea before?”; clustering answers “which incoming articles are actually the same story?”

The semantic grouping decision is embedding-only in this design. It does not ask an LLM to classify each article, invent topics, or summarize a cluster. The time window is an eligibility boundary rather than a second semantic classifier. Feed metadata is retained for attribution, chronology and display, while the similarity threshold decides whether two eligible articles describe the same story.

This distinction matters for product communication: Rinwa can truthfully describe local embeddings as the engine for news aggregation without implying that vectors replace publication time, source identity, user corrections, or the original articles.

Non-goals#

  • No clustering of historical/backlog articles. Only a recent window is clustered (see below); older items are never touched or reordered.
  • No cross-language "same story" matching beyond what the embedding model naturally provides.
  • No topic naming/summarization of a cluster (no AI-generated titles). The primary member's own title represents the cluster.
  • No manual "merge these two" action. Only splitting a wrong cluster.

Prerequisites & the backlog problem#

Embeddings are an optional feature, and vectors are only produced for items ingested after it is enabled (enqueueEmbeddingIndex is called from ingestion-service and content-fetch-queue; there is no backfill path).

Therefore:

  • Clustering requires embeddings to be enabled. When they are not, the feature is off and the UI says so plainly (with a link to enable it) rather than silently doing nothing.
  • On enabling clustering, backfill vectors for items in the clustering window only — items within the last N days that have no embedding are enqueued for indexing. This is a bounded job (hundreds of items, not the whole archive), and it makes the feature work immediately instead of "correct itself over the next few days".

Clustering rules#

  • Window: only items whose COALESCE(published_at, created_at) is within N days (default 3, configurable in settings) are eligible. This is the span over which duplicate coverage of one event actually happens.
  • Similarity: two items belong together when their embedding cosine similarity ≥ threshold (default 0.86, configurable). Comparisons are restricted to candidates inside the window, so this is not an N² scan of the archive.
  • Grouping: single-link agglomeration within the window — an item joins the cluster of the most similar item above the threshold; otherwise it forms its own (size-1) cluster. Size-1 clusters are not shown as clusters.
  • Muted items never participate in clustering (they're already hidden).
  • Clustering runs after ingestion for newly inserted items, and on demand when the window/threshold settings change.

Presentation#

Inline (the normal list)#

  • A cluster renders as one row. The primary member is the earliest published item in the cluster — "who broke it" — and its title/summary represents the cluster.
  • The row's sort position follows the most recent member, so a cluster with fresh follow-up coverage rises back to the top. (Explicitly accepted tradeoff: the list can re-order as follow-ups arrive; in exchange, active stories stay visible.)
  • The row shows "+N sources"; expanding lists every member with its feed, and any member can be opened and read normally.
  • Read propagation: reading any member marks the whole cluster read. Same event, already read — this is the point of the feature, and it keeps unread counts honest.
  • A new member joining an already-read cluster does NOT make it unread again. If it was similar enough to cluster, it is duplicate coverage; re-surfacing it would undo the noise reduction. (The cluster still moves per the sort rule above, so it remains discoverable.)
  • Unread counts count a cluster as one unread item, matching what is shown.

Hot topics panel#

  • A separate view listing clusters within the window, sorted by member count descending (ties broken by most recent member), showing the primary title and the contributing sources.
  • Only clusters of size ≥ 2 appear. Reading from here behaves like the inline list (opens a member; the cluster becomes read).
  • This view never hides anything — it is a lens over the same data.

Explainability & escape hatch#

  • A cluster exposes why these were grouped: the member titles/sources and the similarity basis (e.g. "grouped by content similarity ≥ 0.86, within 3 days").
  • "Split this cluster" breaks it apart: members become individual rows again, and the pairing is remembered so they are never re-clustered together. This is the answer to a wrong merge — the failure mode users actually resent.
  • Splitting is user data (a deliberate correction), not a disposable cache: it is preserved across re-clustering runs and synced.

Data model#

story_clusters#

CREATE TABLE IF NOT EXISTS story_clusters (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  global_id TEXT NOT NULL UNIQUE,
  primary_item_id INTEGER NOT NULL,
  member_count INTEGER NOT NULL DEFAULT 1,
  earliest_at TEXT NOT NULL DEFAULT '',
  latest_at TEXT NOT NULL DEFAULT '',
  created_at TEXT NOT NULL,
  updated_at TEXT NOT NULL,
  FOREIGN KEY(primary_item_id) REFERENCES items(id) ON DELETE CASCADE
);

story_cluster_members#

CREATE TABLE IF NOT EXISTS story_cluster_members (
  cluster_id INTEGER NOT NULL,
  item_id INTEGER NOT NULL,
  PRIMARY KEY(cluster_id, item_id),
  FOREIGN KEY(cluster_id) REFERENCES story_clusters(id) ON DELETE CASCADE,
  FOREIGN KEY(item_id) REFERENCES items(id) ON DELETE CASCADE
);
CREATE INDEX IF NOT EXISTS idx_story_cluster_members_item ON story_cluster_members(item_id);

story_cluster_splits (user corrections — durable)#

CREATE TABLE IF NOT EXISTS story_cluster_splits (
  item_global_id_a TEXT NOT NULL,
  item_global_id_b TEXT NOT NULL,
  created_at TEXT NOT NULL,
  PRIMARY KEY(item_global_id_a, item_global_id_b)
);

Pairs are stored with the two global ids sorted so the pair is order-independent. Clusters and members are derived data (recomputable, not synced). Splits are user data and are synced via a story_cluster_split op-log entity, so a correction made on one device holds everywhere.

Settings (enabled, windowDays, threshold) live with the other feature settings and sync like them.

Engine — apps/api/src/services/story-cluster-service.ts#

  • clusterRecentStories() — recompute clusters for the window: select eligible items (in-window, not muted, having an embedding), compare within the window, apply the split blocklist, then rewrite story_clusters / story_cluster_members in one transaction. Recomputation must be idempotent and must not disturb read state.
  • splitCluster(clusterGlobalId) — record split pairs for every member pairing in that cluster, delete the cluster, record sync ops.
  • getClustersForItems(itemIds) — cluster lookup for list rendering.
  • listHotClusters(limit) — size-desc listing for the panel.
  • Read propagation hooks into the existing item-state path: marking a member read marks all members read and records item_state ops for the changed ones.

Failures are contained: a clustering run that throws must never break ingestion or reading (same isolation rule used by the rules engine).

API#

  • GET /clusters?scope=... — clusters overlapping the current list scope.
  • GET /clusters/hot?limit= — hot topics panel data.
  • POST /clusters/:globalId/split — split, write-guarded.
  • POST /clusters/recompute — manual recompute (used after settings change), write-guarded.
  • Cluster info is attached to item rows for list rendering (cluster global id, member count, whether the item is the primary).

Web UI#

  • List row: when an item is a cluster primary, render the row with a "+N sources" control; expanding shows members (feed + title), each openable. Non-primary members are omitted from the list (they live inside the cluster).
  • Hot topics panel: a virtual entry near saved searches; rows sorted by member count with the contributing source names.
  • Cluster menu: "Why grouped?" (shows the basis + members) and "Split this cluster" (with a toast confirming it won't regroup).
  • Settings: enable/disable, window days, threshold; disabled state explains that embeddings must be on, with a link to enable, and triggers the bounded backfill when turned on.
  • i18n for en, zh-CN, zh-TW, ja, identical key sets, plain Chinese copy.

Testing#

apps/api:

  • Two near-identical items in-window cluster; two unrelated items do not.
  • Items outside the window never cluster; muted items never cluster; items without embeddings are skipped (not errors).
  • Primary is the earliest member; latest_at tracks the newest member.
  • Reading one member marks all members read and records ops for exactly the changed items; a new member joining a read cluster does not reset it to unread.
  • Split: pairs recorded, cluster removed, and a subsequent clusterRecentStories() does NOT regroup those items; splits survive recompute; split sync round-trip.
  • Recompute is idempotent (running twice yields identical clusters, no churn).
  • Threshold/window settings change the outcome as expected.

apps/web (DOM-free): the pure helper that folds a flat item list plus cluster data into render rows (primary rows with member counts, members hidden), and the hot-panel sort ordering.

Definition of done#

  • With embeddings on, duplicate coverage collapses to one row showing "+N sources"; expanding reveals every source; reading one marks the story read.
  • Cluster position follows the newest member; the primary shown is the earliest.
  • The hot topics panel lists the most-reported stories in the window.
  • A wrong cluster can be split, the reason is visible, and split items never regroup — on any device.
  • With embeddings off, the feature is clearly disabled rather than silently broken; enabling it backfills only the window and works immediately.
  • Historical items outside the window are untouched.
  • API and web test suites plus builds pass.