HKUST KnowComp · long-term conversational memory

Safe memory is not one score.
It fails in four places.

MOSAIC asks whether an assistant can remember what a person said, showed, and conveyed across weeks of conversation. The benchmark holds the question, reader, prompt, and budget fixed, then intervenes on one stage at a time.

Leaderboard Paper GitHub Dataset arXiv forthcoming

923questions

784answerable

139refusals

11–24sessions each

Four stages, one question.

Existing memory benchmarks mostly cover text and images, and they report a single end-to-end number. That number cannot say whether the evidence was lost when it was written down, when it was retrieved, when it was packed into the prompt, or when the model read it.

85.2% of answerable questions need two or more facts. 75.0% need two or more modalities. The media carry the decision: spatial layout, fine visual detail, video order and repetition, or vocal prosody.

01WriteHow the history is stored: native media, or a text substitute.
02RetrieveWhether the right evidence is found in the long history.
03DeliverWhether that evidence actually enters the model request.
04ReadWhether the reader answers from the evidence it was given.

What is in the release

Point MOSAIC_DATA_ROOT at the downloaded dataset. The evaluation code reads these paths.

Questions

dataset/stage2_human_filtered/

Each item has a question, an evidence chain, and a timestamped session history.

Media

dataset/media/

Images, audio, and video. Stored paths that contain clean_output/media/ are mapped onto this folder.

Text views

textualized_caption/ and textualized_omni/

The same histories as captions, so a text-only system can be compared with native media.

Indexes and records

indexes/ and records/protocol_v1/

RAG, A-MEM, and MemoryOS indexes, plus the paper’s evaluation records.

Evidence roles

Narrower Context that rules candidates out.
Selector The cue that picks which evidence matters. For audio, this can be prosody rather than the words.
Answer The hop that states the value the question asks for.

Refusal items remove the required evidence. A system should answer when the chain is present and refuse when it is not. Track A holds the reader fixed and is the diagnostic comparison. Track B changes the whole system and is a deployment-style illustration.

What the diagnosis shows

These contrasts keep the question set fixed. Spatial evidence and delivery are the strongest results. Video timing and raw audio versus a transcript are smaller and should not be read as equally settled.

Deliver

Retrieved is not delivered.

On the same top-50 ranking, tightening what enters the request moves the score from 0.708 to 0.670. Complete-chain delivery falls from 0.675 to 0.491. Packing after retrieval is its own failure.

Write · image

Object lists drop the relation.

On the same items, raw images score 32.0 points above OCR. A spatial description scores 15.4 points above a list of objects. Presence of an object is not the layout between objects.

Budget

The tight budget is joint.

The most restrictive setting in the delivery figure changes top-k and the per-modality attachment limit together. The 17.7-point drop belongs to that combined budget, not to the one-media cap alone.

Leaderboard

Paper baselines only. A new row is added to docs/leaderboard.json after the run is checked against the 923-question set and the same judge.

End to end Fixed reader

923 questions. Each system uses its own reader and memory stack. Answerable score is the mean judge score on 784 answerable items. Correct refusal is the rate of refusing the 139 items whose evidence was removed. False refusal is the rate of refusing an answerable item. A high correct-refusal rate with a high false-refusal rate is indiscriminate abstention, not a better memory.

#	System	Memory	Reader	Answerable	Correct refusal	False refusal	Latency (s)
1	ReAct	Native multimodal	MiMo-V2.5	0.732	0.410	0.018	73.6
2	Naive RAG	Native multimodal	Gemini 3.5	0.687	0.496	0.022	31.2
3	ReAct	Native multimodal	Gemini 3.5	0.674	0.417	0.037	46.4
4	A-MEM	Caption text	DeepSeek-v4-Flash	0.623	0.719	0.173	—
5	Naive RAG	Native multimodal	MiMo-V2.5	0.614	0.734	0.143	29.6
6	A-MEM	Omni text	DeepSeek-v4-Flash	0.611	0.763	0.177	—
7	ReAct	Omni text	DeepSeek-v4-Flash	0.419	0.799	0.390	17.5
8	ReAct	Caption text	DeepSeek-v4-Flash	0.415	0.820	0.404	18.6
9	Naive RAG	Caption text	DeepSeek-v4-Flash	0.390	0.835	0.431	5.8
10	Naive RAG	Omni text	DeepSeek-v4-Flash	0.375	0.871	0.421	6.2
11	MemoryOS	Omni text	DeepSeek-v4-Flash	0.255	0.892	0.614	228.4
12	MemoryOS	Caption text	DeepSeek-v4-Flash	0.243	0.885	0.649	205.4

784 answerable questions. The reader is Gemini 3.5 for every row, so the score isolates the memory backend. Chain coverage is the fraction of items whose full required evidence chain is delivered. MemoryOS coverage is not observable in this run.

#	Backend	Memory	Chain coverage	Score
1	A-MEM	Caption text	0.787	0.684
2	A-MEM	Omni text	0.702	0.664
3	Flat RAG	Native multimodal	0.466	0.641
4	Flat RAG	Caption text	0.698	0.522
5	Flat RAG	Omni text	0.705	0.500
6	MemoryOS	Omni text	—	0.279
7	MemoryOS	Caption text	—	0.268

Submit a result

Use the released 923 questions and the paper’s judge. Report the answerable score, and for an end-to-end run also the correct-refusal and false-refusal rates. Accepted rows are written into the leaderboard file.

The issue is filed on HKUST-KnowComp/MOSAIC. Include enough detail for someone else to rerun the number.