VoiceTrace: A Benchmark and Retrieval Framework for Who-Said-What Speech Retrieval
Benchmark Construction
VoiceTrace-Bench evaluates hybrid speech retrieval across single- and multi-speaker scenarios: each query pairs a text query with a reference speech utterance, and a candidate document is relevant only if the queried content is spoken by the speaker specified in the reference speech.
We build it from four corpora — VoxCeleb / VoxCeleb2 (single-speaker) and VoxConverse / Seamless Interaction (multi-speaker) — through three steps:
- Sampling: pair reference speech and candidate documents per speaker (single-speaker), or via diarization for target speakers within multi-speaker segments
- Query generation & verification: transcribe with Qwen3-ASR, generate text queries with DeepSeek-Chat, then automatically verify relevance
- Manual quality control: human review and audio editing on the evaluation set
This yields 46K single-speaker and 140K multi-speaker training samples, plus a manually verified evaluation set of 1,580 single-speaker and 500 multi-speaker samples.
VoiceTrace-Bench Samples
Filter:
Dataset Processing Pipeline
Prompts — Single-speaker
Generate Prompt
Role: You are an expert in Audio Retrieval-Augmented Generation (Audio RAG) data synthesis.
Task Background:
We are building an Audio RAG system. In this system, a user gives a "Query Audio" (to identify the speaker‘s voice) and a "Question" (to specify the semantic content) to retrieve a specific audio segment from a massive audio corpus.
Your goal is to generate the "Question" that would allow the user to retrieve this specific audio segment from the audio corpus.
Task:
Based on the provided transcript of the specific audio segment, generate a natural and concise "Question" that a user would ask to retrieve this specific audio segment from the massive audio corpus.
STRICT CONSTRAINTS:
1. The output MUST be a single natural question ending with a question mark (?).
2. The question should refer to the speaker as "this person", "this speaker", "the speaker", or "this individual".
3. Use diverse and descriptive verbs (e.g., "reflect on", "discuss", "explain", "describe", "mention", "express").
4. Vary the opening of your questions. Use a diverse range of interrogatives such as "Where", "How", "What", "In which segment". Do not over-rely on "What".
5. Focus on the core content of the transcript.
6. Adopt an external observer's perspective, convert second-person references into objective third-person identifiers or abstract topics.
7. Answer in English, the total length should not exceed {max_words} words.
Transcript of the specified audio segment: {document}
Output Requirements: Provide only the "Question" string. Do not include any prefix or explanation.
Output:
Verify Prompt
Role: You are a Quality Control Assistant for RAG data.
Task: Evaluate if the Query (Q) is a valid retrieval instruction for the given Transcript (T).
Transcript (T): {document}
Query (Q): {query}
Criteria:
- The query is a readable single question.
- The query is semantically clear and meaningful.
- The query can be used to retrieve the transcript.
- The total length of the query does not exceed {max_words} words.
Output requirements:
Respond with "YES" if the query satisfies ALL criteria.
Respond with "NO" if it fails any criterion.
Do not provide any other text.
Output:
Prompts — Multi-speaker
Generate Prompt
Role: You are an expert in Audio Retrieval-Augmented Generation (Audio RAG) data synthesis for conversational scenarios.
Task Background:
We are building an Audio RAG system. In this system, a user provides a "Query Audio" (a voice sample of a specific speaker) and a "Question" (specifying the semantic content) to retrieve a multi-speaker conversation segment (Document Audio) where that specific speaker is participating.
Task:
The provided transcript of the conversation audio segment involves multiple speakers (e.g., [S1], [S2]).
Your goal is to focus specifically on the contributions and context of the target speaker **"{target_label}"**.
Generate a natural and concise "Question" that a user would ask to find this specific conversation segment based on what **"{target_label}"** said or how they interacted.
STRICT CONSTRAINTS:
1. The output MUST be a single natural question ending with a question mark (?).
2. REFERENCING: Refer to the target speaker as "this person", "this speaker", "the speaker", or "this individual". Do not use labels like "{target_label}" in the final output.
3. PERSPECTIVE: Adopt an external observer's perspective, convert second-person references into objective third-person identifiers or abstract topics.
4. VERBS: Use diverse verbs (e.g., "discuss", "respond to", "explain", "mention", "reflect on").
5. VARIETY: Vary the opening of your questions (e.g., "Where does...", "What does...", "How does...", "In which part..."). Do not over-rely on "What".
6. FOCUS: Ensure the question is anchored in the target speaker's ({target_label}) speech or their reaction within the dialogue.
7. LANGUAGE: Answer in English. Total length must not exceed {max_words} words.
Transcript:
{transcript}
Output Requirements: Provide only the "Question" string. Do not include any prefix or explanation.
Output:
Verify Prompt
Role: You are a Quality Control Assistant for Audio RAG data.
Task: Evaluate if the Question (Q) is a valid retrieval instruction for a specific speaker "{target_label}" in the given conversation Transcript (T).
Context:
- In the Question (Q), terms like "this speaker", "the person", or "the individual" specifically refer to the Target Speaker **"{target_label}"**.
- The Question is valid if it correctly describes **"{target_label}"**'s specific statements, reactions, or their role within this conversation.
Transcript (T): {transcript}
Question (Q): {query}
Target Speaker to Verify: {target_label}
Validation Criteria:
1. The question is a single natural question ending with a question mark.
2. The question is correctly anchored in the target speaker's ({target_label}) speech or reaction within the dialogue. It is acceptable if the question mentions other speakers' topics as context, as long as the focus is on "{target_label}"'s response.
3. The question is semantically clear and meaningful.
4. The question is within the {max_words}-word limit.
Output Requirements:
Only respond with "YES" if the question satisfies ALL criteria.
Respond with "NO" and the criteria it violates if it fails any criterion.
Do not provide any other text.
Output:
Processing Examples
License
The repository is licensed under CC BY-NC 4.0 (Creative Commons Attribution-NonCommercial 4.0 International).
Acknowledgements
- VoxCeleb / VoxCeleb2 for providing large-scale benchmark datasets for speaker-related research.
- VoxConverse for providing multi-speaker conversational recordings with diarization annotations.
- Seamless Interaction for open-source resources that support speech and multimodal interaction research.
Citation
If you find this work useful, please consider contributing to this repo and citing:
@misc{yee2026voicetrace,
title={VoiceTrace: A Benchmark and Retrieval Framework for Who-Said-What Speech Retrieval},
author={Aaron Yee and Fengjie Lu and Jiarui Hai and Chenang Jiang and Helin Wang and Siwei Tu and Weitao You and Lingyun Sun},
year={2026},
eprint={2609.18521},
archivePrefix={arXiv},
primaryClass={cs.SD},
url={https://arxiv.org/abs/2609.18521},
}