Authors: Young-Bum Kim, Dongchan Kim, Joo-Kyung Kim, Ruhi Sarikaya
Conference: Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL 2018, Industry Papers), pp. 16–24
Preprint: arXiv:1804.08064 [cs.CL]
Location: New Orleans, Louisiana | June 2018
Direct Links: ACL Anthology N18-3003 | PDF Fulltext
Abstract
Intelligent personal digital assistants (IPDAs), a popular real-life application with spoken language understanding capabilities, can cover potentially thousands of overlapping domains for natural language understanding, and the task of finding the best domain to handle an utterance becomes a challenging problem on a large scale.
In this paper, we propose a set of efficient and scalable neural shortlisting-reranking models for large-scale domain classification in IPDAs. The shortlisting stage focuses on efficiently trimming all domains down to a list of $k$-best candidate domains, and the reranking stage performs a list-wise reranking of the initial $k$-best domains with additional contextual information. We show the effectiveness of our approach with extensive experiments on 1,500 IPDA domains.
Two-Stage Shortlisting-Reranking Pipeline
[Raw Spoken Utterance]
│
▼
┌───────────────────────────────────────────────────┐
│ Stage 1: Neural Shortlisting Filter │
│ - Optimized for high recall & minimal latency │
│ - Dual-encoder projections prune thousands of │
│ candidate domains down to top-k (e.g. k=5..10) │
└─────────────────────────┬─────────────────────────┘
│
Top-k Candidate Set
│
▼
┌───────────────────────────────────────────────────┐
│ Stage 2: Contextual Neural Reranker │
│ - Evaluates cross-attention between candidates │
│ - Fuses multi-turn context, device & location │
│ - List-wise loss maximizes final top-1 accuracy │
└─────────────────────────┬─────────────────────────┘
│
▼
[Final Domain Target]- Stage 1 (Shortlisting):
- Optimized for high candidate recall under strict milliseconds constraints.
- Leverages fast dual-encoder projections to reduce candidate universe from thousands to $k \in [5, 10]$.
- Stage 2 (Reranking):
- Evaluates deep interactions between candidate domains, historical dialogue turns, entity recognition signals, and device modalities.
- Performs list-wise ranking loss optimization to select the globally optimal domain.
Empirical Highlights
- Evaluated on an internal benchmark of 1,500 distinct conversational domains with real user interaction logs.
- Delivered substantial improvements in top-1 domain routing accuracy over single-stage monolithic classifiers.
- Reduced computational latency by more than $70%$ compared to running heavy monolithic neural rankers over the full catalog.
Citation
@inproceedings{kim2018scalable,
title={A Scalable Neural Shortlisting-Reranking Approach for Large-Scale Domain Classification in Natural Language Understanding},
author={Kim, Young-Bum and Kim, Dongchan and Kim, Joo-Kyung and Sarikaya, Ruhi},
booktitle={Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 3 (Industry Papers)},
pages={16--24},
year={2018},
doi={10.18653/v1/N18-3003},
url={https://aclanthology.org/N18-3003/}
}