Authors: Young-Bum Kim, Karl Stratos, Dongchan Kim
Conference: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017, Volume 2: Short Papers), pp. 657–662
Paper Link: ACL Anthology P17-1060 | PDF Fulltext
Location: Vancouver, Canada | July 2017
Abstract
An important problem in domain adaptation is to quickly generalize to a new domain with limited supervision given $K$ existing domains. One approach is to retrain a global model across all $K + 1$ domains using standard techniques, for instance Daumé III (2009). However, it is desirable to adapt without having to re-estimate a global model from scratch each time a new domain with potentially new intents and slots is added.
We describe a solution based on attending an ensemble of domain experts. We assume $K$ domain-specific intent and slot models trained on respective domains. When given domain $K + 1$, our model uses a weighted combination of the $K$ domain experts’ feedback along with its own opinion to make predictions on the new domain. In experiments, the model significantly outperforms baselines that do not use domain adaptation and also performs better than the full retraining approach.
Architectural Design: Mixture-of-Experts with Domain Attention
When conversational platforms expand to introduce new domains:
- Retraining monolithic networks across all historical and new domains is computationally prohibitive and risks negative transfer or catastrophic forgetting.
- Completely disjoint models cannot leverage lexical or structural patterns learned by existing mature domains.
[Input Utterance]
/ | \
Expert 1 Expert 2 ... Expert K
\ | /
──> [Domain Attention] <──
│
[Weighted Synthesis]
│
[Target Domain K+1 Prediction]Attention-Weighted Expert Combination
- Each expert $k \in {1, \dots, K}$ produces an internal representation for intent and slot prediction.
- A dynamic attention network calculates relevance scores between the target input utterance and each expert domain representation.
- The synthesized context vector informs the target domain model, allowing new domains to achieve high zero-shot and few-shot accuracy with minimal supervised labels.
Key Empirical Takeaways
- Demonstrates that selectively attending across specialized domain experts outperforms both isolated domain models and expensive global model re-estimation.
- Provides a modular paradigm for continuous domain expansion in voice assistant architectures.
Citation
@inproceedings{kim2017domain,
title={Domain Attention with an Ensemble of Experts},
author={Kim, Young-Bum and Stratos, Karl and Kim, Dongchan},
booktitle={Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
pages={657--662},
year={2017},
doi={10.18653/v1/P17-1060},
url={https://aclanthology.org/P17-1060/}
}