Authors: Young-Bum Kim, Karl Stratos, Dongchan Kim
Conference: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL 2017, Volume 1: Long Papers), pp. 1297–1307
Paper Link: ACL Anthology P17-1119 | PDF Fulltext
Location: Vancouver, Canada | July 2017
Abstract
Two types of data shift common in practice are:
- Transferring from synthetic data to live user data (a deployment shift), and
- Transferring from stale data to current data (a temporal shift).
Both cause a distribution mismatch between training and evaluation, leading to a model that overfits the flawed training data and performs poorly on the test data. We propose a solution to this mismatch problem by framing it as domain adaptation, treating the flawed training dataset as a source domain and the evaluation dataset as a target domain.
To this end, we use and build on several recent advances in neural domain adaptation such as adversarial training (Ganin et al., 2016) and domain separation networks (Bousmalis et al., 2016), proposing a new effective adversarial training scheme. In both supervised and unsupervised adaptation scenarios, our approach yields clear improvement over strong baselines.
Technical Formulation: Adversarial Neural Domain Adaptation
In commercial conversational NLU development:
- The Cold-Start Trap: New conversational skills rely on rule-based grammar templates (synthetic data).
- The Temporal Shift Trap: Mature skills rely on historical user logs collected months or years prior (stale data).
Directly optimizing standard cross-entropy loss causes the neural encoder to memorize synthetic patterns or obsolete lexical phrasing:
[Input Utterance x] ──> [Feature Encoder E] ──┬──> [Task Classifier C] ──> Intent / Slot
│
└──> [Domain Discriminator D] (Gradient Reversal)
└─> Predicts: Source (Synthetic) vs. Target (Live)Shared vs. Private Subspace Modeling
- Our framework decomposes representations into domain-invariant (shared) features and domain-specific (private) features.
- A minimax adversarial objective enforces that the shared feature extractor fools the domain discriminator while the task classifier preserves high semantic predictive power.
- Scale-invariant similarity constraints ensure that live user utterances align with the shared manifold without being corrupted by synthetic distribution artifacts.
Key Empirical Results
- Evaluated across multiple intent classification and slot filling domains in production dialogue datasets.
- Demonstrates statistically significant error rate reductions in both:
- Zero-shot / Unsupervised adaptation: Where zero labeled target live queries are available during training.
- Few-shot / Semi-supervised adaptation: Where a small set of live logs is blended with large synthetic pools.
Citation
@inproceedings{kim2017adversarial,
title={Adversarial Adaptation of Synthetic or Stale Data},
author={Kim, Young-Bum and Stratos, Karl and Kim, Dongchan},
booktitle={Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
pages={1297--1307},
year={2017},
doi={10.18653/v1/P17-1119},
url={https://aclanthology.org/P17-1119/}
}