Lattice Uncertainty Visualization:
Understanding Machine Translation and Speech Recognition
Christopher Collins, Sheelagh Carpendale, Gerald Penn

Abstract
Lattice graphs are used as underlying data structures in many statistical processing systems, including natural language processing. Lattices compactly represent multiple possible outputs and are usually hidden from users. We present a novel visualization intended to reveal the uncertainty and variability inherent in statistically-derived outputs of language technologies. Applications such as machine translation and automated speech recognition typically present users with a best-guess about the appropriate output, with apparent complete confidence.

Through case studies in cross-lingual instant messaging chat and speech recognition, we show how our visualization uses a hybrid layout along with varying transparency, colour, and size to reveal the various hypotheses considered by the algorithms and help people make better-informed decisions about statistically-derived outputs.

Many current applications, such as the majority of those in the realm of natural language processing are statistically based. Their outputs represent a “best guess” by the algorithm, given some training data, parameter settings, and input. These best-guess outputs come from a very large collection of possibilities, each ranked with a score. However, these systems present their result in a black box fashion, showing only a single response. Since no details about probabilities, uncertainties, or the workings of the algorithms are provided, it is easy to misconstrue the output as having a low uncertainty. This lack of detail deprives us of the context necessary to make well-informed decisions based on the reliability of that output.

Building on the generalizations of human-computer optimization by Scott et al., we hypothesize that by including a human “in-the-loop” we can leverage the intelligence of the human and the processing power of the computer to quickly solve the same problems with better solutions.
To meet this goal, we identified several constraints to guide our design process:

• ensure easy readability of paths in the lattice;
• provide an intuitive visual mapping of uncertainty within the lattice which supports the ordering of nodes;
• provide for visual pop-out of nodes of high certainty and nodes in the optimal path identified by the algorithm;
• provide alternative representations of the data to clarify meaning, where possible;
• in most cases, require no interaction;
• where interaction is necessary (providing detail in context and manipulation of best-path tracing), it should be lightweight and easy to learn.

Our solution uses the inherent structure in the data to reveal the alternative solutions considered by statistical natural language processing systems. We use lattice graphs, hybrid graph and force-directed layout, and two forms of uncertainty encoding on node borders to show alternative solutions and their associated uncertainty. Any path through the lattices we present is a potential solution, with the algorithm's`best guess' solution shown along the bottom and with green linking edges for easier readability.

We applied our solution to two case studies:

Translation of Instant Messaging Chat

Machine translation offers much promise for improving workplace communication among colleagues situated in offices in different parts of the world. Many corporations use instant messaging chat as a means of facilitating communication, however current translation quality is too low to feasibly
use it in a critical setting. In this case study we present a prototype visualization system for instant messaging conversations which uses our lattice uncertainty visualization to reveal the uncertainty in the translation and provide alternative translations when available. The user can explore alternative translations for this span of the sentence, or, if no reasonable alternatives exist, use the chat to request clarification from the author of the original message. When out-of-vocabulary words are encountered, or the translation uncertainty is particularly high, photos are retrieved from Flickr to help clarify the intended meaning. Below is a screen shot of the chat client and a sample translation output lattice visualization.

Visualizing Uncertainty in Speech Recognition

We apply the same technique to reveal the uncertainty in automatic speech recognition outputs. Here we show a gradient encoding on the node borders to show uncertainty. Less certain nodes have a fuzzier border:

This work was created with the excellent prefuse information visualization toolkit and the Phramer machine translation system.