The title of my talk for NECHFES2008 student conference was “Multi-threaded spoken dialogs in hands-busy and eyes-busy environments: Human-human experiments”. That talk was mostly about our new experiments that we plan to run. We will use variation of a Twenty Questions game in our experiments. This game allows us to control the search space for the solutions and clear identification of adjacency pairs. I’ll be posting more information about the experiments later.

Thanks,

Alexander Shyrokov

The Fall 2008 e-Newsletter of the Speech and Language Technical Committee of the IEEE Signal Processing Society includes an article about the driving simulator-based research going on within Project54. Thanks to Mike Seltzer for his help with editing the article.

Btw, the image that appears in the article was created by Alex Shyrokov and may be included in an upcoming book by Matthias Wölfel and John McDonough entitled Distand Speech Recognition.

Andrew Kun

Probably everybody has at least some experience with the GPS-based personal navigation aids. They usually provide directions using voice prompts and information displayed on LCD screens. While such devices appear to be less distracting to use than paper directions, in-car displays may distract drivers from their primary task, driving.

In order to assess how different in-car navigation aids affect driving performance and visual attention, we conducted an experiment using our high-fidelity driving simulator. The simulator is also equipped with an eye-tracker which provides information about subjects’ visual attention. The following picture illustrates how the experimental setup looked like.

There were three navigation aids that subjects tested in this experiment: paper directions (turn-by-turn directions with a map printed on a sheet of paper), standard PND directions (a map displayed on an LCD screen with turn-by-turn directions delivered through voice prompts), and voice-only directions (just voice-delivered turn-by-turn directions). Participants were driving on a two-lane city road with markings and a light traffic was introduced. The following picture shows one snapshot from the simulation.

We recorded three measures of driving performance from our driving simulator: lane position, steering wheel angle, and velocity. Higher variances of these variables represent worse driving performance. We also calculated the percent dwell time on the outside world for each subject using the data collected by the eye-tracker.

Our results showed that using paper directions degrades driving performance (lane position variance, steering wheel angle variance, and the mean velocity were significantly different between the paper and other navigation aids) and visual attention significantly more than using either a navigation device that provides voice prompts with a map or a device that provides voice prompts only.

Regarding the visual attention, the time participants spent looking at the road ahead was significantly higher when using the voice-only aid than when using either the paper or the voice and visual aid. On average participants spent around 94% of time looking at the road when using the voice only aid and around 89% when using the voice and visual aid.

While these findings supported our hypotheses, one interesting thing was that the majority of the participants expressed that they would prefer using the navigation aid which provides both the visual and voice directions. Based on this, we propose for our future work to model the glancing behavior of the drivers, which would enable us to predict when one would require next instruction, so we would be able to issue a voice prompt. This in turn would enable drivers to keep their eyes on the road at all times, while still being able to have the next instruction in a timely manner.

Lots of other experiments regarding this issue will follow, so we’ll keep you posted!

Zeljko Medenica

As posted a few months ago, Matthew Lape and I were given the opportunity to work on KLAS, the Kingsbury Location Awareness System. With the help of Professor Andrew Kun as our adviser and Oskar Palinko as a source of valuable insight, we were able to create a simple navigation application and tour guide application using the Project54 framework as our software development platform. While the work is still in progress, here are some of the results of our summer work.

Taking a look at KLAS, there are primarily two software components: the location determination application and the user-interactive applications. I will briefly describe the user-interactive applications while Matt will discuss the location determination side of things in another post. KLAS is a location awareness system that utilizes IEEE 802.11 RSSI readings to obtain a user’s location. This location information can be useful in various applications. The applications we decided to focus on for KLAS is a tour guide application and the navigation application.

This is an example of the map display in the KLAS Navigation application. The PDA on the left depicts what a user will see if they are in the shaded orange region. The arrow is the direction the user needs to go. If the user follows the arrow, the image on the right PDA will be shown. The right PDA image depicts where the user is currently and shows how to get to the destination.

This is an example of the Tour Guide application for KLAS. The image on the left depicts where the user is currently (somewhere in the shaded orange region) and a selected room (in red) that the user wishes to learn more information about. The user can cycle through nearby rooms to learn more information about it.

The image on the PDA on the right is shown if the user selects the “Room Info” button in the left image. Currently, the room number, the occupants, and some information about the room and occupant is given. The user can also cycle through the rooms in this screen and also press “Map” to navigate back to the map display.

As we continue to work on this, we plan on playing a lot more with the GUI and using Project54’s SUI as well. We hope to perform some user studies and experiments with different methods of pedestrian navigation and also conveying information. Stay tuned for some more information about KLAS brought to you by Matthew Lape!

Mark Taipan

Here’s the 10^100 website and a CNN story on the topic.

Andrew Kun

I was just talking to Mark Taipan and Matt Lape who are working on an indoor navigation project that utilizes handheld computers. Their application allows users to see where they are on a map. Of course the small size of handheld computer displays limits the type of maps you can use. This is even more of a problem for many mobile phones (even smaller displays), which are likely to be the actual devices used in any commercially viable indoor navigation application.

One solution may come from projector phones such as this one. If you had a projector phone you could display a map on the wall and presumably get better resolution, or at least larger characters, than what you can get on a mobile phone display. It would be interesting to see if people would be willing to use such a system.

Andrew Kun

Last Friday Oskar Palinko defended his MS thesis. Oszkar’s thesis was centered around the cool push-to-talk (PTT) glove he has designed.

Oszkar ran a rather large user study (24 participants) to evaluate if the PTT glove outperforms a fixed PTT button. While in comparing driving performance when using the two PTT solutions Oszkar didn’t find a main effect, he did find that the experiment participants looked down at the steering wheel more often when using the fixed PTT. Is this a problem? Maybe. While the total amount of time subjects spent looking at the steering wheel when using the fixed PTT button amounted to about 1% of the total experiment time, the average fixation was around 300 ms long. If such a fixation came at the wrong time (e.g. at the moment a leading vehicle started to brake), this could be a problem.

Congratulations Oszkar on a job well done!

Andrew Kun

This fall I’ll be teaching a graduate level course on ubiquitous computing (ECE 992). The course will look at recent research in this field, or more accurately in a few segments of this field. Specifically, we’ll look at the visions of ubicomp, some applications, human-computer interfaces, context awareness and privacy.

The course will revolve around reviews of the recent ubicomp research literature. I envision something similar to an NSF review panel, with instructions on what to pay attention to (they won’t be exactly NSF instructions but will have similarities to those), a lead reviewer who starts off the review process, a scribe who takes notes and where the entire class constitutes the panel.

Students will also work on a ubicomp project - basically either the evaluation of an existing ubicomp system, the design of the specifications for a new system or the implementation of a new system. I hope that people who decide to take the course will bring their own ideas from their research or work and that we can create projects that will be useful to them in those domains. I know a couple of students from my lab will be able to do this.

You can check out the course website here and relevant links here. Questions? Send me an email.

Andrew Kun

Last week Tim Paek of Microsoft Research visited the Project54 lab at UNH. Tim and I have been collaborating for about two years now, and this summer Tim is the supervisor of my PhD student, Zeljko Medenica, during Zeljko’s summer internship with MSR.

As part of his visit Tim gave a talk on his research in the field of mobile speech interaction. The talk covered three topics. First, Tim discussed his work on the Voice Command. Tim worked on utilizing user models to reduce the semantic error rate. Next, he talked about using reinforcement learning in a voice enabled browser. Finally, Tim talked about a mobile voice search application. The application provides intuitive ways to help the search application find the right answer. One way this is accomplished is by allowing multimodal refinements to the search. E.g. the user can help the speech recognizer by typing in the first letters of a problem word in an utterance. Another way is by allowing uncertain queries in the search (you can say “something” as part of the search to indicate you’re not sure what the exact search term should be). As part of the talk Tim gave a live demo of the mobile voice search application:

The talk was very interesting and the demo was impressive - thanks Tim! For more pictures click here.

Andrew Kun

A couple of weeks ago I spent a day visiting Microsoft Research and gave a talk on our lab’s work on in-car speech user interfaces. My hosts were two Microsoft researchers, Tim Paek and Ivan Tashev. I was also accompanied by one of my graduate students, Oszkar Palinko.

As part of the visit Ivan gave us a tour of two Microsoft Research labs. First we visited the acoustic anechoic chamber. This was a really neat experience. Check out the video below in which Ivan introduces this lab.

While it’s impossible to show in a video how quiet it gets in the anechoic chamber, check out the following video which shows me clapping inside the chamber. You may be able to notice that there’s no echo (thus anechoic!). In the chamber you could also notice this just by listening to people and hearing the dramatic change in perceived speaking volume as they (or you) turned.

Ivan also showed us his newly installed driving simulator. The simulator is made by STI and Ivan plans to use it very soon in his ongoing work on Commute UX, a “telephone dialog system for location-based services.” Oszkar and I had a chance to test the simulator and some of the cool “traps” or scenarios in which a distracted driver may end up in an accident. Here’s a video of me testing the MSR driving simulator.

I uploaded some picture from this visit here.

Thanks Ivan for the tour and thanks to both Ivan and Tim for hosting.

Andrew Kun