In our last post Mark explained the software components and structure of KLAS (Kingsbury Location Awareness Systems). These components allow the system to take in data from an external source, estimate the user’s location (more on this in a minute) and then utilize that data to provide both navigation and tour guide capabilities.

The location aware portion of the system had to have a method of determining the user’s location through the use of some external infrastructure (e.g. GPS, Wi-Fi, RFID, etc.). After some reading and discussion, Mark and I decided that we needed a reliable indoor infrastructure, one that could either be easily developed, or ideally, one that was already established. We found, in the reading, that GPS indoors was considered mostly as an unreliable method, and RFID, although very accurate, was relatively expensive and too short range for our application. Looking at Wi-Fi, we found that it was acceptably reliable in this type of environment. The best part is that because Wireless Internet is so prevalent, the infrastructure in many locations is already established. It was thus why we chose to use Wi-Fi (IEEE 802.11) for our system.

To develop our specific location system using Wi-Fi, we had to investigate our target location (Kingsbury Hall) to determine the coverage and location of the Wi-Fi Access Points. We found eight UNH Wireless Access Points located on the second floor, equally spaced to cover the majority of the area. We then added two more Access Points in a few weak areas to give our system a total of ten Access Points to utilize.



Here we see two examples of the KLAS Access Points. The one on the left is a typical UNH Wireless Access Point, and on the right is one of the two Access Points that we added to provide stronger coverage in a few areas.

Our next step was to create an algorithm which would read in the Relative Signal Strength Indicator (RSSI) values from the Wi-Fi card on our PDAs and then determine the user’s location based on that data. To do this we spent a bit of time investigating the pros and cons of the methods used by other researchers, utilizing either probabilistic or deterministic models. We found, through testing and our reading, that Wi-Fi signals are highly susceptible to environmental changes such as the thickness and density of walls, the amount of people between the Access Point and the user, and even the relative humidity of the area. We decided, with the help of Professor Andrew Kun and Oskar Palinko, that due to the size of the area and the system’s environment, using a calibrated deterministic model would be best. This essential meant that we would evenly space master location points (also called reference points) throughout the floor. These points would then be compared to the user’s actual location to determine which master location they are closest to.

Through this testing we found that although the Wi-Fi structure was able to provide a reasonable estimate of the location, our initial points were too close, causing multiple real-time points to map to one reference point. This allowed use to gauge the reference location spreading, which eventually provided us with a total of 46 points for the entire second floor. After further testing with these points, it was found that there were still areas that had a tendency to “group” together. This was seen because of the fact that the building’s features caused certain areas to estimate the same master location. Recognizing this, we were able to break up the floor even more generally into 16 areas, which could be used by our system to categorize the user’s current location.



This map shows the even spread of the 16 user areas throughout the floor, as well as the locations of the ten KLAS Access Points. Dividing the floor into these areas allows for our system to have a built in buffer for the variations in the environment, as well as any variations that might be present in the user’s device itself. This allows for a more stable, reliable system, which is used, as Mark mentioned in his post, to provide navigation and tour guide capabilities for our users.

We hope to continue to refine our system in the next few months, making it even more reliable and stable. This will allow us to conduct more research in the area of the user interface, and hopefully give us the opportunity to develop a simple, intuitive system which will make KLAS a useful tool for all.

Matthew Lape

Probably everybody has at least some experience with the GPS-based personal navigation aids. They usually provide directions using voice prompts and information displayed on LCD screens. While such devices appear to be less distracting to use than paper directions, in-car displays may distract drivers from their primary task, driving.

In order to assess how different in-car navigation aids affect driving performance and visual attention, we conducted an experiment using our high-fidelity driving simulator. The simulator is also equipped with an eye-tracker which provides information about subjects’ visual attention. The following picture illustrates how the experimental setup looked like.

There were three navigation aids that subjects tested in this experiment: paper directions (turn-by-turn directions with a map printed on a sheet of paper), standard PND directions (a map displayed on an LCD screen with turn-by-turn directions delivered through voice prompts), and voice-only directions (just voice-delivered turn-by-turn directions). Participants were driving on a two-lane city road with markings and a light traffic was introduced. The following picture shows one snapshot from the simulation.

We recorded three measures of driving performance from our driving simulator: lane position, steering wheel angle, and velocity. Higher variances of these variables represent worse driving performance. We also calculated the percent dwell time on the outside world for each subject using the data collected by the eye-tracker.

Our results showed that using paper directions degrades driving performance (lane position variance, steering wheel angle variance, and the mean velocity were significantly different between the paper and other navigation aids) and visual attention significantly more than using either a navigation device that provides voice prompts with a map or a device that provides voice prompts only.

Regarding the visual attention, the time participants spent looking at the road ahead was significantly higher when using the voice-only aid than when using either the paper or the voice and visual aid. On average participants spent around 94% of time looking at the road when using the voice only aid and around 89% when using the voice and visual aid.

While these findings supported our hypotheses, one interesting thing was that the majority of the participants expressed that they would prefer using the navigation aid which provides both the visual and voice directions. Based on this, we propose for our future work to model the glancing behavior of the drivers, which would enable us to predict when one would require next instruction, so we would be able to issue a voice prompt. This in turn would enable drivers to keep their eyes on the road at all times, while still being able to have the next instruction in a timely manner.

Lots of other experiments regarding this issue will follow, so we’ll keep you posted!

Zeljko Medenica

As posted a few months ago, Matthew Lape and I were given the opportunity to work on KLAS, the Kingsbury Location Awareness System. With the help of Professor Andrew Kun as our adviser and Oskar Palinko as a source of valuable insight, we were able to create a simple navigation application and tour guide application using the Project54 framework as our software development platform. While the work is still in progress, here are some of the results of our summer work.

Taking a look at KLAS, there are primarily two software components: the location determination application and the user-interactive applications. I will briefly describe the user-interactive applications while Matt will discuss the location determination side of things in another post. KLAS is a location awareness system that utilizes IEEE 802.11 RSSI readings to obtain a user’s location. This location information can be useful in various applications. The applications we decided to focus on for KLAS is a tour guide application and the navigation application.

This is an example of the map display in the KLAS Navigation application. The PDA on the left depicts what a user will see if they are in the shaded orange region. The arrow is the direction the user needs to go. If the user follows the arrow, the image on the right PDA will be shown. The right PDA image depicts where the user is currently and shows how to get to the destination.

This is an example of the Tour Guide application for KLAS. The image on the left depicts where the user is currently (somewhere in the shaded orange region) and a selected room (in red) that the user wishes to learn more information about. The user can cycle through nearby rooms to learn more information about it.

The image on the PDA on the right is shown if the user selects the “Room Info” button in the left image. Currently, the room number, the occupants, and some information about the room and occupant is given. The user can also cycle through the rooms in this screen and also press “Map” to navigate back to the map display.

As we continue to work on this, we plan on playing a lot more with the GUI and using Project54’s SUI as well. We hope to perform some user studies and experiments with different methods of pedestrian navigation and also conveying information. Stay tuned for some more information about KLAS brought to you by Matthew Lape!

Mark Taipan

Here’s one for the books!  First, let me ask a few questions.  What do you do if your shiny new application causes your previously working system to crash?  Well, that’s simple, you search through your code for days, line by line, until some function returns an error, then correct for it.   What if you’ve searched through your code, turned days of searching into weeks, and your debugging techniques have only made things worse?  What if your system now fails randomly, or in applications which use to work?  What if it fails trying to load, trying to shut down, or at simple COM procedures in which failure is VIRTUALLY impossible?  What you should do is throw your application away and forget the whole thing ever happened, but we thought it would be better to take the other route, so let me shine some light on a very infrequent, system crippling, mobile disaster.

The majority of our deployments have been made on Symbol devices, with some running Window Mobile 2003 upon a Windows CE 4x framework and others utilizing Windows Mobile 5.0 upon Windows CE 5x.  The introductory paragraph held only when our system operated on the older version of Windows Mobile/Windows CE, which ended some of my frustrations since I could blame the failures on a possible limitation of an older operating system.  However, in doing so, my solution rested only upon an assumption, I had no evidence.  In hopes of seeking some expert advice, I created an MSDN forum entitle “Experiencing COM errors using Windows Mobile 2003”.  After some time, Christopher Fairbairn replied and referenced me towards an article entitled “Slaying the Virtual Memory Monster”.  The rest of this post may be a bit technical, but I’ll leave out as many details as I can.

What is virtual memory?  “Virtual memory is a computer system technique which gives an application program the impression that it has contiguous working memory, while in fact it may be physically fragmented and may even overflow on to disk storage.” – Wikipedia.  Basically, it’s a nice way of keeping everything in line.  In Windows CE 4x/5x, only 32 processes are allowed to be running at any one time, and a virtual memory (VM) space of 4GB is partitioned into many sections, allowing for each of the 32 processes to consume a possible 32MB.  These processes reside in locations called slots, where slot 0 houses the current process, slot 1 contains system (execute in place – XIP) DLLs, and the remaining 30 slots contain “resting” processes.  In the figure below, the “Active Process” section represents slots 0 and 1, and the “Reserved” section corresponds to slots 2-31. 

Now this is where things get tricky.  The figure below shows a more detailed layout of slots 0 and 1, and since it covers two slots, it represents 64MB of VM.  As I said before, the process to be executed is paged in from its “resting” slot to slot 0.  The base code of the process (executable – .exe) is loaded first and can be seen at the bottom of the figure, slightly above a small reserved portion.  Above this are various data and resource sections, along with dynamic heap and stack spaces, which grow upward as allocations are made within the process.  At the top of the figure, “COREDLL.DLL” and “Other XIP DLLs” can be found.  XIP DLLs begin loading at the top of slot 1 and continue in a downward manner, possibly overflowing into slot 0 (where the current process is).  Now, process specific DLLs are loaded at the top of slot 0 and also grow in a downward manner (shown by dashed arrows in the figure).  I’ve kept things somewhat simple in my description, so if you find yourself confused or want more VM detail, you may want to reference the articles at the end of this post.  To finish the thought, we have three different VM positions, all of which are moving toward a central location, a good recipe for a bad day.

Following the VM analysis methods proposed in part two of “Slaying the Virtual Memory Monster”, I created a script which visually and numerically organized the VM data of their dumpmem.exe program.  Shown in the upper portion of the figure below is slot 0 of the VM address space of Project54 under Windows CE 5x.  As we can see, there is space between the two blue, vertical lines, which is good!  The leftmost blue line represents the top of the process stack (which grows upward, or in this case to the right), while the blue line to the right marks the lowest DLL position (the lowest VM address occupied by system or process specific DLLs, which grows downward, or in this case to the left).  Segments shown in red are wasted memory segments due to a minimum VM page size, that is, the minimum allowable VM chunk granted when a DLL loads.  The bottom portion of the figure may depict the wasted VM space in a more appropriate manner, and is shown as an “Allocation request/return ratio”.  Confused?  Yes, so was I.

Slot 0 of Windows CE 4x, on the other hand (figure not shown), was entirely exhausted, and the two blue lines collide in VM.  This is what caused the craziness described in the introductory paragraph.  COM and memory related errors were occurring not because we were out of physical memory, but because we were out of VM.  The randomness of the errors was related to the way Project54 loads its applications, randomly.  The imaging application worked fine, it just happen to be that one application which used the last remaining pages of VM (the space between the blue lines). 

So, why would Windows CE 4x fail and not 5x?  Well, 4x has a VM page size of 64KB, compared to 16KB in 5x.  So as more DLLs were loaded, more VM was wasted, because most DLLs were much less than 64KB (remember that ratio I talked about?).  Desktop developers almost never have to worry about their utilization of VM.  They simply are not under the same restrictions.  But not to worry, Windows CE 6x aims to change everything…

References:

Slaying the Virtual Memory Monster - part 1

Slaying the Virtual Memory Monster - part 2

Windows CE .NET Advanced Memory Managment

Effective Memory, Storage, and Power Managment in Windows Mobile 5.0

Last Friday Oskar Palinko defended his MS thesis. Oszkar’s thesis was centered around the cool push-to-talk (PTT) glove he has designed.

Oszkar ran a rather large user study (24 participants) to evaluate if the PTT glove outperforms a fixed PTT button. While in comparing driving performance when using the two PTT solutions Oszkar didn’t find a main effect, he did find that the experiment participants looked down at the steering wheel more often when using the fixed PTT. Is this a problem? Maybe. While the total amount of time subjects spent looking at the steering wheel when using the fixed PTT button amounted to about 1% of the total experiment time, the average fixation was around 300 ms long. If such a fixation came at the wrong time (e.g. at the moment a leading vehicle started to brake), this could be a problem.

Congratulations Oszkar on a job well done!

Andrew Kun

As a part of Project54, we have a newly formed Radio Testing Lab which is used for performing tests on projects related to the usage of Land Mobile Radios. Currently we are working on two projects that attempt to solve problems associated with the utilization of mobile radios. While the first project provides a solution for the interoperability problem using radio patching, the second project is based on the implementation of APCO Project 25 radio standard.

In the lab, for test equipment, we have more than fifteen radios made by different manufacturers (Motorola, Kenwood, and E. F. Jonson), several digital phosphor oscilloscopes (Tektronix TDS3000B series), a vector signal analyzer (Agilent 89441A with RF section), and a bunch of PCs and laptops. With this equipment and our experience we can assure that our projects are well tested and verified. Below are two pictures from the lab.

Using the radio patching method, we have developed a solution for the radio interoperability problem. It is based on more affordable off-the-shelf devices and represents an adequate solution for small public agencies. The system supports full radio patching with proper PTT functioning and has audio signal latency bellow what is specified by ITU-T standard. It turned out that radio patching is the cheapest and fastest solution to the radio interoperability problem. More information about the system can be found in this technical report, and here about its precursor system.

Project 25 seems a perfect solution to constant growing needs for data communications in cruisers because it supports both data and voice transmissions over radio channels. Also, most public safety agencies already have and use Project 25 compliant radios. However, in order to fully utilize their radios and Project 25 data capabilities those agencies have to purchase very expensive Project 25 data capable base stations. Small public safety agencies usually cannot afford such expensive pieces of equipment. To address this problem, we have developed a software-based Project 25 data base station as an affordable way for small agencies to bring the data into their cruisers. The project is in its final testing stage and very soon will be ready for deployment. You can read more about our base station in this previous post. More technical information about the implementation of the transmission side can be found in Eric’s thesis.

Ivan Elhart

Over the past few months I’ve spent much of my time developing the Project54PDA imaging applications.  Finally, after some coding revisions and a small in-lab experiment, we’re just about ready for deployment.  From the results, it seems as though tagging of images using handheld devices has a bright future, along with the note-taking and memo-recording capabilities the application provides.

Back in April we met with Lieutenant Mark Liebl of the NH State Police who gave his insight to the applications’ possible uses.  Mark had stated that he occasionally comes across a driver who has forgotten to carry or had misplaced their driver’s license.  In these circumstances information may be taken verbally from the subject and checked against the records database as normally would be done.  However, this assumes that the driver was being truthful.  This is where the imaging application comes into play.  At the time of questioning an officer may take a photograph of the subject and pair it with tagging, note, and/or memo data that may be reviewed later if necessary, providing some physical data on the event.  Along with this, Mark also believed that such data may be gathered at the scene of minor motor vehicle accidents.

Below are some screenshots of the imaging application in use.  Since most people are fairly familiar with the operation of a digital camera, I tried to have the imaging GUI replicate that functionality as much as possible.  Like all commands the application provides, images may be captured using the graphical buttons or through speech interaction and reviewed by using the scrolling features shown in the upper corners of the shot.  In the adjacent figure, the tagging mode of the application is depicted with the tags “poster” and “project54 logo” already paired with image (lower text area).  The upper text area displays previous tags the user has entered into the system, and allows the tagging process to be completed (through graphical or speech interaction) more rapidly.  That’s right, speech tagging!  Note-taking is performed using the text editor of the PDA, and the memo feature allows for a maximum 30 second audio recording.

With all of this data now stored on the PDA, I found it necessary to develop a set of applications which allow for extraction to and easy viewing on a desktop machine, because viewing data on a PDA is about as slow and painful as it gets.  The transfer application (not shown) handles the extraction, and the management application offers a centralized interface for data review (see the figure below).  All-in-all it was a lengthy process to get these applications working in harmony, but I think the data collection and organization methods they provide will prove to be quite valuable.

Michael Farrar

In our research for our KLAS Project, Mark Taipan and I ran across a few interesting articles on Wi-Fi indoor location systems, specifically ones which utilize RFID. One of the most informative research papers was the Sensor-Assisted Wi-Fi Indoor Location System for Adapting to Environmental Dynamics. This paper dealt with adding RFID sensors to a Wi-Fi indoor location system in order to add accuracy. The researchers realized that Wi-Fi location systems are highly susceptible to accuracy variances due to “environmental changes” (e.g. open vs. closed doors, humans absorbing/blocking the signal, and even to changes in humidity). Therefore the system was tested in various cases, with a radio map created for each. The researchers then added RFID sensors to the system to measure the different conditions (e.g. the humidity, how many doors were open vs. closed, etc.). This data was then used to determine which radio map would provide the greatest accuracy for the system at any given time. Because of this adaptation, the system was able to gain overall better accuracy and was less susceptible to these “environmental changes”.

This idea has helped us better understand the concerns and cautions of working with 802.11 Wi-Fi and the accuracy issues that surround it. We therefore recognize that these are issues for which we will have to deal with and thus we plan on compensating for them, either in the manner that was outlined in this paper, or in some other way.

Matthew Lape

A group of students from St Paul’s Advanced Studies Program Artificial Intelligence course recently visited the Project54 lab. The students were first given a demonstration of the Project54 system in our lab car, and then were given a chance to experiment for themselves. Next the students were taken into the simulation room and were given the opportunity to drive the Project54 simulator. A couple of pictures from this visit can be seen below.

Here students look on as Nathan Purmort assists a student using the Project54 system.

A St Paul’s student laughs while driving the Project54 simulator.

It was Great to have St Paul’s Artificial Intelligence class visit Project54 once again, and to see the number of students participating grow.

David Garneau

At the beginning of this semester our lab obtained a device called ProComp Infinity (see picture below) which is used for collecting physiological measurements (we want to measure heart rate and skin conductance).

In order to gain experience with this equipment and to see the sensitivity of the measurements to different environmental factors, we performed a modified Stroop Color Word test. This test is known to be demanding for most people and it tests our mental vitality. The subject is presented with a list of color names. The words are printed in color and the colors used for printing individual words do not match the color that the word describes (e.g. blue, black, red). The task is to say aloud the name of each color used in the list of words. What makes this relatively difficult is that people can read much faster than they can identify the name of a color. Thus, the test requires subjects to override the initial result of processing the list, which they arrive at by reading. During the test subjects sometimes fail to override the initial result and say the word they read and not the color used to print the word.

In order to make the test even more demanding, we introduced three independent variables that controlled the difficulty: number of words shown (5 or 8), available time (6 or 8 sec), and background music type (none, relaxing or annoying). How difficult can this be? Watch the short video bellow which shows one small portion of the experiment and try to say aloud the names of the colors as fast as possible, since the available time is limited.

While performing the experiment we recorded subjects’ heart rate and skin conductance. After analyzing the data we found that heart rate was mostly sensitive to available time and number of words, while the skin conductance was mostly sensitive to music type. But one thing is for sure, the next time when you listen to some annoying music you can be certain that your skin conductance will jump!

Zeljko Medenica