Here’s one for the books!  First, let me ask a few questions.  What do you do if your shiny new application causes your previously working system to crash?  Well, that’s simple, you search through your code for days, line by line, until some function returns an error, then correct for it.   What if you’ve searched through your code, turned days of searching into weeks, and your debugging techniques have only made things worse?  What if your system now fails randomly, or in applications which use to work?  What if it fails trying to load, trying to shut down, or at simple COM procedures in which failure is VIRTUALLY impossible?  What you should do is throw your application away and forget the whole thing ever happened, but we thought it would be better to take the other route, so let me shine some light on a very infrequent, system crippling, mobile disaster.

The majority of our deployments have been made on Symbol devices, with some running Window Mobile 2003 upon a Windows CE 4x framework and others utilizing Windows Mobile 5.0 upon Windows CE 5x.  The introductory paragraph held only when our system operated on the older version of Windows Mobile/Windows CE, which ended some of my frustrations since I could blame the failures on a possible limitation of an older operating system.  However, in doing so, my solution rested only upon an assumption, I had no evidence.  In hopes of seeking some expert advice, I created an MSDN forum entitle “Experiencing COM errors using Windows Mobile 2003”.  After some time, Christopher Fairbairn replied and referenced me towards an article entitled “Slaying the Virtual Memory Monster”.  The rest of this post may be a bit technical, but I’ll leave out as many details as I can.

What is virtual memory?  “Virtual memory is a computer system technique which gives an application program the impression that it has contiguous working memory, while in fact it may be physically fragmented and may even overflow on to disk storage.” – Wikipedia.  Basically, it’s a nice way of keeping everything in line.  In Windows CE 4x/5x, only 32 processes are allowed to be running at any one time, and a virtual memory (VM) space of 4GB is partitioned into many sections, allowing for each of the 32 processes to consume a possible 32MB.  These processes reside in locations called slots, where slot 0 houses the current process, slot 1 contains system (execute in place – XIP) DLLs, and the remaining 30 slots contain “resting” processes.  In the figure below, the “Active Process” section represents slots 0 and 1, and the “Reserved” section corresponds to slots 2-31. 

Now this is where things get tricky.  The figure below shows a more detailed layout of slots 0 and 1, and since it covers two slots, it represents 64MB of VM.  As I said before, the process to be executed is paged in from its “resting” slot to slot 0.  The base code of the process (executable – .exe) is loaded first and can be seen at the bottom of the figure, slightly above a small reserved portion.  Above this are various data and resource sections, along with dynamic heap and stack spaces, which grow upward as allocations are made within the process.  At the top of the figure, “COREDLL.DLL” and “Other XIP DLLs” can be found.  XIP DLLs begin loading at the top of slot 1 and continue in a downward manner, possibly overflowing into slot 0 (where the current process is).  Now, process specific DLLs are loaded at the top of slot 0 and also grow in a downward manner (shown by dashed arrows in the figure).  I’ve kept things somewhat simple in my description, so if you find yourself confused or want more VM detail, you may want to reference the articles at the end of this post.  To finish the thought, we have three different VM positions, all of which are moving toward a central location, a good recipe for a bad day.

Following the VM analysis methods proposed in part two of “Slaying the Virtual Memory Monster”, I created a script which visually and numerically organized the VM data of their dumpmem.exe program.  Shown in the upper portion of the figure below is slot 0 of the VM address space of Project54 under Windows CE 5x.  As we can see, there is space between the two blue, vertical lines, which is good!  The leftmost blue line represents the top of the process stack (which grows upward, or in this case to the right), while the blue line to the right marks the lowest DLL position (the lowest VM address occupied by system or process specific DLLs, which grows downward, or in this case to the left).  Segments shown in red are wasted memory segments due to a minimum VM page size, that is, the minimum allowable VM chunk granted when a DLL loads.  The bottom portion of the figure may depict the wasted VM space in a more appropriate manner, and is shown as an “Allocation request/return ratio”.  Confused?  Yes, so was I.

 

Slot 0 of Windows CE 4x, on the other hand (figure not shown), was entirely exhausted, and the two blue lines collide in VM.  This is what caused the craziness described in the introductory paragraph.  COM and memory related errors were occurring not because we were out of physical memory, but because we were out of VM.  The randomness of the errors was related to the way Project54 loads its applications, randomly.  The imaging application worked fine, it just happen to be that one application which used the last remaining pages of VM (the space between the blue lines). 

So, why would Windows CE 4x fail and not 5x?  Well, 4x has a VM page size of 64KB, compared to 16KB in 5x.  So as more DLLs were loaded, more VM was wasted, because most DLLs were much less than 64KB (remember that ratio I talked about?).  Desktop developers almost never have to worry about their utilization of VM.  They simply are not under the same restrictions.  But not to worry, Windows CE 6x aims to change everything…

References:

Slaying the Virtual Memory Monster - part 1

Slaying the Virtual Memory Monster - part 2

Windows CE .NET Advanced Memory Managment

Effective Memory, Storage, and Power Managment in Windows Mobile 5.0

Subscribe to the comments through RSS Feed