Yer in a "Heap" of trouble now: Avoiding random crashes in Maya/Max due to running out of Windows resources!

Note; This tip has nothing to do with mental ray at all, but since the issue I describe tend to manifest on Maya when opening the mental ray control panel (due to the sheer number of tiny windows w. subwindows) I mention it here.


Have you ever run, for example, Maya (on Windows XP or earlier - Vista works better here) with a ton of other programs running, and tried to open a particularily complex AE template for some shader, to find the application go "plop" and disappear?

While there are any number of probable causes, I was debugging a particular case and actually found the cause, and it was sometihng I never really suspected, and is pretty non-obvious. And doesn't have anything whatsoever to do with mental ray, as I said above.

So what's up?

Back in the day of 16 bit Windows 3.0, running out of "window handles" (each window on screen has a "handle", like an identifier, used by the operating system to keep track of it) was extremely common, and every Windows user recognized the symptoms... windows wouldn't open, or buttons/texts would be missing from dialogs, or windows opened as only borders, or a strange "placeholder text version" of dialogs popped up.... whatever.

You'd think that a "modern" operating system wouldn't have a limit on the number of windows it could have opened, right? With multipled gigabytes of RAM, this shouldn't be an issue... right?

Well... think again. ;)

First of all the number of possible handles is apparently - still - limited to a 16 bit number (that's 65536 possible handles). But the real limit doesn't lie there, coz that's plenty. The real limit lies in the fact that each "window handle" also implies a memory allocation of a special data structure, that lives in a special memory area called the "desktop heap". And this memory area is small. As a matter of fact, by default, on XP (32 AND 64 bits) ... it's 3 megabytes!!!! Thankfully, on Vista it's larger, so Vista users do not need to do anything (except cry over all the other issues with Vista.... but that's for another blog day...)

Practically, this means that at around a few thousand windows (or other handles), things start failing.

Now, the user interface in Max, Maya and XSI are "windowed" UI's. This means that each control in each dialog is considered a "window" (from the point of view of the operating system). And sub-dialogs, little rollouts, etc. are all subwindows, with controls sub-sub-windows, and buttons can be sub-sub-sub-sub-windows, etc.

Anyone who has seen a compelex Max dialog box or Maya UI, and consider that counting every single little box, control, and thing, you'll find that that's a lot of windows. Maya easily consumes a couple of thousand handles all on it's own. (You can see this by telling your task managers "processes" tab to display the "GDI OBJECTS" column)

(Incidentally, this issue is known by Microsoft, to the extent that they try to write their software "windowless", i.e. when you run, say, Internet Exploder, each little thing on the webpage is not a separate window from the OS's point of view, precicely because of this problem!)

All is lost! The Sky Is Falling! Or?

Not at all.

See, the cool thing is that you can set up the size of the "desktop heap" by fiddling with the registry.

However, there are two things to consider when doing this, one fairly benign, and one less so.

The benign issue is that the "desktop heap" actually comes out of a larger memory area called the "Session View" (don't you just love these names?). And the size of the "Session View" is also limited (to 48Mb by default). Which is again adjustable, but that in turn lives in Kernel address space, which also is limited... to another, larger, value. And so on. You get the idea.

So the point is, you can increase the "desktop heap" size by editing the registry, but you can't just do it willy nilly and put in any old value, because you will get other problems, of running out of "Session View" space.

What I did was to up it from 3Mb to 4Mb, and the problems I had pretty much went away immediately.

The second, less benign issue, is that this is a really hairy registry location, and a horribly convoluted registry key, which actually contains a long string of some 200 characters where you have to change one thing in the middle, and the problem is the fact that doint it wrong can turn your PC into a brick.

Let me repeat that: Doing the registry edit INCORRECTLY CAN TURN YOUR PC INTO A BRICK. A DOORSTOP. A BOAT ANCHOR. MKAY!!??

So proceed with caution!

So what do I do?

Well, first of all, be aware of the above. I will take no responsibility for you turning your machine to scrap metal by a clumsy registry edit. Really. Proceed at your own risk, your own peril, and totally on your own responsibility. If you work at a studio, and the machine is not your own, talk to the TD or machine responsible person, point him to this webpage, and let him do it.

Okay?

Made that clear?

If things blow up, do not come crying to me. I warned you. Ok? Good.


The registry key in question is (and if you don't know how to edit the registry, you probably shouldn't be trying this...):


HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows


The default value of this key is something like this

  %SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows
  SharedSection=1024,3072,512 Windows=On SubSystemType=Windows
  ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3
  ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off
  MaxRequestThreads=16

...except it's all in one gargantuan long line, requiring you to scroll horizontally. See that "SharedSection" thing in there? After it it says "1024,3072,512"? Well, the 3072 in the middle, that's the "3 megabytes" we are talking about. 4 megabytes is 4 * 1024, i.e. 4096. So what we want to do is change "3072" to "4096".

Having done that, and rebooted, you should be gold.

More information about the desktop heap can be found here, or using, well, Google.

/Z