Yer in a "Heap" of trouble now: Avoiding random crashes in Maya/Max due to running out of Windows resources!

Note; This tip has nothing to do with mental ray at all, but since the issue I describe tend to manifest on Maya when opening the mental ray control panel (due to the sheer number of tiny windows w. subwindows) I mention it here.

Have you ever run, for example, Maya (on Windows XP or earlier - Vista works better here) with a ton of other programs running, and tried to open a particularily complex AE template for some shader, to find the application go "plop" and disappear?

While there are any number of probable causes, I was debugging a particular case and actually found the cause, and it was sometihng I never really suspected, and is pretty non-obvious. And doesn't have anything whatsoever to do with mental ray, as I said above.

So what's up?

Back in the day of 16 bit Windows 3.0, running out of "window handles" (each window on screen has a "handle", like an identifier, used by the operating system to keep track of it) was extremely common, and every Windows user recognized the symptoms... windows wouldn't open, or buttons/texts would be missing from dialogs, or windows opened as only borders, or a strange "placeholder text version" of dialogs popped up.... whatever.

You'd think that a "modern" operating system wouldn't have a limit on the number of windows it could have opened, right? With multipled gigabytes of RAM, this shouldn't be an issue... right?

Well... think again. ;)

First of all the number of possible handles is apparently - still - limited to a 16 bit number (that's 65536 possible handles). But the real limit doesn't lie there, coz that's plenty. The real limit lies in the fact that each "window handle" also implies a memory allocation of a special data structure, that lives in a special memory area called the "desktop heap". And this memory area is small. As a matter of fact, by default, on XP (32 AND 64 bits) ... it's 3 megabytes!!!! Thankfully, on Vista it's larger, so Vista users do not need to do anything (except cry over all the other issues with Vista.... but that's for another blog day...)

Practically, this means that at around a few thousand windows (or other handles), things start failing.

Now, the user interface in Max, Maya and XSI are "windowed" UI's. This means that each control in each dialog is considered a "window" (from the point of view of the operating system). And sub-dialogs, little rollouts, etc. are all subwindows, with controls sub-sub-windows, and buttons can be sub-sub-sub-sub-windows, etc.

Anyone who has seen a compelex Max dialog box or Maya UI, and consider that counting every single little box, control, and thing, you'll find that that's a lot of windows. Maya easily consumes a couple of thousand handles all on it's own. (You can see this by telling your task managers "processes" tab to display the "GDI OBJECTS" column)

(Incidentally, this issue is known by Microsoft, to the extent that they try to write their software "windowless", i.e. when you run, say, Internet Exploder, each little thing on the webpage is not a separate window from the OS's point of view, precicely because of this problem!)

All is lost! The Sky Is Falling! Or?

Not at all.

See, the cool thing is that you can set up the size of the "desktop heap" by fiddling with the registry.

However, there are two things to consider when doing this, one fairly benign, and one less so.

The benign issue is that the "desktop heap" actually comes out of a larger memory area called the "Session View" (don't you just love these names?). And the size of the "Session View" is also limited (to 48Mb by default). Which is again adjustable, but that in turn lives in Kernel address space, which also is limited... to another, larger, value. And so on. You get the idea.

So the point is, you can increase the "desktop heap" size by editing the registry, but you can't just do it willy nilly and put in any old value, because you will get other problems, of running out of "Session View" space.

What I did was to up it from 3Mb to 4Mb, and the problems I had pretty much went away immediately.

The second, less benign issue, is that this is a really hairy registry location, and a horribly convoluted registry key, which actually contains a long string of some 200 characters where you have to change one thing in the middle, and the problem is the fact that doint it wrong can turn your PC into a brick.

Let me repeat that: Doing the registry edit INCORRECTLY CAN TURN YOUR PC INTO A BRICK. A DOORSTOP. A BOAT ANCHOR. MKAY!!??

So proceed with caution!

So what do I do?

Well, first of all, be aware of the above. I will take no responsibility for you turning your machine to scrap metal by a clumsy registry edit. Really. Proceed at your own risk, your own peril, and totally on your own responsibility. If you work at a studio, and the machine is not your own, talk to the TD or machine responsible person, point him to this webpage, and let him do it.


Made that clear?

If things blow up, do not come crying to me. I warned you. Ok? Good.

The registry key in question is (and if you don't know how to edit the registry, you probably shouldn't be trying this...):

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows

The default value of this key is something like this

%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows
SharedSection=1024,3072,512 Windows=On SubSystemType=Windows
ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3
ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off

...except it's all in one gargantuan long line, requiring you to scroll horizontally. See that "SharedSection" thing in there? After it it says "1024,3072,512"? Well, the 3072 in the middle, that's the "3 megabytes" we are talking about. 4 megabytes is 4 * 1024, i.e. 4096. So what we want to do is change "3072" to "4096".

Having done that, and rebooted, you should be gold.

More information about the desktop heap can be found here, or using, well, Google.



Ino said...

WOW !!! thx again zap

Stephan Wagemann said...

iam on xp64 and it looks like its bigger than xp32 :)

%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,768 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ProfileControl=Off MaxRequestThreads=16

Sylvain Berger said...

Man you are scaring me with your knowledge ... and your ease at explaining complex stuff and making simple and interesting! Great post!

coolroy said...

hi zap!
why your tips are so max and maya specific?
XSI uses mantal ray too and its default render there.

Master Zap said...

Coz in XSI everything works perfectly anyway, without tips? ;)

Honestly, stay tuned, XSI tips will be coming.


coolroy said...

Good news, thanks!

kevin said...

Hey Zap, has it been known for mental ray in max9 to exceed these limitations in the process of rendering a frame? If so, what would impact, if any, would it have on the resulting render? a crash? a less accurate final gather solution? any visual difference?

Master Zap said...

Hey Zap, has it been known for mental ray in max9 to exceed these limitations in the process of rendering a frame?

THIS problem? No, never. mental ray doesn't use window handles (at least, if one or two is used for some socket communication, those are established when ray starts).

But natuarally you can run out of any other finite storage....


GeckoKid said...

hello zap, i have been reading your blog for a while now, and all those tips are very helpful :)

however I have this problem when rendering, i have done a short animation, it's nothing very heavy, about 70000 polys and 5 lights, 300 frames...

but it's not really that heavy like those movie quality stuff! anyway my render just decides to not want to render frames 222 or 228 depending on the set up (i moved the camera a bit) while i could render the frames after it, i wonder why it just decides to crash at that point? i tried rendering at 1280*4xx and 200*100 (!) and both lead to the same crash...

here's a screenshot of the crash:


i really hope you could help, as I was looking on google but i get no conclusive solutions!

thanks in advance!

Aaron Bockelie said...

Hi, just thought I'd let you know you need to change allocation of two other constants in order for this to be fixed correctly, GIDProcessHandleQuota and UserProcesHandleQuota. Otherwise you're just giving more memory to the same finite number of handles without actually allowing more handles to spawn. When setting the session manager memory limit, you can safely set memory to 8192 on 32 bit windows versions, as well.

GDIProcessHandle's max value is 16384, and User is 18000.

For the comment about xp64 the reason your numbers are bigger is because your address space (64 bits) allows much larger memory allocation for handles.

I hope it fits in the comment but here's a reg key that you can copy into a text file and save as .reg and it will set user and gdi sessions to 16384 and session manager to 8mb.

Copy and paste everything below this line in my comment into the .txt file and rename it to .reg. (If renamed correctly the icon turns into a picture of little blue squares)

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session Manager\SubSystems]

[HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Windows]

Aaron Bockelie said...

Ok, so the hex data doesn't copy correctly out of comments, so don't use that part. Ugh. Just use zap's directions but change the memory to 8192 instead. The other two entries under [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Windows] work right though...