I’ve been running into a couple of issues where my systems are crashing, with GPUs failing during renders and similar problems. Several threads are running where this behavior is observed, but in short, this one project—which is rather complex with a lot of scattered planes and objects—is crashing nonstop.

Here’s one of those threads: https://www.sidefx.com/forum/topic/100169/. [www.sidefx.com]

In another pass on the same project, it’s crashing completely, not even falling back to CPU rendering. I’ve attached a worker log from that case, along with an analysis from Grok, if it means anything.
In this scenario, I have two single RTX 3090 systems that are happily rendering with no problems at all, while all of my RTX 4090 systems are crashing with this error.

So on one side, there’s an error like in the first example: "KarmaXPU: device Type:Optix ID:0 has registered a critical error, so will now stop functioning. Future error messages will be suppressed."

In another case, it’s a complete crash, as described in the attached files.

We’re out of clues. The deadline to deliver this project is today, and rendering it has been a nightmare. On top of that, the project is 70 GB, but the upload limit for support is 10 GB, and they can’t download directly from my NAS due to external network restrictions. So, we’re hitting a wall at the moment.

So far it looks mostly like Karma XPU and VRAM on GPUs issue but then again why 3090 is rendering fine...
Out of clues.. Any insights would be wellcomed... Simply hard to understand that With Karma XPU we cannot render scenes like this ...

Just a small update: With pruning and hiding everything possible—like people and props inside the building (aerial shot with environment, plants, and buildings)—the scene blooms and renders much faster with no crashes. Now I’m wondering: Does Karma load absolutely everything in the scene, even if it’s not visible? Do we need to manually hide objects nonstop?

Not sure if you know this already but there is a kind of a hidden toggle inside Display Options(D in the viewport)>Guides>Render Stats.

Makes it allot easier to debug the memory usage in realtime and where the memory goes.

But every object that is not hidden should be loaded incase a ray is going to hit it I would think.

Thanks!!
This helped me, I wasn't aware of it before.
I went through the scene, pruning all and turning them off one by one following VRAM usage..

And yes finally it was a simple VRAM crash after all.
Was able to locate a faraway scattered forest that is using up 8.5GB in geometry...
So it is just a matter of reducing that impact on VRAM budget I guess..

Wasted days chasing segmentation errors, optix fail, and simial crashes...

Thanks again for this info Heileif!