Husk.exe stuck not finishing task

   7688   32   11
User Avatar
Member
61 posts
Joined: Oct. 2013
Offline
We've gone back to rendering in Houdini 19.0 for now, which is an awful solve even short-term as the resolver format changed between the two versions so we're being forced to flatten scenes down before sending them to the farm.

It's definitely frustrating that multiple studios have flagged this for multiple renderers and yet no one seems to have any ideas.

Personally I feel like if the same RenderMan version stalling in 19.5 and not in 19.0 that puts the ball in SideFX court but ultimately we just need it fixed :\
Grant Miller
VFX Supervisor
Ingenuity Studios
User Avatar
Member
44 posts
Joined: Feb. 2020
Offline
Agree ^
User Avatar
Staff
2617 posts
Joined: July 2005
Offline
Does someone has found a solution to husk getting stuck. I am having this problem when trying to render volume.
I have tried with all the Houdini 20 update. Now I am using 20.506 and I still have the problem.

https://www.sidefx.com/changelog/?journal=20.0&categories=58&body=&version=20.0&build_min=582&build_max=582&show_versions=on&show_compatibility=on&items_per_page= [www.sidefx.com]

There was a hanging issue with volumes fixed in 20.0.582 (which 20.0.506 seems quite old).

We have tried to reproduce the issue with husk hanging when rendering but without any success. If some renders complete, but take longer, then, it sounds like it might be something other than a hanging issue. We've also been unable to reproduce this.

As a note, if you're rendering on Linux or OSX, you can send husk a USR1 signal, which will cause husk to save out a snapshot of the current state. Not sure if this might help.

It is something we're continuing to look into.
User Avatar
Member
4 posts
Joined: Sept. 2016
Offline
The update has resolved the problem and now volume for me are rendering fine.
Thanks
User Avatar
Member
44 posts
Joined: Feb. 2020
Offline
I am still running into the same issue as originally when I started this thread. I can see some comments about volumes being fixed, but I just want to iterate that this is very much still an issue.

1. The problem is that when launching a task via Husk it gets stuck after a certain random frame and the process does not "Complete" and therefore Husk is unable to move onto the next frame. Again this is random. It can occur on frame 1001 and just as likely on frame 1017 for example.

2. We have noticed that the problem seems to occur more frequently (but not exclusively) on larger scenes. E.g many dependencies or larger polycount scene. I will try and flatten down a scene where the issue can be reproduced and send it across to SideFX, but honestly this can be very difficult as often the layered nature of many USD files can be difficult to flatten down and work 100% the same as the non flattened version. The scene I am trying to render now have over 25,000 descendants for example, which is made up of 1000's of .USD files on disk.

3. When SideFX says they have investigated the problem and can NOT reproduce the issue, I would urge SideFX to share the details or share their scene so we can see what has been tested and how. A sphere in a scene with some lights will most likely work fine, it is when the scene complexity starts to rise the issue occurs.


I launched a render again yesterday and was rendering for + 10 hours. It rendered 1 frame and I got a .exr file on disk. However the "task" never completes and husk continues to use 4-5% of my CPU and keeps hanging unable to move to the next frame. I launched the same job this morning and it then got stuck on frame 1009, so again random. I am on Houdini 19.5.640, RenderMan 25.2, Windows 11.

Here is the process stuck in Windows Task Manager:


Here is the task in PDG Window, forever cooking:
Edited by kskovbo - Feb. 12, 2024 09:00:01

Attachments:
Screenshot 2024-02-12 095131.png (243.6 KB)
Screenshot 2024-02-12 095005.png (15.6 KB)

User Avatar
Member
9 posts
Joined: Feb. 2020
Offline
I have the same issue, not sure if this is relevant but I recognized it more often when I added cryptomatte. anyone find a solution? thx

houdini 20.0.547 and karma
Edited by zatlax - Feb. 20, 2024 06:27:03
User Avatar
Member
5 posts
Joined: June 2022
Online
I'm having the same issues with USD renders "Hanging" on random frames. I have a fairly complex scene but have also had the issue on fairly simple scenes. The only solution at the is to constantly baby the HQ render farm and restart and render that looks to be hanging.

Which is incredibly painful.

Houdini 20.0.653 & Arnold 6.3.1.0
User Avatar
Member
2 posts
Joined: Jan. 2015
Online
Unfortunately I do have the same problem.
Houdini 19.5.640 Py3.7 on Windows 10 Renderman 25.2

In 99 percent of these cases the render hangs after ALF_PROGRESS 100%, but sometimes the following error message appears on those stuck renders.
Hopefully this helps with finding the issue?

Thank you very much.
Martin

ALF_PROGRESS 98%
ALF_PROGRESS 99%
ALF_PROGRESS 100%
HDF5: infinite loop closing library
      D,T,F,FD,P,FD,P,FD,P,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E,E
Edited by mrtn - May 13, 2024 11:40:04
User Avatar
Member
7 posts
Joined: Nov. 2014
Offline
I have the same or simular issue that I can send Jobs to the farm via Deadline, but it looks like the render progress hangs. Even if frames are rendered, he is not finishing the task and continues with the next frame.

Houdini 20.0.688 and Karma. Submitted via Stage Editor and USD Render Rop. Even if you do it via OUT/ Fetch and Deadline Submitter Node I have no success. But I also have Cryptomattes active.

Cheers,
Kay
User Avatar
Member
32 posts
Joined: Aug. 2018
Offline
I'm having the same issues on Apple Silicon, Houdini 20.0.724, and Karma - husk could get stuck after a few frames, or a few hundred, and continues to consume CPU. I've let it run for hours, and it never finishes.

Cheers,
Tim
User Avatar
Member
1 posts
Joined: Nov. 2017
Offline
kskovbo
I am still running into the same issue as originally when I started this thread. I can see some comments about volumes being fixed, but I just want to iterate that this is very much still an issue.

1. The problem is that when launching a task via Husk it gets stuck after a certain random frame and the process does not "Complete" and therefore Husk is unable to move onto the next frame. Again this is random. It can occur on frame 1001 and just as likely on frame 1017 for example.

2. We have noticed that the problem seems to occur more frequently (but not exclusively) on larger scenes. E.g many dependencies or larger polycount scene. I will try and flatten down a scene where the issue can be reproduced and send it across to SideFX, but honestly this can be very difficult as often the layered nature of many USD files can be difficult to flatten down and work 100% the same as the non flattened version. The scene I am trying to render now have over 25,000 descendants for example, which is made up of 1000's of .USD files on disk.

3. When SideFX says they have investigated the problem and can NOT reproduce the issue, I would urge SideFX to share the details or share their scene so we can see what has been tested and how. A sphere in a scene with some lights will most likely work fine, it is when the scene complexity starts to rise the issue occurs.


I launched a render again yesterday and was rendering for + 10 hours. It rendered 1 frame and I got a .exr file on disk. However the "task" never completes and husk continues to use 4-5% of my CPU and keeps hanging unable to move to the next frame. I launched the same job this morning and it then got stuck on frame 1009, so again random. I am on Houdini 19.5.640, RenderMan 25.2, Windows 11.

Here is the process stuck in Windows Task Manager:
Image Not Found


Here is the task in PDG Window, forever cooking:
Image Not Found

Hi every one, I have the same issue with houdini 20.0.653 and Karma. I have intel i7 13700k and nvidia 4060Ti 16Gb. I see this


Supported Pro Graphics Cards

NVIDIA RTX 6000 Ada, A6000, RTX 6000, RTX A5000, RTX 5000, RTX A4500, RTX A4000, RTX 4000

Radeon Pro V520 MxGPU
AMD Radeon PRO W7900, W7700, W6800, W6600, W6400
AMD WX 9100, WX8200, WX 7100, WX 5100 , WX4100, WX3200, WX3100
AMD Radeon Pro SSG, AMD Radeon Pro VII
AMD W9100, W8100, W7100, W5700, W5500, W5100
AMD W9000, W8000, W7000, W5000

Supported Consumer Graphics Cards

NVIDIA GeForce RTX 4070, 4080, 4090
NVIDIA GeForce RTX 3070, 3080, 3090
NVIDIA GeForce RTX 2060
NVIDIA GeForce 1080, 1070, 1060
NVIDIA GeForce 980, 970
NVIDIA GeForce Titan, Titan X, Titan Black
NVIDIA GeForce Titan Z (single GPU for rendering or compute only)
NVIDIA GeForce 780 Ti, 780, 770, 760
NVIDIA GeForce 690
in https://www.sidefx.com/Support/system-requirements/supported-graphics-cards/ [www.sidefx.com]

and I hope my graphic is not the problem, before buy it I see other people who recommended it. Any way I try to render with the usdrender_rop in xpu and cpu and in both settings husk kill the process and my screens turn black.

Any one find any solution?

Thank you very much and sorry for my english...
User Avatar
Member
44 posts
Joined: Feb. 2020
Offline
August 2024 and I am still running into this issue. Did anyone find a solution?
User Avatar
Member
44 posts
Joined: Feb. 2020
Offline
Tonight when trying to get renders out, I get the following error on frame 1002 and it was hanging. Frame 1001 went through fine.

Unable to write to image: husk:null_raster_part
Error: Failed to create file "husk:null_raster_part : Permission denied".
R56011 {WARNING} 41 objects had displacement <10.00% of bound. See statistics for details.

anyone seen this before?


Edited by kskovbo - Aug. 12, 2024 17:52:31

Attachments:
huskerror.jpg (28.9 KB)

  • Quick Links