I've developed a wedging setup for our RoyalRender farm that runs primarily on ROPs, but for the contact sheet creation I do rely on PDG. My current issue is that I'm now overloading my system with a large number of work items (24k image files for 24 wedges with 1000 frames each). Simple operations like string splits and so on just lead to Houdini freezing indefinitely, before I even get to the imagemagick part. RAM or CPU don't show a spike.
I was wondering if that is just in and out of itself a ridiculous number to execute, or I'm missing some settings. I thought PDG was supposed to be scalable like this. The setup does work on smaller batches with like 300 frames*16 wedges on other shots perfectly, but even there it does take quite quite some time to compute which I would hope it somehow wouldn't.
I thought about concatenating work items into batches where one work item stores each image path as an array, but the system is quite fragile and I'd probably need to rebuild it from scratch and lose a lot of the simplicity of tops.
PDG itself can scale to orders of magnitude larger graphs, however there could be something in Houdini's UI or in the jobs themselves that are causing such lag.
Is this only happening during the cook? If so maybe Local Scheduler is running too many concurrent processes for your system to handle?
Is it possible to attach the hip file here or on a support ticket?
