Hi I have a setup that receives a bunch of geometry and separate them into work items which are then fed into different simulation setup in other hip files using TOP fetch. I use hqueue scheduler here to send all simulation to farm.

Things work well so far until there are a lot of work items to process. I notice that sometimes TOP fetch node generate too many TOP fetch jobs on the farm at once that quickly consume all available slots preventing the actual PDG cook jobs (sim job) to be processed.

The outcome is that all jobs are waiting for resource and never get done.

Is there anything I miss that cause this issue and how can I fix it?