How to run concurrent batches?

   Views 506   Replies 0   Subscribers 0
User Avatar
Member
28 posts
Joined: Aug. 2020
Offline
Hello,
I have roughly the following scenario. There are X number of .exrs on disk and for each one I give them custom attributes in TOPS like distortion, color etc... which I use to drive a Copernicus network and create multiple variations. For example shot_001.exr gets 20 different color variations. Where I am at right now is i have all the work items set up and I am ROP Fetching the ROP from Copernicus.

The largest slowdown is the File I/O in Copernicus, its taking 4 seconds to load the exr and it really only needs to be done once per unique exr instead of for every work item. I resorted to batching to solve this and ran a test with a single exr input and it was extremely fast(~300 outputs in 25 seconds) due to what i assume was it only loading the exr a single time. Not sure if this would also be achievable with services, by default it doesnt not seem to be?

Where i am stuck right now is extending the batch behavior to the multiple inputs that can run at the same time. Currently I have 26 input exrs and some of them have 20 variations, others have 200. I would like to run 3~4 instances at the same time where each instance is working on a batch, which is the variations of a single input exr and therefore only loads the exr once.

I've got the scheduler set to total slots and slots per work item to control resources and max concurrent processes. I tried defining batches with the frame and range attributes then using the "frames per batch" parameter on the ROP Fetch to get this behavior but I believe this doesn't work with a changing batch size? In my case some exrs have more/less variations. Another problem i encoutered was high vram usage (8gb per instance w/ 50mb exr) and random slowdowns towards the end of batches(going from .25 seconds coook to 5 seconds).

Update:
Also noticed that my items are shifted by one. For example i have a work items with attributes color_name (for the top output name) and color (for the color used in cops). with red, green and blue. I have to shift the color attribute back by one so one work item had color_name=red, color=green then color_name=green,color=blue
Edited by Latimerias - Aug. 22, 2024 04:29:19
  • Quick Links