Geometry Clip Sequence Lop or how to handle large data
8571 38 11- sekow
- Member
- 238 posts
- Joined: Nov. 2013
- Offline
Hey.
I am interested in good practice of handling large caches.
Love the way how fast it is to work in Lops but as soon as there are time dependencies involved all that gets slow really fast ...
I understand the way time is handled in Solaris and I am using the "cache lop/ write usd to disk" quite frequent.
Now I am usually working with heavy fx caches. Massive Rbds, surface meshing, volumes , insane point counts etc.
And as those are already cached out as bgeo/vdb sequences I thought that the Geometry Clip Sequence could handle that.
But I've learned that I need to sample a frame range to get rid of time dependencies again, so I need to write out those caches as usd anyway. .
Now I can reference those back into the stage... Sounds redundant to me and disk space is sacred.
Have a look at the example hip file where I explore that a bit.
I am interested in good practice of handling large caches.
Love the way how fast it is to work in Lops but as soon as there are time dependencies involved all that gets slow really fast ...
I understand the way time is handled in Solaris and I am using the "cache lop/ write usd to disk" quite frequent.
Now I am usually working with heavy fx caches. Massive Rbds, surface meshing, volumes , insane point counts etc.
And as those are already cached out as bgeo/vdb sequences I thought that the Geometry Clip Sequence could handle that.
But I've learned that I need to sample a frame range to get rid of time dependencies again, so I need to write out those caches as usd anyway. .
Now I can reference those back into the stage... Sounds redundant to me and disk space is sacred.
Have a look at the example hip file where I explore that a bit.
Edited by sekow - Dec. 7, 2023 11:58:16
- tamte
- Member
- 8833 posts
- Joined: July 2007
- Offline
didn't check the file, but after watching H20 Solaris presentation there seem to be some improvements in that area
1. For rigid bodies there is a new "procedural" mode
https://youtu.be/R4SLw5EdzQ8?si=tF-h1z6FA8QYPJno&t=1450 [youtu.be]
2. Geometry Clip Sequence LOP seems to be able to reference .bgeo* files directly and I'd assume probably other like .vdb, so I'd assume there should be no need to recache them again as .usd, but I haven't tried
https://youtu.be/R4SLw5EdzQ8?si=NZ4F9U2zwDvISC0s&t=1815 [youtu.be]
1. For rigid bodies there is a new "procedural" mode
https://youtu.be/R4SLw5EdzQ8?si=tF-h1z6FA8QYPJno&t=1450 [youtu.be]
2. Geometry Clip Sequence LOP seems to be able to reference .bgeo* files directly and I'd assume probably other like .vdb, so I'd assume there should be no need to recache them again as .usd, but I haven't tried
https://youtu.be/R4SLw5EdzQ8?si=NZ4F9U2zwDvISC0s&t=1815 [youtu.be]
Edited by tamte - Dec. 7, 2023 13:57:16
Tomas Slancik
FX Supervisor
Method Studios, NY
FX Supervisor
Method Studios, NY
- sekow
- Member
- 238 posts
- Joined: Nov. 2013
- Offline
tamte
2. Geometry Clip Sequence LOP seems to be able to reference .bgeo* files directly and I'd assume probably other like .vdb, so I'd assume there should be no need to recache them again as .usd, but I haven't tried
https://youtu.be/R4SLw5EdzQ8?si=NZ4F9U2zwDvISC0s&t=1815 [youtu.be]
Hey Tomas, hope you good!
yeah, in the presentation you can see the lop went time dependent. Which will recook the whole graph later on.
Geometry Clip Sequence Lop makes it a bit more convenient to prep the cache out to usd though.
- sekow
- Member
- 238 posts
- Joined: Nov. 2013
- Offline
- RiotPixels
- Member
- 12 posts
- Joined: Jan. 2017
- Offline
It can be done using value clips. I'm not sure why this is so unpopular. Should the first thing in any solaris tutorial.
1. create a bgeo cache and make sure you have the usd timesamples on the geo (usd configure).
2. Create a Reference Node and point it to the first bgeo.
3. Create a value clip link the sequence and also the manifest file. All can be bgeo.
1. create a bgeo cache and make sure you have the usd timesamples on the geo (usd configure).
2. Create a Reference Node and point it to the first bgeo.
3. Create a value clip link the sequence and also the manifest file. All can be bgeo.
- spektra
- Member
- 27 posts
- Joined: June 2022
- Online
i've been trying to figure out the best workflows with geometry clip sequence lop as well. since now ive used a usd rop followed by the value clips rop which works but isnt ideal.
i have tested the geometry clip sequence - loading the bgeo sequence into it, unchecking the flatten clip files and sampling my frame range. after its cached the files, its fast and renders nicely on the farm.
the only problem i've encounter thus far is that if my bgeo sequence changes, it still shows the old one until i submit a new render. im guessing because the topology file gets overwritten only as i save the usd out in the end. so would be nice to have a way of writing them out on the node or maybe im still misunderstanding the best use case.
i have tested the geometry clip sequence - loading the bgeo sequence into it, unchecking the flatten clip files and sampling my frame range. after its cached the files, its fast and renders nicely on the farm.
the only problem i've encounter thus far is that if my bgeo sequence changes, it still shows the old one until i submit a new render. im guessing because the topology file gets overwritten only as i save the usd out in the end. so would be nice to have a way of writing them out on the node or maybe im still misunderstanding the best use case.
- robp_sidefx
- Staff
- 503 posts
- Joined: June 2020
- Offline
- sekow
- Member
- 238 posts
- Joined: Nov. 2013
- Offline
robp_sidefxsekow
in the presentation you can see the lop went time dependent
Yes, one of the things I didn't show (but really should have) is that, as with many of our other LOPs, we can generate a range of time samples directly from this LOP and avoid introducing the time dependency.
Its just that usually those geo caches are several Gb big, and in my case its a fairly long shot. In that case I wont be going around the fact that I need to cache those timesamples out as USD (via the geometry clip seq lop).
Just making sure there is not an other way of handling that.
- robp_sidefx
- Staff
- 503 posts
- Joined: June 2020
- Offline
sekow
I wont be going around the fact that I need to cache those timesamples out as USD
If you have "Flatten Clip Files" disabled (which I appreciate I left enabled in the screenshot & example above), then there shouldn't be any BGEO->USD conversion. You'll end up with a direct link back to the BGEO in your scene.
def Xform "geoclipsequence_noTimeDep" (
clips = {
dictionary default = {
double2[] active = [(1, 0), (2, 1), ...]
asset[] assetPaths = [@../geo/sphere.0001.bgeo@, @../geo/sphere.0002.bgeo@, ...]
- sekow
- Member
- 238 posts
- Joined: Nov. 2013
- Offline
robp_sidefx
If you have "Flatten Clip Files" disabled (which I appreciate I left enabled in the screenshot & example above), then there shouldn't be any BGEO->USD conversion. You'll end up with a direct link back to the BGEO in your scene.def Xform "geoclipsequence_noTimeDep" (
clips = {
dictionary default = {
double2[] active = [(1, 0), (2, 1), ...]
asset[] assetPaths = [@../geo/sphere.0001.bgeo@, @../geo/sphere.0002.bgeo@, ...]
thanks, getting closer to understand what is going on here.
But I still need to cache all time samples into memory first (setting the geo clip seq lop to a framerange). So the whole framerange.
Other than writing a python script is there a way to avoid loading those time samples into the stage beforehand?
I am just trying to avoid sitting here and waiting for 500 frames of heavy geometry getting loaded into ram just to be written down into a dictionary inside the usd.
I guess I could come up with and hda/top network where the manifest, topology and reference usd will be written out immediatley after caching out the bgeo files.
- tamte
- Member
- 8833 posts
- Joined: July 2007
- Offline
sekowWhy would it need to load any file to memory?
I am just trying to avoid sitting here and waiting for 500 frames of heavy geometry getting loaded into ram just to be written down into a dictionary inside the usd.
It just needs to write timesamples for file paths over that range, I imagine that's reasonably fast
But again, I haven't tried, but that's what I assume it would do in that mode
Edited by tamte - Dec. 8, 2023 08:53:35
Tomas Slancik
FX Supervisor
Method Studios, NY
FX Supervisor
Method Studios, NY
- protozoan
- Member
- 1717 posts
- Joined: March 2009
- Offline
tamte
Why would it need to load any file to memory?
It used to be like that, but that was a long time ago and now should not be necessary anymore. The value clip generation should zing just through, even for large datasets.
robp: A lot of people seem to struggle with this. Maybe this warrants a case-study tutorial like "Solarifying production caches efficiently from start to finish" without skipping any checkboxes etc.
Edited by protozoan - Dec. 8, 2023 10:48:06
Martin Winkler
money man at Alarmstart Germany
money man at Alarmstart Germany
- tamte
- Member
- 8833 posts
- Joined: July 2007
- Offline
How is geometry interpolation between time samples handled in value clips currently?
- Does USD figure it out when sampling the value clip? Similarly to Packed Disk Sequence prims?
As that would be ideal since topology can differ in so many ways including attribute presence
- does it currently interpolate geo at all when sampling times between 2 authored time samples?
- I can imagine this could be prerecorded in "times" metadata during creating of the value clip and for that it would indeed need to load the geo for every file to figure it out, is that what's happening?
- Does USD figure it out when sampling the value clip? Similarly to Packed Disk Sequence prims?
As that would be ideal since topology can differ in so many ways including attribute presence
- does it currently interpolate geo at all when sampling times between 2 authored time samples?
- I can imagine this could be prerecorded in "times" metadata during creating of the value clip and for that it would indeed need to load the geo for every file to figure it out, is that what's happening?
Tomas Slancik
FX Supervisor
Method Studios, NY
FX Supervisor
Method Studios, NY
- Hamilton Meathouse
- Member
- 200 posts
- Joined: Nov. 2013
- Offline
protozoantamte
Why would it need to load any file to memory?
It used to be like that, but that was a long time ago and now should not be necessary anymore. The value clip generation should zing just through, even for large datasets.
robp: A lot of people seem to struggle with this. Maybe this warrants a case-study tutorial like "Solarifying production caches efficiently from start to finish" without skipping any checkboxes etc.
A big +1 for this idea
- spektra
- Member
- 27 posts
- Joined: June 2022
- Online
tamtesekowWhy would it need to load any file to memory?
I am just trying to avoid sitting here and waiting for 500 frames of heavy geometry getting loaded into ram just to be written down into a dictionary inside the usd.
It just needs to write timesamples for file paths over that range, I imagine that's reasonably fast
But again, I haven't tried, but that's what I assume it would do in that mode
thats what im curious about too. currently it feels like there is 2 options to render bgeo caches with karma.
1)i can leave the geoclipsequence time dependent and set the usd rop to render out all frames. During the writing out, all the bgeo files are loaded.
2)i can set the sampling behavior on the geoclipsequence to sample frame range. this way the usd rop can render out one frame very fast but ill need to wait for the caching to happen beforehand.
on both cases the files written to disk (the usd file from the rop, manifest and topology) are identical and a couple of kb in size.
the question is if its somehow possible to write those files out without having to load all the bgeo files? or is that necessery for setting up the value clips correctly?
- mtucker
- Staff
- 4525 posts
- Joined: July 2005
- Offline
It is technically necessary for writing out the value clips correctly. Generating a correct value clip requires generating a manifest file. Generating a manifest file requires loading the scene graph hierarchy of every clip file, and looking for time sampled attributes, but doesn't actually look at any of the attribute _values_. If your source files are USD this is all super efficient and fast, because USD just loads the bits of the file it needs (hierarchy information, basically). If your source files are BGEO files, getting the "USD hierarchy" of the BGEO file requires doing a full load of the BGEO file into memory and a full translation of the BGEO data into equivalent USD data. This is obviously way more expensive than making the equivalent queries on USD files.
We have two options to address this problem. One would be to rewrite the BGEO file format plugin to allow it to more efficiently extract just the hierarchy information. This would be very complicated and risky, and may not even help that much given how complex the translation from BGEO data to even a USD scene graph hierarchy is. So this isn't likely to happen any time soon.
The other option (which already exists as an RFE) is to provide a checkbox on the Geo Clip Sequence LOP that says "trust me, my scene graph hierarchy doesn't change from file to file, just generate the manifest from the first BGEO file". This will make the "multi frame" cooking mode of the Geo Clip Sequence very fast even for BGEO sequences as it will only need to load one BGEO file to create the manifest. Of course depending on your BGEO sequence, turning on this check box may generate an incorrect manifest file, but at least you'll be able to control this tradeoff between speed and "correctness". We don't have a timeline for this change, but this issue does keep coming up.
And yes, we definitely need to put out more information about this process.
We have two options to address this problem. One would be to rewrite the BGEO file format plugin to allow it to more efficiently extract just the hierarchy information. This would be very complicated and risky, and may not even help that much given how complex the translation from BGEO data to even a USD scene graph hierarchy is. So this isn't likely to happen any time soon.
The other option (which already exists as an RFE) is to provide a checkbox on the Geo Clip Sequence LOP that says "trust me, my scene graph hierarchy doesn't change from file to file, just generate the manifest from the first BGEO file". This will make the "multi frame" cooking mode of the Geo Clip Sequence very fast even for BGEO sequences as it will only need to load one BGEO file to create the manifest. Of course depending on your BGEO sequence, turning on this check box may generate an incorrect manifest file, but at least you'll be able to control this tradeoff between speed and "correctness". We don't have a timeline for this change, but this issue does keep coming up.
And yes, we definitely need to put out more information about this process.
- sekow
- Member
- 238 posts
- Joined: Nov. 2013
- Offline
- sekow
- Member
- 238 posts
- Joined: Nov. 2013
- Offline
mtucker
The other option (which already exists as an RFE) is to provide a checkbox on the Geo Clip Sequence LOP that says "trust me, my scene graph hierarchy doesn't change from file to file, just generate the manifest from the first BGEO file". This will make the "multi frame" cooking mode of the Geo Clip Sequence very fast even for BGEO sequences as it will only need to load one BGEO file to create the manifest. Of course depending on your BGEO sequence, turning on this check box may generate an incorrect manifest file, but at least you'll be able to control this tradeoff between speed and "correctness". We don't have a timeline for this change, but this issue does keep coming up.
btw I would be totally fine with this extra control
- tamte
- Member
- 8833 posts
- Joined: July 2007
- Offline
Seems like this has been recently added
20.0.577
Added a new toggle to the Geometry Clip Sequence LOP which allows the topology and manifest files to be generated from a single clip file instead of using all the clip files. This can make this node much, much faster, and reduce memory usage, especially when using BGEO files for the clip files. Also made some other performance improvements to this node which should apply in all situations.
Thu. January 4, 2024
Tomas Slancik
FX Supervisor
Method Studios, NY
FX Supervisor
Method Studios, NY
- mtucker
- Staff
- 4525 posts
- Joined: July 2005
- Offline
-
- Quick Links