We've encountered an issue where if user-A has referenced/sublayered a USD file in solaris from our file server, then user-B tries to overwrite that USD file, they can't because the file is locked. As you can imagine, this makes a collaborative pipeline based on USD problematic. If you have a pipeline where someone doing lighting is referencing a layout USD, which is in turn referencing a bunch of model assets, the only way to update an asset is if everyone in the whole pipeline closes houdini so that asset file can be unlocked and overwritten.

From my understanding, the problem is that Pixar code keeps referenced files opened, and SMB security doesn't allow open files to be altered in any way. There is another thread where this problem came up and it sounds like some people might have worked around it by switching their file sharing from SMB to NFS. I tried this and it did seem to eliminate the file locking issue, but introduced a host of other issues. I experienced slower performance and several seconds long lag when accessing NFS shares. I was getting a lot of crashing in houdini, I suspect caused by that lag, although I can't be sure. The crashing did go away when I switched back to SMB though.

I have two questions for the forum:

1 - What is the current thinking on this problem? Do we know if Pixar is aware it and working on a solution?
2 - Has anyone configured their pipeline to work around this problem? Maybe by referencing copies of USD files, so as not to lock the originals, or something more clever?

This is another thread where people are talking about this issue, as well as others. This thread was getting confusing, so I wanted to start a clean one to focus just on the locking of referenced files on SMB shares.
https://www.sidefx.com/forum/topic/70960/?page=1 [www.sidefx.com]

Thanks,

Brad T.

Few days ago there was an interesting temp solution posted on Github, but not sure why author took it down. Perhaps he found a flaw?
https://github.com/PixarAnimationStudios/USD/issues/849 [github.com]

He suggested to force SMB Dialect (version) to 2.1 instead of 2.0.1 or 3.x.

You can check current SMB connections with this command: Get-SmbConnection

I dont have either network shares or NAS, so cannot test it. But following this issue closely, as I will transition to NAS storage sooner or later. Very concerning for Windows users.

From what I've seen, this isn't a USD-only problem, but also affects other binary file formats like Alembic. Are you able to write to .abc files while they're being read by other users/processes? If so, maybe those solutions/workarounds would work for binary USD files as well.

It might be useful to think of binary USD as a streamable format. The idea is that prim and attribute data gets pulled out the file(s) on demand, only when needed. Since file sizes can be huge and contain data for many frames it's an important strategy for good performance, but at an operating system level the file handle(s) need to stay open to make it work. Bugs aside, USD isn't really locking the files as such, it's simply keeping the file handles open while in use. That's not a whole lot different to having a movie file that you expect can be written over while someone else is watching and everything be ok.

From a pipeline perspective one of the most common patterns to avoid read/write contention is simply never re-write a file. Each time a file is output that must be consumed by multiple users, it gets some kind of version suffix or placed in a unique versioned directory. That's nice because it prevents having the rug from pulled under you while you're working. On the reader side, and depending on your workflow, it might by super annoying and time consuming to keep re-pointing your file paths at the latest versions of whatever files you need. USD has path resolver plugins that can help with that. So in your scene files (LOP nodes or other USD files) you can use a version-less path or uri that doesn't actually exist, and the path resolver will know how to convert that to a real file. Another strategy is to maintain a version-less symlink that points to the latest version file.

Ascii USD files on the other hand work differently in that they aren't streamable; the contents get immediately loaded into memory and then the file handle is closed. Using ascii USD is therefore one way to workaround all this but it's not ideal because the file sizes are much larger and loading the entire contents into memory upfront is sort of like loading a 25G blu-ray movie into memory before you start watching. So of course the scalability is limited because a large real-world scene might not fit into available memory. Ascii USD is really intended for simple "structural" files that reference/sublayer other binary files holding real data.

I know that doesn't help with the Windows/Samba issue but hopefully gives a little context as to why binary USD works like does.

Antc, your post is super helpful. Thank you. It's giving me some ideas on how to work around the problem. It sounds like the USD path resolver and versioned files is an elegant approach but maybe more wizardry than I can manage. Writing some python scripts for managing symlinks might be within my skillset though.

BradThompson
Antc, your post is super helpful. Thank you. It's giving me some ideas on how to work around the problem. It sounds like the USD path resolver and versioned files is an elegant approach but maybe more wizardry than I can manage. Writing some python scripts for managing symlinks might be within my skillset though.

what is usd path resolver?

USD's "asset resolution" mechanism is described here: https://openusd.org/dev/api/ar_page_front.html [openusd.org]