Hi - does Karma have any capacity for locally caching texture files once it's loaded them over the network?

I see HOUDINI_RAT_LOCALCACHE is available for Mantra (https://www.sidefx.com/docs/houdini/ref/env.html) - would that work for Karma too?

We're seeing significant network thrashing when rendering scenes with large numbers of 4k UDIM textures and want to explore all options form minimising the impact of network load - any suggestions welcome!

cheers
Ben

(running 19.0 on windows - 19.5 not an option currently)

Very interesting, I've noticed this just today, too. Lots and permanent network IO due to texture use.
If we disable the textures, IO goes down. It's at a level where it saturates the network.



Any input from the devs would be very welcome.

Thank you

Update, we went through the process of NOT using RATs and convert to EXR with iconvert instead.
Low and behold, the network load is back to regular levels.

Edit: While things are partially better, there is still so much more load than on the fly converted TIFs or PNGs. I fully understand the consequences of such a conversion on memory pressure, yet I fail to see the use of the texture cache if there is constant access to the textures causing IO.

We assume there must be a bug when using RATs. From what it looks like, RAT files are constantly read/stream during rendering, causing network saturation. Apparently, the RAM texture cache is ignored for unknown reasons.

Andy_23
Update, we went through the process of NOT using RATs and convert to EXR with iconvert instead.
Low and behold, the network load is back to regular levels.

Edit: While things are partially better, there is still so much more load than on the fly converted TIFs or PNGs. I fully understand the consequences of such a conversion on memory pressure, yet I fail to see the use of the texture cache if there is constant access to the textures causing IO.

We assume there must be a bug when using RATs. From what it looks like, RAT files are constantly read/stream during rendering, causing network saturation. Apparently, the RAM texture cache is ignored for unknown reasons.

are you also using itxmake? I don't think iconvert can create mipmapped exr. Non-mipmapped exr are used with texture streaming instead of converted to an in-memory rat file.

The texture cache settings only help when textures need to be evicted. If new textures are being found as ray traversal populates the scene, I/O will continue to happen.

Thank you for joining the conversation.

Yes, I have tried imaketx and iconvert. Both with the same outcome in terms of network load.

I may be ignorant about the texture caching system, but my observation seems to show, that textures are permanently streamed — for every pixel, or even sample.

This does not compute for me, as it limits rendering performance by I/O to the textures, and it shows in our case.
I wonder if there's an option to optimize that?

I'm also curious of the network topology on larger farms, maybe I can forward that to out IT department.

Thank you.

More experiments show, that using .tx instead of .rat/.exr converted with imaketx betters the situation by a lot.
Instead of 300 seconds/frame due to network I/O we are now seeing 90 seconds/frame. Same as the workstations.

Hi. Can Karma use TX as well as RAT ?

Best

Mark Wallman
Hi. Can Karma use TX as well as RAT ?

Best

Since Karma uses OpenImageIO, yes.

Hi.

How are you converting to TX? iconvert does not seem to work. And ImakteTX does not seem to be able to convert it going by the help . Best. Mark

If you can enable logging/verbosity, it should help provide more clues; though as of 19.0 those may not be as clear as 19.5. There have also been improvements to texture reading in H19.5, but those probably haven't/aren't backportable to 19.0...

If you can compare 19.0 and 19.5, that would be helpful to know. When it comes to network IO, particular configurations can change results wildly. A common recommendation, because network IO can/does affect streaming USD files from disk, not just textures, is to use nfscache. But it would be good to know if the experience in 19.5 is any better

Thanks Goldleaf - we'll be sure to give 19.5 a comparative test when we're able to pick it up.

Just to help us get a completely clear picture: in 19.5 is there any specific mechanism for caching textures to local disk once read over network, or does Karma only cache into RAM and stream via network IO the rest of the time?

I'd like to reopen this, we're having this exact issue right now. Does HOUDINI_RAT_LOCALCACHE work with karma?

I've tried everything, upping the cache limit, even lowering texture quality. But our network is constantly bogged down by non stop requests to read these rat files. Is there no way to tell karma to just put the entire file in memory? Or are we going to have to start making .tx files instead?

You can use the HOUDINI_TEXTURE_DISK_CACHE variable to define a location folder.
With that, take care to set explicitly enough space available with the "size" parameter.
HOUDINI_TEXTURE_DISK_CACHE on documentation [www.sidefx.com]

but I saw slower peroformance when using texture disk scache so test before to make sure it is worthwhile