Houdini 19.5.435
Rocky Linux release 8.6
HtoA 6.1.4.1
Arnold 7.1.4.1
Deadline 10.2
Our studio is working on a project that has two animal heads with full grooms. We've started to run into potential memory issues on our farm and are trying to find the most efficient way to render out our shots. It renders fine locally, both with and without progressive and was rendering fine on our farm up until recently.
A little bit about the scene and the workflow :
Each animal head is composed of 5-6 groom objects. For each shot, the grooms are run through a guide deform and select groom objects get run through a guide sim, which is cached. From here, we cache out each hair generate node as a bgeo sequence for lighting.
The primary hairgen cache's for each animal are huge. A 100 frame sequence is 190.22 gigs (almost 2 gigs a frame). 76 million points. Attributes are minimal (Cd, curveu, P, v and width on points, clumpid, id and uv on prims). The rest of hair gen caches are much smaller and will render just fine on the farm -- but the primary ones must push it over the edge.
Our lighting scenes contain file nodes that reference each of these .bgeo sequences for convenience. We've tried migrating the original hairgens into the lighting file to avoid hairgen caching but it became difficult to manage since each hairgen has a sop network that needs to be kept up to date. The groom itself is not very portable since it requires juggling many object level nodes and we've yet to determine a great way to pipeline it all.
The other odd thing is that the specs on the farm nodes that are failing seem more than adequate and the log indicates that it has plenty to spare :
Worker Name: xxxxx
Version: v10.2.0.10 Release (3b87216c7)
Operating System: Linux
Machine User: xxxxxx
IP Address: xxxxx
MAC Address: xxxxxx
CPU Architecture: x86_64
CPUs: 96
CPU Usage: 0%
Memory Usage: 1.6 GB / 184.4 GB (0%)
Free Disk Space: 26.002 GB
Video Card:
This is the memory error that we're getting in the same log of the arnold task :
=======================================================
Error
=======================================================
Error: OutOfMemoryException : Exception of type 'System.OutOfMemoryException' was thrown.
at System.Text.StringBuilder.ToString()
at Deadline.IO.PathMappingUtils.c(String bxl, String bxm, String bxn, String bxo, String bxp, IList`1 bxq, Boolean bxr, DataController bxs, GenericDelegate1`1 bxt, String bxu, OS bxv)
at Deadline.IO.PathMappingUtils.CheckPathMappingInFileAndReplace(String inFileName, String outFileName, String forceSeperator, String stringsToReplace, String newStrings, DataController dataController, GenericDelegate1`1 logFunction, String regionID, Boolean readFileAsBytes)
at Deadline.IO.PathMappingUtils.CheckPathMappingInFileAndReplace(String inFileName, String outFileName, String forceSeperator, String stringsToReplace, String newStrings, DataController dataController, GenericDelegate1`1 logFunction, String regionID)
at Deadline.IO.PathMappingUtils.CheckPathMappingInFileAndReplace(String inFileName, String outFileName, String stringsToReplace, String newStrings, DataController dataController, GenericDelegate1`1 logFunction, String regionID)
at Deadline.IO.PathMappingUtils.CheckPathMappingInFileAndReplaceSeparator(String inFileName, String outFileName, String separatorToReplace, String newSeparator, DataController dataController, GenericDelegate1`1 logFunction, String regionName)
at Deadline.Scripting.RepositoryUtils.CheckPathMappingInFileAndReplaceSeparator(String inFileName, String outFileName, String separatorToReplace, String newSeparator) (Python.Runtime.PythonException)
File "/var/lib/Thinkbox/Deadline10/workers/render20166/plugins/640ddfe4df22acd4b3cdd006/Arnold.py", line 88, in RenderArgument
RepositoryUtils.CheckPathMappingInFileAndReplaceSeparator( filename, localFilename, "\\", "/" )
at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
at __FranticX_GenericDelegate0`1\[\\]Dispatcher.Invoke()
at FranticX.Processes.ManagedProcess.RenderArgument()
at Deadline.Plugins.DeadlinePlugin.RenderArgument()
at FranticX.Processes.ManagedProcess.Execute(Boolean waitForExit)
at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
Doe anyone have any insight into how to troubleshoot hair/fur/groom scenes like this? Is there a better way to optimize our groom caches for rendering with Arnold? If it's render locally on a lower spec machine than what we have on our farm -- does that make any sense?
Cheers!
