Possible Memory Leak

H13

I seem to be having a very strange problem. Regular ocean surface 60 by 30 and 5 depth , passing velocities to a flip tank with similar size. Its like Igor Zanics coast beach Flip simulation

I have somehow detailed collision mesh on one side with like 200K faces and with a voxel size 0.1 it gives me sufficient detail almost 3mil VDB voxels. So VDBs as static collision( no deforming or translating collision objects here)

My machine has 32 gb ram and I am running out of memory with only around 20-25 mil points, which I believe is low as far as I can remember, Many videos on Vimeo are doing much more points with less ram.

The funny thing is I tried the same simulation on another PC with 190GB ram and 50 million particles were just passing 100 GB ram usage,

The other thing I noticed is that, the ram usage just continues to increase and even triples itself over a period of 500 frames, however the particle count pretty much (or the file size) doesn't change so much at all.

Please help me out here. I am about to reinstall a linux distro to see if there is sth wrong with my windows

With H13 we moved to a new memory allocator under Windows that is much faster than the default one provided by Microsoft. Unfortunately it can fragment memory over time, which looks a lot like a leak, though technically speaking it is not.

Until we can work out a solution, we're recommending running very large simulations under Linux if at all possible.

ok I see,thanks man. This has been driving me crazy, is this an official announcement, or is this from manual .

johner works for sidefx, so he would know That being said, I was running into the same thing recently, just wasn't sure exactly what was going on. It seems to fill up my available ram, but in terms of simming speed it held steady, and the computer was still quite responsive. Still, guess I'll be setting up dual boot this weekend.

I suppose this issue applies to other tools in Houdini 13 also, like Pyro, RBD multisolvers etc, on windows, please confirm.

Solitude
IIt seems to fill up my available ram, but in terms of simming speed it held steady, and the computer was still quite responsive. Still, guess I'll be setting up dual boot this weekend.

Yes, the allocator (tbbmalloc) allocates memory from the OS in pools and manages them itself for speed. When memory gets fragmented, it can't release these pools back to the OS, although it can still use the free memory within them. The Linux allocator (jemalloc) does a much better job of returning that memory to the OS.

Linux has always been the best OS for large scale simulation, but it's even more true at the moment.

tricecold
I suppose this issue applies to other tools in Houdini 13 also, like Pyro, RBD multisolvers etc, on windows, please confirm.

With Reseeding and the boundary layers in the OceanFX tools, the FLIP solver is often creating and deleting large numbers of particles each timestep, which puts a lof of pressure on the memory allocator. So it seems to show the problem the worst by far. Many other tools that don't create and delete memory as frequently won't be affected much (if at all.)

Seems to me that simulating whitewater also suffers from this. Growing memory in windows and stable in linux

Maybe it should be an option to disable the tbbmalloc and use the old one instead if possible.

update,

The problem exists only in Windows but unlike hogging the machine to a halt, the simulation continoues, but you do start hearing the HD going bananas and eventually Windows returns not enough memory error, maybe crashing Houdini along the way, happened on the 192GB ram machine with 50 mil particles.

I tested the same scene on my Ubuntu, another scene with 25mil pts this time, simulation starts around 6gb ram usage and when the collisions happening, it peaks around 20-25 gb usage but I still find this amount of ram usage abnormal compared to previous versions or what Naiad was able to do with 8GB ram .

So far however at work and at home on linux this doesnt seem to be a big issue, but on windows , flips are a no go in my particular situation.

Not sure about your houdini settings, but i manage to run 80mil particle sim in houdini with only 20GB of ram on linux,
https://vimeo.com/74197060 [vimeo.com]

And 55mil, with small grid scale ( use more memory) with only 18Gb of ram
https://vimeo.com/82942136 [vimeo.com]

Be sure to optimize your scene before run some high res simulations.

Igor

I can confirm that I have been having difficulties dealing with large geometries (scan data) and memory. I do have a machine with little ram though.. only 12. However, it seems that certain operations that were not as bad in 12.5 are getting a bit heavy in memory, and, as mentioned before, it is hard to get that memory back without killing houdini ):

Looking forward to a solution on the tbbmalloc… pretty please

The immediate workaround is to use the VC11 Houdini build on Windows and then copy/replace the tbbmalloc_proxy.dll over from a H12.5 install into H13.0.

This will forcefully disable tbbmalloc_proxy.

thank you for the tip edward! I am going to give it a whirl for the short term

but just to know, is there an ETA on a potential fix for this or even a way to officially switch off tbbmalloc?

I think it be can fixed by upgrading Houdini to use the latest TBB version. The memory usage is still more than Linux but at least its growth is stable. There has been a few issues that have cropped up and we've had to work through them alongside with other tasks. There has been some question as to whether it is a good idea to do this for all platforms, or just Windows.

Unfortunately, there's no way to switch off tbbmalloc_proxy automatically at run time, it must be all or nothing.

Thats good news Edward!

We are are almost forced to switch to linux atm because of this, but we do like windowsland alot better since we already have everything conifgured for windows.

Yes ,I had some others problems about memory and RAM on the Windows.
But when I switch on the Linux ,My simulations worked really good !

I tested some different project about RAM usage on Windows and Linux ,RAM Usage on the Linux is must better than Windows ,Even Rendering time with Mantra on the Linux is faster than Windows !

If you didn't work with Linux and you like Windows Interface ,I suggest you to install Linux Mint ,It's have pretty lovely user interface exactly like windows .

Also I found another way to simulation large scale project (Pyro , Flip Fluid) on the Windows to prevent Ram Usage problems ,I set “Cache Memory (MB)” parameter in the AutoDopNetwork to zero and used “Explicit Cache Files” option to save simulation's caches on the disk ,I don't know But maybe this solution can be useful in your project too :?

Also If you have SSD drive ,Format this drive as EX4 and save your cache files on this drive ,In my personal experience and projects ,SSD with ex4 on the Linux is 2.5 times faster that SSD with ntfs on the Windows ,Also HDD ex4 drive is about 1.5 times fastr than ntfs 8)

Korhon
We are are almost forced to switch to linux atm because of this, but we do like windowsland alot better since we already have everything conifgured for windows.

IMHO, moving to Linux for production is the right thing to do. Linux is still much faster on many fronts than Windows. This is coupled with the fact that the majority of paying customers (and developers) are on Linux.

The new TBB version is not a panacea, the memory usage is still high enough on Windows that I expect that some people will still need to swap. And if you need to swap, then it's game over already.

Note also that if you turn off the use of tbbmalloc_proxy (on Windows only, Linux uses jemalloc), then you're again several times slower. Even in the best case scenarios with Windows using tbbmalloc_proxy vs Linux on jemalloc, Windows is still slower.

grayOlorin
but just to know, is there an ETA on a potential fix for this or even a way to officially switch off tbbmalloc?

On Windows, Houdini 13.0.322 now uses the latest TBB version (4.2 update 2), which contains the improved memory allocator.

EDIT: Corrected version

edward

grayOlorin
but just to know, is there an ETA on a potential fix for this or even a way to officially switch off tbbmalloc?

On Windows, Houdini 13.0.323 now uses the latest TBB version (4.2 update 2), which contains the improved memory allocator.

That's very good ,I should test it later ,Because sometimes we have to use windows instead of Linux ,But not all