Okay, I'm starting to lose my mind. I've been working with distributed FLIP sims for the past 2 weeks or so... I'm trying to create a small scale breach simulation— think head coming out of a tub of water. Right now I'm doing some RND with a basic flat tank setup and a colliding sphere with an animated mountain deformer, super simple.

On my machine the sim runs as expected, but when the simulation is sent to the farm it starts having volume loss issues.

I really don't understand why. I'm thinking it has something to do with loss of particle attributes/velocities while being transferred between computers, but I don't know how to fix that if that is the cause.

The screenshots below show frame 14 of each iteration, distributed and non-distributed sim. In the first screenshot you can see that the sim sent to the farm begins to sink early and will most likely continue as it runs. The sim that cooked on a single machine was stable, as seen in the second screenshot.




Also, if Boundary Layers is enabled, the sim gets a little more whacky, see screenshots below. I'm not even sure how to explain what's happening here, but it's still sinking...




Note: The first two screenshots are a simulation with the same values. The boundary layer screenshots are from a previous sim setup with slightly different values.


Here's the parameter values used in the first two screenshots.

-Flip Object Parameters-
Particle Separation: 0.01 (will need to lower in the future for final sim, around 0.03)
Particle Radius Scale: 1.2
Grid Scale: 1.5
Closed Boundaries: +X, -X, -Y, +Z, -Z
Input Type: Narrow Band

-Flip Solver Parameters-
Substeps: Locked 2
Reseeding: Enabled at default values
Separation: Enabled at default values
Volume Limits: all at default when using flat tank shelf tool
Narrow Band: Enabled with bandwidth set to 3

Any thoughts or ideas as to why this happens? Appreciate any help!