Berk Erdag

To prevent them from wobbling make sure the scattered geos do not go inside the ground geo. So maybe after the attribute interpolate node you can put down a Peak SOP and slightly make them move outside. Also you can try to give them a simple collision shape like a box instead of a convex hull.

Maybe try first with some boxes. See if they also wobble, so try to find out if the issue is with the rbd sim/scattered points or the instanced geos.

Another dirty and fast idea would be to activate the scattered rbd geos whenever they need to start moving. It might not work for your case since I don't know your scene but just an idea.

I do not know how you approached coloring these but Adam Swaab explains how to make color work with MaterialX materials here: https://www.youtube.com/watch?v=IyWseMa1yZk&t=5188s [www.youtube.com]

Start watching at 1:29:50

The Rhino Scene:
Karma CPU was taking too long for me to wait for it. It said the ETA was 2 hours after waiting for 10 minutes.
Karma XPU: 06 minutes 45 seconds

Pyro Scene:
Karma CPU again couldn't wait for it to finish, was stating that it will take 35 minutes.
Karma XPU: 39 seconds

i7 9700F 3.0GHz
32GB RAM
RTX 2060