I keep having the same issue after putting down a remesh node before piping my cloth into a vellum cloth constraint, and vellum solver.
There's definitely no collision with the cloth and mesh from the initial frame, it seems to get to about frame three and then looks like it's being yanked off the body.
On the solver I have substeps to 5, and constraint interactions 300.
It does correct itself a couple of frames later, but the dress just goes right the body, even after upping the collision iterations to 150.
I notice when turning off the remesh, I still get the yanking effect, but it seems to collide better with the body geo.
Im not sure if its the collision geo a fault, or the remesh doing something odd.
Does anyone have any idea?