I try to test any Apprentice release on OS X 10.5.2-5 and always had problems with running and interface of Houdini. At last I check Console and got errors

Library not loaded: /usr/X11R6/lib/libSM.6.dylib

The installation of OS X have only X11 folder with same files. I copied X11 folder, rename it to X11R6, copy back to /usr/ and I HAPPY. Houdini running fine withought any interface problems . . .

Instead of copying it, why don't you make a symbolic link to the X11 directory?