It uses SideFX Labs "Mocap stream" node as a generic UDP client that receives data from a program I created called VML Streamer.
VML Streamer runs on both windows and linux, uses openCV, and for now it can stream the webcam image + Google's MediaPipe hands/body/face realtime trackers from a simple webcam. (body still needs work!)
The code was conceived in such a way that anything can be plugged in easily, and streamed into Houdini as a python dictionary, with very little effort on the UI and data handling side.
In Houdini, a number of hdas are able to decode received data into geometry, using opencl nodes where possible for faster performance. For realtime trackers, users can snapshot poses into a pose library and create triggers anywhere based on pose matching.
There are also a couple onnx node helpers that facilitate transforming image grids to and from tensors.
If time allows, I'd like to implement an OpenPose binding (to get better full body tracking since MediaPipe is not really suited to 3d world coordinates, as stated by the devs), and also openCV Aruco markers, that can help with rigid object tracking and camera tracking.
Here's VML Streamer (python), it needs a bit of testing on different envs as I was mainly focusing on the alpha features for now: VML Streamer Github [github.com]
A speed up demo (no sound):
playground:
I'll be posting any updates here before the final submission.