ML SKETCHING

Introduction

This is a tutorial on how to teach Houdini to understand your drawings using the example-based machine learning (ML) nodes that were released with Houdini 20.5. This scene demonstrates how a variety of ML setups can now be built entirely from within Houdini just by putting down ML nodes and setting parameters, without the need to write custom training scripts. The tutorial is in addition to the corresponding project files released in the content library. You can download them here!

This is one of the projects I presented at the Houdini HIVE Horizon event hosted by SideFX in Toronto. You can watch the full presentation here:

PCA Shenanigans and How to ML | Jakob Ringler | Houdini Horizon

This text tutorial dives into more specific setup details.

General Idea

The high level goal is controlling geometry generation using a canvas style drawing UI. The main problem here is that it is super hard to predict proper topology or even meshes at all. Neural networks like grid-like data topology as inputs/outputs.

Shapes like arrays, images or voxel volumes work well. Any one of these with a fixed size can be used as input or output for training relatively easily. Houdini's procedural workflows come in very handy here: We can create an HDA or other type of generator setup that builds our desired object. What we want the ML model to learn, is to map from a silhouette drawing to the parameters that drive the generator, to produce the correct object procedurally.

float image grid --> float array

Doing so, we avoid all sorts of pitfalls and pain trying to predict geometry. The architecture comes with different pros and cons, which I will explain later. However, most of them are rather desirable, such as having clear constraints on what the model can output and not being too limited by the small resolution of the canvas:

- no matter what input is given, the generator will always generate a somewhat sensible result 
- the generator can create model details, that don't need to be reflected in the input drawing

Generator

Another fun thing that is underrated is pack and unpack folders, which were originally introduced as part of the APEX/Character workflow. They are really useful to store different geometry streams in one, and splitting  them apart again later based on their folder name. It's really convenient and basically lets you create a on-the-fly directory structure on your geometry.

Data Generation

Compared to the ML Groom Deformer demo, where the data generation is automated using TOPs, this time everything runs in a simple for each loop. This makes more sense because each example only takes a second or less to generate, and spinning up entire Houdini instances in TOPs would create a bottle neck. Depending on the generation time and amount of examples needed, it might make sense to implement a mix of both approaches: 

Generate a bunch of examples in one workitem, but have multiple instances run at the same time.

Because the generator is just the compiled block, we need to randomize the input dictionary. This uses a super simple detail expression that randomizes each value based on the current iteration. Make sure to give a different offset/seed to each, otherwise we get the same value in each parameter.

The invoke compiled block SOP then runs the generator and spits out our merged geometries, which we can access using the unpack folder SOP as mentioned above. The rest is just some data wrangling and rasterizing the outline on the image plane before storing it together with the parameter value list as a ml example.

We use that loop to generate 10,000 examples and store those on disk.

PCA Subspace Compression

In the ML Groom Deformer project, we used PCA to compress all the displacement samples into a smaller subspace, which allowed us to only store the weights for each example and not the whole geometry. We can do the same with our input images this time. Storing a 64x64 px image would mean feeding the neural network an array of 4096 float values. We can compress the image space of the 10,000 examples into much fewer components. 

What helps a lot is converting the black and white mask into an SDF using the new COPs SDF tools. That way, we don't have to deal with only 0 or 1 values and have a smooth gradual falloff around the bottle outline.

A really cool side effect of PCA compressing our images, is that we can really visualize what each component looks like:

Because the components are ordered based on importance we can really see how much actual information each component stores before they gradually become less and less influential on the  reconstruction. In the top left, we can still make out the bottle shape, while in the bottom right it almost looks like noise. 

Almost!

Having access to visually nice and somewhat sensible components also allows us to visualize how they should be weighted to reconstruct a given input.

Say a user draws the following shape (which gets converted into an SDF under the hood).

Feeding this drawing together with the precomputed subspace components into the PCA SOP gives us the corresponding weights. If we apply those weights to their corresponding components, we can really see which components make up the final reconstruction.

Adding them all together gives us the reconstructed SDF... which looks correct. Yey!

To recap: this working well with so few weights allows us to map from 64 input values (PCA weights) instead of 4096 float values to the 8 generator parameters.

Data Overview

This is all the data we have now:

Our training examples are just two value lists:

- input: weights that describe the bottle outline image in the PCA subspace

- target: parameter values that describe the bottle shape

We store those on disk as a data_set.raw file using the ML Examples Output SOP and load it in the ML Regression Train TOP.

Training

Once the data set is prepared and stored on disk as .raw file, the rest is pretty straight forward. 

To start training something, we only need to make sure to specify the correct directory and name of our dataset on the files tab. By default, this should be correct though. Be careful : only specify the filename without the extension. It won't run if we add the ".raw". (last tested in H20.5.370)

Then we can just cook the TOP node and hope our idea works out and we have enough training data.

Inference

Having all those parameters available on the node in TOPs opens the door to do some wedging! This is common practice and is usually called hyperparameter tuning. We could run multiple experiments to find the right combination of parameters that give us the best performing model. The parameters I mentioned above are a good place to start wedging. For the groom deformation example here, I only wedged the amount of layers and the size of each layer.

Performance

The whole inference step is pretty fast and runs in realtime. The bottle neck here is the geometry generation step. When we hook the system up to a very involved HDA that e.g. builds ships instead of simple bottles, we will run into performance problems. Might be worth having 2 different HDAs in that case:

COPs

Combine that with some procedural COPs texture setup and we get something that looks much more involved than actually is!

Source: Texture Setup was created by Dixi Wen

Constraints

This can be a downside or a benefit. We will always receive a somewhat sensible output no matter what the input is. The output is constrained by what the generator can create.

What if? Predicting Full SDFs

Remember how SDFs compress really well with PCA? What if instead of learning to predict the parameters, we try to predict a full 3d SDF? 

After converting it and cleaning it up a bit using vdb meshing techniques and then projecting clean uv'ed geometry onto it, we get our bottle.

Having 2 Generators

The main downside of this approach is that we need 2 generators. One to generate our training data and one that turns SDFs into usable geometry with good topology and uvs. That is also the most constraining part. Not every object can be converted so easily to good topology and a bottle is one of the easiest shapes.

The left one is the original generator, which is used to create the training data. Instead of storing the parameter values as targets, it stores the fog SDF that is derived from the bottle geometry.

Inference

For the inputs, we do the same (mirror, to SDF, PCA projection) thing we did before. We then run the outputs through PCA to reconstruct the 3D SDF based on the predicted weights. We feed this SDF into our second generator to turn it into clean geometry.

Performance

Accuracy and flexibility are improved, however generation speed suffers.

Because the second generator is the bottleneck, the speed is much slower compared to the earlier version. However, its outputs are much more flexible and we can draw "in between" shapes that didn't even exist in the training data. There are still problems with sharp turns and overhangs that could be improved upon with other techniques or better training data.

Credits

Thanks to the amazing team at SideFX for all the help with this project!