| On this page |
The TOP context, also referred to as PDG, allows you to create a recipe for all stages of machine learning. Using a TOP network, the stages include the generation of a data set, preprocessing, writing raw data sets, training using the data set, varying hyperparameters, and running a trained model over a set of inputs.
A TOP network allows you to lay out all these stages in a step-by-step recipe. By cooking the last TOP node in this recipe, all the desired steps are taken automatically without having to manually go to various nodes and save their output to disk.
Useful TOP nodes ¶
Geometry Import can perform a type of computation repeatedly, based on the number of primitives that exists in a geometry. Each primitive could correspond to a single data point of a data set to be generated or preprocessed. This provides a clean way to control the size of the data set near the source of the data in SOPs, without having to create custom expressions in TOPs. Geometry Import generates a set of work items, one for each primitive. If the geometry referenced by Geometry Import is created in an earlier stage of in TOP network, then make sure to set the parameter Generate When to All Upstream Items are Cooked.
Use 
Attribute Create to assign each work item a data-point index, which can be referenced from a SOP network to cook each data point.
A 
ROP Fetch helps ensure that each data point is written out to disk. Afterwards, a separate ROP Fetch would ensure the entire merged data set is written to a file.
As an alternative to generating each data point of a data set as a single work item in PDG, the entire data set may be generated using a for-loop in SOP. In this case, a ROP Fetch helps write the entire data set in the form of a single geometry to a file. You can also use a batch approach where each work-item in TOPs corresponds to a batch of data points. Each batch can be cooked using a for-loop in SOPs.
TOP nodes for training ¶
After the data has been written out, training may proceed using a combination of 
Python Virtual Environment and Python Script. This helps run your own python training script that uses pytorch, exporting your model to ONNX.
For doing regression ML, use the more specialized node 
ML Regression Train. Even if your ML application falls outside the example-based framework, this training node and the script it references can be a good starting point for writing your own ML training script that works with Houdini.
You can use 
ROP Geometry Raw Output to write your data set to a file that you can reference from your training script. The example-based ML toolset has its own output node 
ML Example Output.
The ML training can be re-run for various hyperparameters such as the number of hidden layers and a weight decay parameter. There may be others depending on the script you write. To automatically repeat the training stage of the TOP recipe for various combinations of these hyperparameters, use a 
Wedge above your training script.
ML on a farm using HQueue ¶
You can do your entire data generation and training on a farm using the 
HQueue Scheduler. It’s recommended change the following settings:
-
In Scheduler/HFS, set the Python parameter to From HFS. This ensures the python version that ships with Houdini runs the training script.