Houdini 20.5 Executing tasks with PDG/TOPs

Cooking/Executing the TOP network

How to cause the TOP network to actually do the work and produce the results specified by the network.

On this page

See also visualizing TOP network performance.

Overview

The TOP network attempts to minimize extra work by only re-doing work that changed (or might have changed) since the previous cook. This means a cook might do no work at all if nothing has changed.

TOPs uses a concept of marking work items “dirty” if they are (or might be) out of date and need to recook. The following rules how the TOP network decides what to recook.

  • If a node generates different work items than it did in the previous cook, it dirties all subsequent nodes.

  • Nodes that deal with external files (files not created within the TOPs network), such as File Pattern and Geometry Import, scan the files when they start and mark existing work items dirty if the corresponding file has changed.

  • Nodes that reference external Houdini nodes (such as ROP Geometry Output) check the references data when they start and mark existing work items dirty if the data has changed.

  • TOPs does not currently check input and output result files. For example, deleting or replacing an intermediate file will not automatically dirty work items that use that file as input.

  • Any work item whose attributes have changed since the last cook are considered dirty.

  • Changing parameters on a node dirties the node (since most work items are generated based on both incoming work items and parameter values).

    When parameters on a TOP node change (meaning the node must recook), the status icon next to the node becomes dim:

  • You can manually mark a work item, node, or the entire network dirty to force it to recook.

Cooking

To...Do this

Cook the parts of the network that need cooking

Choose Tasks ▸ Cook Output Node.

Cook a single node (only if it needs cooking)

  • Select the node and choose Tasks ▸ Cook Selected Node

    or

  • Right-click the node and choose Cook this node.

Force Houdini to recook the entire network

Choose Tasks ▸ Dirty and Cook Output Node.

Force a single node to cook

  • Select the node and choose Tasks ▸ Dirty and Cook Selected Node.

    or

  • Right-click the node and choose Dirty and cook this node.

Mark the entire network dirty but don’t recook yet

Choose Tasks ▸ Dirty All.

Generate all static work items but do not cook anything

Choose __Tasks > Generate Static Work Items___.

Generate work items for a particular node, dynamic or static

Right-click the node and choose Generate Node

Recook a single work item

Right-click the work item dot and choose Cook task.

Mark a single work item dirty but don’t recook yet

Right-click the work item dot and choose Dirty task.

Limit the number of resources used by the local scheduler

See resource limits

Cook the entire graph on the farm, disconnected from the current session

Each farm scheduler node has UI for this, which is called Submit Graph As Job. For example see HQueue.

Output flag and output nodes

  • Inside a TopNet, the output flag controls which node the network “works toward”. Only nodes upstream from the node with the output flag cook when you ask the network to cook.

  • Inside a TOP subnetwork, if the subnet has an Output node, it represents the output of the subnet, ignoring any output flag.

  • If you create multiple Output nodes inside a TOP subnet, the subnet will have multiple output connectors, similar to the Split TOP.

Cooking From Command Line

Often a pipeline will need to execute a top net non-interactively from a script of some kind.

To...Do this

Cook a top network from hscript

Running hbatch gives you an HScript command-line environment. Use the topcook hscript command to execute the network:

% hbatch myscene.hip
hbatch Version ... (Compiled on ...)
help topcook
topcook /tasks/topnet1

Cook a top network using python

Executing the topcook.py utility script with the hython command line program:

% hython $HHP/pdgjob/topcook.py --hip myscene.hip --toppath /tasks/topnet1
Running Houdini ... with PID ...
Loading .hip file .../myscene.hip
Given Node 'topnet1', Cooking Node '..'
Local Scheduler: Max Slots=..., Working Dir=...
Finished Cook
Work Item States:
...

Cooking From HOM

To...Do this

Cook a TOP network in a HOM script

To tell Houdini to cook a TOP network in HOM, find the parent network, find the TOP node inside with the output/display flag, and call the hou.TopNode.cookWorkItems method on it.

hou.hipFile.load(graph_file)
net = hou.node('/tasks/topnet1')
to_cook = net.displayNode()

# Blocking cook of PDG graph
to_cook.cookWorkItems(block=True)

Cook a TOP network while watching for a file change

Similar to the above, but done while monitoring for a file change in a loop. Instead of a file we could be checking for a Perforce change, or polling a message queue. Note that this could be improved by using a robust file watching library instead of simply checking mtime.

import hou
import os
import time

graph_file = '/tmp/doit.hip'
watch_file = '/tmp/watch.txt'

hou.hipFile.load(graph_file)
net = hou.node('/tasks/topnet1')
to_cook = net.displayNode()

# Make sure the network is initialized (PDG network created from TOP network)
to_cook.cookWorkItems(block=True, tops_only=True)

print('Watching for changes - Ctrl-C to exit...')

mtime = os.path.getmtime(watch_file)
try:
    while True:
        mtimep = os.path.getmtime(watch_file)
        if mtimep != mtime:
            mtime = mtimep
            # Blocking cook (execute) of PDG graph
            to_cook.cookWorkItems(block=True)
            # Reset the graph
            to_cook.dirtyAllTasks(False)
        time.sleep(1)
except KeyboardInterrupt:
    print('Finished')

Adding TOP network cook controls to any node

You can control the cook of your TOP networks from any node in your node network by adding TOP network cook controls to those nodes.

This is very useful when you want to control the cook of TOP networks that live at different levels inside your node network from the top-level of your network.

For example:

  • If you have an HDA that contains a TOP network and you want to control its cooks from the HDA’s interface, you can add the TOP network cook controls to the HDA. This would let you control the TOP network’s cook without having to dive into the HDA.

  • If you have a complex node network with multiple TOP networks at different levels, you can add Null nodes representing those TOP networks to the highest level of your network and then add the TOP network cook controls to those nodes. This would give you quick access to the TOP network cook controls as well as cook progress badges and counts on the Null nodes in the viewport.

How-to

  1. Select the node you want to add the TOP network cook controls to.

  2. Open the Edit Parameter Interface window.

    Do one of the following:

    • In the viewpoint, -click the node and then select Parameters and Channels ▸ Edit Parameter Interface from the context-sensitive menu that appears.

    • In the Parameter Editor, select gear menu icon ▸ Edit Parameter Interface.

  3. In the Edit Parameter Interface window, select the Create Parameters column > Node Properties tab.

  4. In the Node Properties tab, select TOP Network Controls ▸ TOP Cook Controls ▸ TOP Network Cook Controls.

  5. Do one of the following:

    • Click the right arrow to copy the parameters over to the Existing Parameters column.

    • -drag the selected parameters over to the Existing Parameters column.

  6. Click Accept.

    The TOP Network Cook Controls now appear in the selected node’s parameters. See the Parameters section below.

  7. In the TOP Network Cook Controls parameters, specify the TOP network you want to link the cook controls to in the Top Network parameter field.

You can now use the TOP Network Cook Controls buttons to control the cook of all the work items in all of the TOP nodes inside the TOP network you specified.

A progress badge and count for the specified TOP network’s cook as well as dots for all the TOP network’s work items also now appear on your node in the viewport.

Parameters

TOP Network Cook Controls

Top Network

Specifies the path to the TOP network with which the cook controls are associated.

For more information on path syntax, see Node and parameter paths.

Generate Static Work Items

Generates static work items in all nodes in the TOP network. None of the work items will be cooked, and dynamic nodes will do nothing.

Cook Output Node

Cooks the output node in the TOP network and any out-of-date nodes connected to it. If you want to recook all the nodes in the TOP network, click Dirty all first and then click this button.

Cancel Cook

If any nodes in the TOP network are currently cooking, cancels their cook.

Dirty All

Dirties all the work items in all of the TOP nodes inside the network and marks every node inside the network as needing to recook.

Delete All File Outputs From Disk

Deletes all previous cook files from disk and then dirties all the work items in all of the TOP nodes inside the network.

Executing tasks with PDG/TOPs

Basics

Beginner Tutorials

Next steps

  • Running external programs

    How to wrap external functionality in a TOP node.

  • File tags

    Work items track the results created by their work. Each result is tagged with a type.

  • PDG Path Map

    The PDG Path Map manages the mapping of paths between file systems.

  • Feedback loops

    You can use for-each blocks to process looping, sequential chains of operations on work items.

  • Service Blocks

    Services blocks let you define a section of work items that should run using a shared Service process

  • PDG Services

    PDG services manages pools of persistent Houdini sessions that can be used to reduce work item cooking time.

  • Integrating PDG with render farm schedulers

    How to use different schedulers to schedule and execute work.

  • Visualizing work item performance

    How to visualize the relative cook times (or file output sizes) of work items in the network.

  • Event handling

    You can register a Python function to handle events from a PDG node or graph

  • Tips and tricks

    Useful general information and best practices for working with TOPs.

  • Troubleshooting PDG scheduler issues on the farm

    Useful information to help you troubleshoot scheduling PDG work items on the farm.

  • PilotPDG

    Standalone application or limited license for working with PDG-specific workflows.

Reference

  • All TOPs nodes

    TOP nodes define a workflow where data is fed into the network, turned into work items and manipulated by different nodes. Many nodes represent external processes that can be run on the local machine or a server farm.

  • Processor Node Callbacks

    Processor nodes generate work items that can be executed by a scheduler

  • Partitioner Node Callbacks

    Partitioner nodes group multiple upstream work items into single partitions.

  • Scheduler Node Callbacks

    Scheduler nodes execute work items

  • Custom File Tags and Handlers

    PDG uses file tags to determine the type of an output file.

  • Python API

    The classes and functions in the Python pdg package for working with dependency graphs.

  • Job API

    Python API used by job scripts.

  • Utility API

    The classes and functions in the Python pdgutils package are intended for use both in PDG nodes and scripts as well as out-of-process job scripts.