Houdini 20.5 Executing tasks with PDG/TOPs

Commonly used TOP nodes

A quick guide to the most commonly used TOP nodes.

On this page

This is not a list of all TOP nodes. This is a list of the most commonly useful nodes. These nodes should give you an idea of how to get started with common workflows in TOPs. See

Kicking off

If these nodes have no input, they create a number of static work items specified by their parameters. If their input is connected, they create the specified number of new work items for each incoming work item.

File Pattern

Creates a work item for each file that matches a pattern (for example, *.exr). You can then add processor nodes to do work on each file.

Download File

Generates work items to download one or more URLs to disk.

Wedge

Creates a work item for each variation of one or more attributes you specify. See wedging attributes for more information.

ROP Mantra Render

Creates a work item to render each specified frame.

ROP Geometry

Creates a work item to generate geometry for each specified frame.

FFmpeg Extract Images

Creates a work item for each frame image extracted from a video file.

CSV Input,
JSON Input,
XML Input

Create new work items for each “row” in an input file.

SQL Input

Creates new work items for each row returned by a database query.

Doing work

HDA Processor

Cooks a digital asset and writes the asset’s output to a file.

Invoke

Cooks another node in the same HIP file as this node.

Python Script

Runs a Python script for each incoming work item. This is the node you will usually use for making TOPs do custom work. You can set new output attributes on the work item from this script based on the work done.

This is easier to use than creating a custom processor code with Python Processor, but doesn’t have the full flexibility (for example, this node can’t create static work items from scratch, or generate multiple new items for each incoming item).

Generic Processor

Runs a command line executable for each incoming work item. This does not allow you to set attributes based on the result of the executable, so it is not as flexible as the Python Script node. However, for simple jobs where you can infer the results (for example, you know the output files have a certain pattern), this may be useful.

Send Command

Command servers let you start up a persistent shared “server” process (such as an instance of Houdini or Maya), send it commands, and then shut it down when all work is finished.

See wrapping external programs and command servers for more information.

Waiting for results

See waiting for results.

Image/video outputs

FFmpeg Encode Video

Compiles a set of frame images into a video file.

ImageMagick

Manipulates an image file using the ImageMagick library.

Overlay Text

Writes text over an image. This is very useful for overlaying metadata (for example, frame number, wedge attributes, filename) over non-final images.

File management nodes

Make Directory

Creates a work item to create a new directory on disk. This can be useful for organizing output files and creating temporary storage areas.

File Rename

Renames or moves a file. Turn on Async to perform the rename(s) in work items at run time.

File Copy

Copies a file from one name/location to another. Turn on Async to perform the copies in work items at run time.

Remove File

Deletes a file. Turn on Async to perform the copies in work items at run time.

Tip

You may want to avoid destructive file operations such as move/rename and delete, to avoid making the dependency system do extra work because of missing files.

See file management tips.

Utilities

Attribute Create

Adds one or more attributes to the incoming work items. You can use expressions to compute the contents of the new attribute(s).

Split

Splits the incoming work items into separate outputs based on some criteria. This lets you process different types of work items using different node chains.

Work Item Expand

“Recreates” the work items that were merged into a given partition. This can be useful to wait for a group of items (by partitioning them) but then continue on to generate separate work from the items.

Finishing

Wait for All

This is a partitioner that puts all incoming work items in a single partition, so it doesn’t proceed until all upstream work is done. This is usually needed at the end of the network (before clean up and/or notification nodes), and sometimes at the end of logically distinct pipeline steps, if the next step requires all work to be finished.

Send Email

Creates a work item that sends an email. This can be useful after a Wait for All at the very end of a large network, to notify a user/group that the work is finished.

Python Script

For any other types of notifications besides email, you can use a Python Script node to use a notification API. For example, you could post a Slack message at the end of the network:

import os
from slackclient import SlackClient


slack_token = os.environ["SLACK_API_TOKEN"]
client = SlackClient(slack_token)

client.api_call(
    "chat.postMessage",
    channel="C0XXXXXX",
    text=":tada: The big long job finished!"
)

Executing tasks with PDG/TOPs

Basics

Beginner Tutorials

Next steps

  • Running external programs

    How to wrap external functionality in a TOP node.

  • File tags

    Work items track the results created by their work. Each result is tagged with a type.

  • PDG Path Map

    The PDG Path Map manages the mapping of paths between file systems.

  • Feedback loops

    You can use for-each blocks to process looping, sequential chains of operations on work items.

  • Service Blocks

    Services blocks let you define a section of work items that should run using a shared Service process

  • PDG Services

    PDG services manages pools of persistent Houdini sessions that can be used to reduce work item cooking time.

  • Integrating PDG with render farm schedulers

    How to use different schedulers to schedule and execute work.

  • Visualizing work item performance

    How to visualize the relative cook times (or file output sizes) of work items in the network.

  • Event handling

    You can register a Python function to handle events from a PDG node or graph

  • Tips and tricks

    Useful general information and best practices for working with TOPs.

  • Troubleshooting PDG scheduler issues on the farm

    Useful information to help you troubleshoot scheduling PDG work items on the farm.

  • PilotPDG

    Standalone application or limited license for working with PDG-specific workflows.

Reference

  • All TOPs nodes

    TOP nodes define a workflow where data is fed into the network, turned into work items and manipulated by different nodes. Many nodes represent external processes that can be run on the local machine or a server farm.

  • Processor Node Callbacks

    Processor nodes generate work items that can be executed by a scheduler

  • Partitioner Node Callbacks

    Partitioner nodes group multiple upstream work items into single partitions.

  • Scheduler Node Callbacks

    Scheduler nodes execute work items

  • Custom File Tags and Handlers

    PDG uses file tags to determine the type of an output file.

  • Python API

    The classes and functions in the Python pdg package for working with dependency graphs.

  • Job API

    Python API used by job scripts.

  • Utility API

    The classes and functions in the Python pdgutils package are intended for use both in PDG nodes and scripts as well as out-of-process job scripts.