Houdini ships with several built-in nodes for running external programs/scripts in TOPs: ImageMagick, FFmpeg Encode Video, ShotGrid Download, and so on. However, you will often need to make your own or other third party scripts/programs usable in TOPs.

You can run external programs from a TOP network in a few ways:

• You can use a Python Script node and type the script you want to run into the node.

  The Python Script node automatically creates new work items to run a work script once for each incoming work item. You can access the incoming work item’s attributes in the script, and set output attributes on the outgoing work items based on the work the script does.

  • Very useful for experimentation, and for “one-off” scripts that do something specific to one particular network.

  • Most convenient option, only requires writing a work script.

  • Not as flexible. If you need to create work items from scratch, or create multiple work items from upstream work items, you need to create a new processor type instead of using the Python Script node.

  • Can’t have a custom user interface.

  See writing a one-off script below.

• The Generic Generator node generates a new work item to execute a given command line for each incoming work item. You can set this command line to execute a work script. This is effectively the same as the Python Script node but with an external script file instead of having the script embedded in the node.

• You can create a new Python Processor node.

  The Python Processor node lets you write custom Python to generate work items. You can create a digital asset from this node, allowing you to share the new functionality across networks and users.

  • Requires more advanced Python coding using the pdg API, as well as writing a work script.

  • More flexible: you have full control over generating work items. For example, you can create multiple work items based on incoming items. (You cannot generate fewer work item than are incoming. For that you would create a partitioner.)

  • You can give the node a custom interface using spare parameters, to allow the user to control how the node works.

  See making a new processor type below.

Python Script will probably be your first option when building a network. If you find you need the greater flexibility of a custom Python Processor, you can still re-use the work script you created for the Python Script node.

1. If the work done for each work item is small or localized such that it wouldn’t make sense to schedule it on a process queue or render farm (for example, copying files), you can turn on Evaluate in process to do the work in the main process.

   When Evaluate in process is on, you can use the self (the current pdg.Node object) and parent_item (the upstream pdg.WorkItem that spawned the current work item) variables are available in the work script.

2. If Evaluate in process is off, you can set which Python executable to use to execute the script. If you need access to Houdini’s hou module, use “Hython”. If the script is pure Python (this includes the pdgjson and pdgcmd modules), you can set this to “PDG Python”. The regular Python starts faster than Hython, and doesn’t use a Houdini license.

3. Write the script to execute in the Script editor.

   For each incoming work item, the node will create a new work item that runs this script. A pdg.WorkItem object representing the current work item is available in the script as work_item.

   See how to write a work script for more information.

   Keep in mind that (unless Evaluate in process is on) this script might be scheduled to run on a render farm server. Try not to make assumptions about the local machine. See Job API

1. In a TOP network, create a Python Processor node.

2. If you know that this node can’t create new work items until upstream work items are finished (that is, it needs to read data on the upstream items to know how many new items to create), set Generate When to “Each Upstream Item is Cooked”. Otherwise, leave it set to “Automatic”.

3. The Command field on the Processor tab doesn’t do anything. It’s meant to be referenced in the generation script as a spare parameter, to set the command line of new work items. Many people find it makes for easier editing to set up the command line in a parameter and reference it, rather than hard-coding it in the script.

4. Add spare parameters to allow the user to control how the node works.

   This is much more flexible and convenient than hard-coding values into the Python code. You can continue to refine the interface as you work on the code, adding or removing parameters as you work out the code.

   See the help for building a spare parm interface below.

5. Fill in the Generate tab. The callback script on the Generate tab controls what work items the node creates, what attributes the work items have, and the command line run by each work item.

   See the help for the generate tab below. See the generate example for one example of a generation callback.

   The generation callback is called differently depending on whether the node is static or dynamic.

   • If the node is static, the generate callback is called once. The upstream_items list contains the full list of upstream work items, since generation will only happen after the input node(s) have finished generating.

   • If the node is dynamic, the generate callback is called multiple times. The upstream_items list contains at least one cooked work item. If multiple upstream work items finish around the same time PDG will group them together and call the generation logic on that list of work items. In the worst case there will be one generate callback call for each cooked upstream work item, with a input list size of one.

     If the node is dynamic but you want to wait until all upstream items are available, insert a Wait for All node above the Python Processor. Then in the Python Processor node you can access the WorkItem objects inside the partition using:

     upstream_items[0].partitionItems
     

     Alternatively, you can also change the Generate When parameter from Each Upstream Item is Cooked to All Upstream Items are Cooked.

   Note

   You should always use the parent keyword argument to specify which new work item it generated from which upstream item. This is necessary even if the node is dynamic.

6. We recommend that the command line run a Python work script. This is more flexible than running an executable directly, and allows the script to report back results.

   See the help for writing a work script below.

   You can put the work script in a known location in the shared network filesystem, and then use that shared location on the command line.

   or

   You can put the script somewhere on the Houdini machine, and set it as a file dependency on this node in the File tab. This causes the file to be copied into a directory on each server. Then you can use __PDG_SCRIPTDIR__ on the command line to refer to the shared script directory. (If you create different types of work items calling different wrapper scripts, you can set them as multiple file dependencies.)

7. You will usually not need to write a custom script on the Regenerate static tab.

   The default implementation dirties work items when any parameters change. This is usually sufficient, but if you want to use more clever logic (for example, checking the etag of a URL and dirtying the work item if it’s changed), you can implement it on this tab.

8. You only need to write a custom script on the Add internal dependencies tab if some work items generated by this node should depend on other new work items generated by this node.

9. If you want to package up the processor as an asset for re-use across users and networks, you can build an asset from a Python Processor.

new_item = item_holder.addWorkItem(index=0)
new_item.setStringAttrib("url", "http://...")

new_item = item_holder.addWorkItem(index=0)

# Evaluate the URL parameter in the context of this new work item
url = self["url"].evaluateString(new_item)
# Put the evaluated value on the work item as an attribute
new_item.setStringAttrib("url", url)

new_item = item_holder.addWorkItem(index=0)
new_item.setCommand("__PDG_PYTHON__ /nfs/scripts/url_downloader.py")

command_line = self["pdg_command"].evaluateString()
new_item.setCommand(command_line)

import pdg

# Get the number of new items to create for each incoming item
per_item = self["peritem"].evaluateInt()

if per_item > 16:
    # Too many!
    raise pdg.CookError(
        "Can't create more than 16 items per incoming item"
    )

# Go on to create new items...

# Define a helper function that evaluates parameters and copies the values
# into a work item's attributes
def set_attrs(self, work_item):
    # Evaluate the parameter ports in the context of this work item
    url = self["url"].evaluateString(work_item)  # URL to download
    outputfile = self["outputfile"].evaluateString(work_item)

    # Set attributes on the new work item
    work_item.setStringAttrib("url", url)
    work_item.setStringAttrib("outputfile", outputfile)

    # Set the command line that this work item will run
    work_item.setCommand('__PDG_PYTHON__ __PDG_SCRIPTDIR__/url_downloader.py')


if upstream_items:
    # If there are upstream items, generate new items from them
    for upstream_item in upstream_items:
        # Create a new work item based on the upstream item
        new_item = item_holder.addWorkItem(index=upstream_item.index,
                                           parent=upstream_item)
        # Set the item's attributes based on the parameters
        set_attrs(self, new_item)
else:
    # There are no upstream items, so we'll generate a single new work item
    new_item = item_holder.addWorkItem(index=0)
    # Set the item's attributes based on the parameters
    set_attrs(self, new_item)

When a work item is actually scheduled on the local machine or render farm, and is run, it executes a command line. This can be a plain command line running an executable, but for flexibility and convenience we recommend you have the command line run a Python work script. This script lets you access attributes on the work item to influence execution, and set output attributes on the work item based on any files generated by the script.

In a Python Script node, the work script is what you type in the Script field, and the node automatically sets the work items it generates to execute it. In a Python Processor node, you should create this script externally (in the network filesystem shared with the render farm), and set the work items you create to execute it.

import os

# The current work item's name is available in an environment variable
item_name = os.environ["PDG_ITEM_NAME"]
# Look up the work item data by name and build a WorkItem object from it
work_item = WorkItem.fromFile(getWorkItemJsonPath(item_name))

# This is equivalent to using WorkItem.fromJobEnvironment function:
same_item = WorkItem.fromJobEnvironment()

Once you have a Python Processor working the way you want, you can convert it to an asset to make it easier to re-use.

1. Select the Python Processor. In the parameter editor, click Save to digital asset.

2. Fill out the dialog and click Accept.

   

   The new asset type will automatically use any spare parameters on the Python Processor as parameter interface on the Processor tab.

   

   File dependencies are automatically added for the support modules.

   

3. You can continue to edit the asset’s parameter interface. However, note the following:

   • If you edit the interface and Houdini asks you what to do with spare parameters, click Destroy all spare parameters.

   • If you don’t reference the Command parameter to set the command line for work items, you should hide or delete it from the interface so it doesn’t cause confusion.