Houdini 20.5 Executing tasks with PDG/TOPs

Service Blocks

Services blocks let you define a section of work items that should run using a shared Service process

On this page

Overview

Service blocks combine PDG services with feedback loops. A service block creates one or more loops with a fixed number of iterations and assigns a service worker process to that loop. The service process is locked to the loop and won’t be used by other work items in the graph until the loop has completed. Iterations within the loop cook sequentially in depth-first order, starting from the Begin node and cooking down to the End node.

You can configure compatible nodes within the block to execute on the service process assigned to the loop by changing their cook type to Service, and enabling the Run on Service Block parameter. The Command Send TOP node always runs on the containing service block. Other nodes such as Attribute Create or File Pattern executes normally, but their work items still follows the depth-first cook order imposed by the loop.

PDG includes several built-in service blocks

How to

Create a Service Block

  1. In a TOP network editor, press ⇥ Tab and choose a service block tool, such as Maya Service Block TOP or Houdini Service Block TOP.

The tool puts down a Block Begin node and a Service Block End node

  1. Select the Begin node. In the parameter editor, choose how to specify the number of sessions:

    • The number of sessions determines how many work items each loop in the block should have. The default is to run the number of iterations specified in the Number of sessions parameter. If the Begin node has upstream items, the block creates a loop of size ‹session_count› for each incoming work item.

    • You can turn on Session count from upstream items to create a single loop, with one iterations for each upstream items.

    • Multiple sessions from the same loop are cooked serially: the block cooks the first session from top to bottom before starting the second session.

      • Sessions from different loops can run in parallel.

  2. Connect Command Send nodes between the start and end nodes in their input and output to make them part of the loop.

    • The Command Send node is able to send script code to the service process.

  3. Depending on which type of service block, it’s possible to run other service-based nodes in the block.

    • For example, if the block uses the Houdini service you can include an HDA Processor or ROP Fetch inside the block, and configure it to run on the shared service process.

    • Other nodes that can’t run using the service can be included, and uses the normal scheduling workflow.

Houdini draws a border around the nodes in the block to help you visualize it.

Shared Servers (Deprecated)

In Houdini 19.5 and older, Service Blocks used their own custom server system called shared servers. This feature is now deprecated, but it’s still possible to configure a service block to use shared servers for compatibility with old files.

You can switch a service block to use a shared server by changing it’s Server Type parameter to Shared Server instead of Service.

There are several key differences to consider

  1. Unlike services, shared servers can only execute work items created using the Command Send TOP node

    • Other types of nodes can be included in the service block, but they run their work items using the normal scheduling work flow

    • Blocks that use services are able to run any type of work item that’s compatible with the service assigned to the block. For example, the Houdini Service Block can execute work items from a Python Script, HDA Processor, or ROP Fetch in addition to the Command Send.

  2. Shared servers are not suitable for use on the farm

    • The long-running server processes can’t be managed using a farm interface, so it can be hard to debug or terminate those tasks when problems occur

    • Shared server processes also don’t benefit from scheduler settings

Notes

  • The Begin node generates work items that lock a service worker process to the block. This ensures no other work items in the graph can interfere with the operations performed by the block.

  • The service worker is unlocked once the last End node work item for the block evaluates. If the graph is canceled or stops cooking before that point, all locked service processes unlocked.

  • You can set the Reset Service parameter on the Begin node to ensure that the state of the service process is reset at the beginning of each loop iteration.

  • The state of each service process can be inspected using the PDG Services list.

  • You should color the start and end nodes of a block the same to make their relationship clear. The default nodes put down by the built-in service block tools have different colors. You can change the node colors. This is useful to distinguish nested loops.

    The border around the block takes on the color of the end node.

Examples

The following directories contain example HIP files demonstrating how to use Houdini and Maya service blocks.

  • $HFS/houdini/help/files/pdg_examples/top_houdinipipeline

  • $HFS/houdini/help/files/pdg_examples/top_mayapipeline

Executing tasks with PDG/TOPs

Basics

Beginner Tutorials

Next steps

  • Running external programs

    How to wrap external functionality in a TOP node.

  • File tags

    Work items track the results created by their work. Each result is tagged with a type.

  • PDG Path Map

    The PDG Path Map manages the mapping of paths between file systems.

  • Feedback loops

    You can use for-each blocks to process looping, sequential chains of operations on work items.

  • Service Blocks

    Services blocks let you define a section of work items that should run using a shared Service process

  • PDG Services

    PDG services manages pools of persistent Houdini sessions that can be used to reduce work item cooking time.

  • Integrating PDG with render farm schedulers

    How to use different schedulers to schedule and execute work.

  • Visualizing work item performance

    How to visualize the relative cook times (or file output sizes) of work items in the network.

  • Event handling

    You can register a Python function to handle events from a PDG node or graph

  • Tips and tricks

    Useful general information and best practices for working with TOPs.

  • Troubleshooting PDG scheduler issues on the farm

    Useful information to help you troubleshoot scheduling PDG work items on the farm.

  • PilotPDG

    Standalone application or limited license for working with PDG-specific workflows.

Reference

  • All TOPs nodes

    TOP nodes define a workflow where data is fed into the network, turned into work items and manipulated by different nodes. Many nodes represent external processes that can be run on the local machine or a server farm.

  • Processor Node Callbacks

    Processor nodes generate work items that can be executed by a scheduler

  • Partitioner Node Callbacks

    Partitioner nodes group multiple upstream work items into single partitions.

  • Scheduler Node Callbacks

    Scheduler nodes execute work items

  • Custom File Tags and Handlers

    PDG uses file tags to determine the type of an output file.

  • Python API

    The classes and functions in the Python pdg package for working with dependency graphs.

  • Job API

    Python API used by job scripts.

  • Utility API

    The classes and functions in the Python pdgutils package are intended for use both in PDG nodes and scripts as well as out-of-process job scripts.