Houdini 20.5 Executing tasks with PDG/TOPs

PDG Services

PDG services manages pools of persistent Houdini sessions that can be used to reduce work item cooking time.

On this page

Overview

A PDG service is a long-running process that acts as a server for cooking work items. When a node is configured to use a PDG service, a cooking work item submits the job to a pool of services. The first service that is available retrieves the work item and perform its work.

PDG services avoids the overhead associated with starting a new process. Services can also perform optimizations specific to their jobs. For example, the HDA Processor service can create the necessary nodes exactly once, reusing the same nodes for each job and avoiding the time associated with creating the network. As a result, PDG services can vastly increase the speed of your jobs.

Another example is to create light weight work to generate a large assortment of different props from a base asset.

This allows you to generate a huge number of variations without restarting the process for every cook.

You can use Service Blocks to run multiple work items in a loop using the same service worker process.

Supported nodes

PDG Services panel

To open the PDG Services panel:

  • In the TOPs Tasks menu, select PDG Services.

  • Create a new Pane Tab and then from the panes menu select New Pane Tab Type ▸ TOPs ▸ PDG Services.

You can now see the PDG Services Panel.

PDG Services list

Lists all available, registered services and if they're currently running. You can select, add, or remove registered PDG services from this list.

Menu Item

Function

Create Service:

Opens the Add PDG Service window.

Delete Service:

This menu removes a registered service. See Delete a registered PDG service.

Start:

Starts the selected PDG service.

Stop:

Stops the selected PDG service.

Automatic:

When on, the selected service is automatically started by the first PDG graph cook where the service is used (if the service is not already running).

Status:

Shows whether the selected service is running or stopped.

Owner:

Shows the owner of the service. Currently, a service can only be owned by the Houdini session.

Type:

Shows the name of the scheduler type that the service is associated with.

MQ Server Authenticated:

Shows whether the MQ server is authenticated by all its clients, or “No Polling Client” if the Houdini session is not actively using the server.

MQ Clients Authenticated:

Shows whether all the MQ clients are authenticated by the server, or “No Polling Client” if the Houdini session is not actively using the server.

Name:

Specifies the name of the selected PDG service.

Pool Size:

Sets the number of instances that will be created for the service pool. More instances will result in faster cook times because more items will be able to be cooked in parallel.

Connection Timeout (S)

The time in seconds that a service process will wait when communicating with the main Houdini process. If the timeout is exceeded, the service proces will terminate.

MQ Server Log Level:

Shows the logging level of the MQ server associated with the service.

MQ Server Log Directory:

Shows the directory where the logging file for the MQ server will be stored.

Client Log Type

Determines where service processes should write their log data. By default, services will redirect their log to a file on disk as specified using the Client Log Directory, however they can also be configured to log directly to standard output so that the logs are picked up by a farm system.

Client Log Directory:

Specifies the folder where service pool clients will write their log files.

Environment Variables:

Specifies the environment variables that should be set in this service’s environment. Click Add Environment Variable to add an environment variable, and right click on an environment variable and click delete to delete one.

Service Client list

The Service Client list allows you to see the status of the worker processes and status pool. This helps to debug any processes that may be stuck and need to reset, restart, or stop.

You can use the service client list to decide if you have sufficient clients. For example, if you only see one or two of your pool status lit, you may be able to reduce resources by reducing the pool size.

Menu Item

Function

Reset Client:

Sends a “reset” command to the service, which causes the service process to clear any loaded scene data

Restart Client

Stops and recreates the service process.

Stop Client

Stops the process.

Create PDG Service window

Menu Item

Function

Type:

The service type associated with the registered service.

Owner:

Controls the owner of the service. Currently, the service can only be owned by the Houdini session. This means that the lifetime of the service is associated with the current Houdini session. If the service is still running when Houdini is exited, the service will be shut down automatically.

Name:

A unique name for the service. This is the name that will be used when referring to the service in node parameters, for example.

Automatically Start Service:

When on, the service is automatically started (if it’s not already running) during the first PDG cook where the service is used.

Pool Size:

The number of clients that are created by the service.

Port:

The defined port configuration.

Automatic: (port option)

The port that the service will use is determined automatically.

Custom: (port option)

The port that the service will use is a custom port you need to specify.

MQ Server Log Level:

Controls what type of information is written to the MQ server’s log file.

None: (MQ Server Log Level option)

No information will be logged.

Errors: (MQ Server Log Level option)

Only errors will be logged.

Warnings: (MQ Server Log Level option)

Warnings and errors will be logged.

All: (MQ Server Log Level option)

General status messages, warnings, and errors will be logged.

MQ Server Log Directory:

Specifies the logging directory in which the MQ server will create its log file.

Client Log Type:

Determines where service processes should write their log data, either to a file on disk or to the standard output of the service process.

Client Log Directory:

Specifies the logging directory in which Clients within the service pool will create their log files.

Add:

Registers the service.

Cancel:

Cancels the service registration operation.

Generic Services

The script to implement a service typically requires access to Houdini’s Python interpreter to properly communicate with the message queue that distributes tasks to the service clients. To support running services that use a different Python interpreter, PDG includes a special Generic Service which can create service clients using any version of Python. The Generic Service is configured through the Extra Arguments field in the service panel. It has two required commandline arguments:

Argument Name

Description

–executable:

The path to the custom Python executable that the service should use, such as /usr/bin/python2, /usr/bin/pypy, or C:/Program Files/Autodesk/Maya2024/bin/mayapy.exe

–script:

The name of the client script module that each service client should use. This module must exist on the Python search path for PDG jobs.

Houdini ships with custom client scripts for running Maya and Nuke services. Also, a generic client script that can be a reference. The client scripts can be found in the $HHP/pdgjob/ directory, for example $HHP/pdgjob/mayaclient.py and $HHP/pdgjob/genericclient.py.

The Maya Service Block TOP and Nuke Service Block TOP have helper buttons that automatically create instances of the Generic Service that are pre-configured to use a locally installed copy of Maya or Nuke.

PDG Services nodes

In addition to creating and managing services with the PDG Services panel, you can also work with PDG services directly in your graphs with the following TOP nodes:

How-tos

These are the steps for using a PDG service with one of the supported node types.

Add a PDG service

  1. In the PDG Services window, click the Create Service button.

    The Add PDG Service window appears.

  2. In the Add PDG Service window, specify your details for the new service including configuring any nodes.

    For this step, see the node’s documentation for information to configure the nodes.

  3. Click Add.

    The new registered service appears in the PDG Services list.

    When any associated configured nodes cook, they use the new PDG services pool to cook their work items.

Start a PDG service

  1. In the PDG Services window, select the service you want to start from the PDG Services list list.

  2. Click Start.

Stop a PDG service

  1. In the PDG Services window, select the service you want to stop from the PDG Services list list.

  2. Click Stop.

Remove a PDG service

  1. In the PDG Services window, click the Delete Service button.

    The selected service is removed from the PDG Services list.

Tip

You can re-register a PDG service at any time.

Add Nodes to PDG Service

The supported nodes for PDG service are HDA Processor, ROP Fetch, and Python Script

  1. Create a __supported node___ for PDG Services This example uses a Rop Fetch TOP node.

  2. Select the ROP Fetch TOP node.

  3. In the Parameters Editor window, use the Cook Type dropdown menu and select Service.

  4. In Service Name, use the dropdown arrow and select a PDG Service.

Your node is added to the selected PDG Service.

Executing tasks with PDG/TOPs

Basics

Beginner Tutorials

Next steps

  • Running external programs

    How to wrap external functionality in a TOP node.

  • File tags

    Work items track the results created by their work. Each result is tagged with a type.

  • PDG Path Map

    The PDG Path Map manages the mapping of paths between file systems.

  • Feedback loops

    You can use for-each blocks to process looping, sequential chains of operations on work items.

  • Service Blocks

    Services blocks let you define a section of work items that should run using a shared Service process

  • PDG Services

    PDG services manages pools of persistent Houdini sessions that can be used to reduce work item cooking time.

  • Integrating PDG with render farm schedulers

    How to use different schedulers to schedule and execute work.

  • Visualizing work item performance

    How to visualize the relative cook times (or file output sizes) of work items in the network.

  • Event handling

    You can register a Python function to handle events from a PDG node or graph

  • Tips and tricks

    Useful general information and best practices for working with TOPs.

  • Troubleshooting PDG scheduler issues on the farm

    Useful information to help you troubleshoot scheduling PDG work items on the farm.

  • PilotPDG

    Standalone application or limited license for working with PDG-specific workflows.

Reference

  • All TOPs nodes

    TOP nodes define a workflow where data is fed into the network, turned into work items and manipulated by different nodes. Many nodes represent external processes that can be run on the local machine or a server farm.

  • Processor Node Callbacks

    Processor nodes generate work items that can be executed by a scheduler

  • Partitioner Node Callbacks

    Partitioner nodes group multiple upstream work items into single partitions.

  • Scheduler Node Callbacks

    Scheduler nodes execute work items

  • Custom File Tags and Handlers

    PDG uses file tags to determine the type of an output file.

  • Python API

    The classes and functions in the Python pdg package for working with dependency graphs.

  • Job API

    Python API used by job scripts.

  • Utility API

    The classes and functions in the Python pdgutils package are intended for use both in PDG nodes and scripts as well as out-of-process job scripts.