Cooking

Generate vs. Cook

Work items are processed in two distinct stages. Nodes will first generate work items, based on the parameter configuration of the node and the attributes and output files of any upstream work items. Generated work items will eventually be cooked by the scheduler associated with the node that owns them, once all of their dependencies are cooked.

Some work items, like the ones produced by the Attribute Create or File Pattern nodes, don't do any work when they cook – they simply exist to hold onto attribute data or a list of output files. Such work items aren't actually processed by a scheduler, and will immediately marked as cooked as soon as all of their dependencies have successfully cooked.

Static vs. Dynamic

Nodes in a PDG graph can be either static or dynamic. Static nodes generate all of their work items at once, when all of the input node(s) have generated their work items. Dynamic nodes, on the other hand, generate work items each time an input work item finishes cooking.

A static processor node has access to the full list of input work items, and can add new work items without waiting for its inputs to cook. The downside is that it can't access cooked output files from those input work items, since the files may not exist yet.

A processor node that is configured to be dynamic is able to access the cooked outputs of upstream work items, and use them to influence the generation of new work items. The downside is that the node has to wait for an input work item to finish cooking before it can generate new work item(s).

It's also possible to configure a node to wait for all input work items to be cooked before generating work items. This has both of the advantages of static and dynamic cooking, but also means that the node will have to wait for all of its inputs to fully cook before doing any processing.

Configuring the Generation Type

By default, PDG will configure nodes automatically based on the files paths and work item attributes they use. If a node needs to read input files from an upstream work item, and PDG knows that upstream work items will produce files when they cook, the node will be set to dynamic. PDG will consider any attribute references found in parameter expressions, the input/output attribute list in the node's template, as well as the input/output attribute configuration specified in the node's PDG_NodeCallback::onConfigureNode() callback.

It's often the case that a node will conditionally use attributes and files from input work items, or conditionally add attributes to the work items that it produces. In order to give PDG the most accurate information, custom nodes should implement the PDG_NodeCallback::onConfigureNode() callback. For example:

PDG_CallbackResult
PDG_CustomProcessor::onConfigureNode(
                            PDG_NodeOptions* node_options)
{
    // Check if the "addfiles" toggle is enabled
    if (evaluateB("addfiles"))
    {
        // Tell PDG that we plan to add output files during the work item
        // generation stage
        node_options->setOutputAttrib(
            PDG_AttributeMap::theOutputFileName,
            PDG_AttributeInfo::eOriginGenerate);
    }

    // Tell PDG that we'll always add a "totalsize" attribute to work items
    // when they cook
    node_options->setOutputAttrib(
        "totalsize", PDG_AttributeInfo::eOriginCook);

    // And, that we'll read an attribute from work items when they generate,
    // based on a parm that specifies the attribute name
    node_options->setInputAttrrib(
        evaluateS("inputattrib"),
        PDG_AttributeInfo::eOriginGenerate);

    return PDG_CallbackResult::eSuccess;
}

It's always possible for the user of a node to explicitly choose the generation type as well, using the Generate When parameters that's present on all processor nodes.