Inheritance |
|
Methods ¶
applicationBin(name, work_item)
→ str
Returns the path to an application specified by the given name. The work_item
argument may be used to evaluate parms on the Scheduler node.
For example: scheduler.applicationBin('python', None)
will return the path to a python executable to be used for python-based jobs.
cleanTempDirectory()
Deletes the temporary directory and its contents.
context
: pdg.GraphContext
Property
The PDG Graph Context which contains this scheduler.
isCompressWorkItemData
: bool
Property
Set to True
if the scheduler is configured to compress work item JSON data.
isWaitForFailures
: bool
Property
True
if the scheduler is configured to wait for failed work items to be resolved before ending the cook.
dependencyGraph(expand_partitions=False)
→ tuple
of dict
Returns the static dependency graph for work items that would normally be processed by this scheduler. The graph is represented as a tuple containing a map of work item->dependencies, work item->dependents and a list of items currently ready to cook.
The expand_partitions
argument configures whether partitions should be expanded into a flat dependency graph containing only regular work items, or kept in the tuple.
name
: str
Property
The name of the scheduler.
cookError(message)
Issues a cook error on this node.
cookWarning(message)
Issues a cook warning on this node.
cookWorkItem(work_item)
Cooks the specified work item in process, if the work item’s isInProcess
property is set to True
.
evaluateBoolOverride(node, prefix, parm, work_item, default_value)
→ bool
Returns the evaluated boolean value of a scheduler parameter, checking if the parameter has been overridden on the node passed to this method.
This method will return the overridden scheduler parameter value if it has been overridden by the node passed to the method. Otherwise, it will return the scheduler’s evaluated parameter value. If the parameter cannot be found on the node or the scheduler, the default value passed to this method will be returned.
The node
argument should be the node of the current work item being cooked by the scheduler.
The prefix
parameter should specify the prefix of the parameter. For example, all overridable Local scheduler parameters are prefixed with “local”.
parm
is the name of the parameter.
work_item
is the work item currently being cooked by the scheduler.
default_value
is the value to return if the parameter cannot be found on the node or on the scheduler.
evaluateFloatOverride(node, prefix, parm, work_item, default_value)
→ float
Returns the evaluated float value of a scheduler parameter, checking if the parameter has been overridden on the node passed to this method.
This method will return the overridden scheduler parameter value if it has been overridden by the node passed to the method. Otherwise, it will return the scheduler’s evaluated parameter value. If the parameter cannot be found on the node or the scheduler, the default value passed to this method will be returned.
The node
argument should be the node of the current work item being cooked by the scheduler.
The prefix
parameter should specify the prefix of the parameter. For example, all overridable Local scheduler parameters are prefixed with “local”.
parm
is the name of the parameter.
work_item
is the work item currently being cooked by the scheduler.
default_value
is the value to return if the parameter cannot be found on the node or on the scheduler.
evaluateIntOverride(node, prefix, parm, work_item, default_value)
→ int
Returns the evaluated integer value of a scheduler parameter, checking if the parameter has been overridden on the node passed to this method.
This method will return the overridden scheduler parameter value if it has been overridden by the node passed to the method. Otherwise, it will return the scheduler’s evaluated parameter value. If the parameter cannot be found on the node or the scheduler, the default value passed to this method will be returned.
The node
argument should be the node of the current work item being cooked by the scheduler.
The prefix
parameter should specify the prefix of the parameter. For example, all overridable Local scheduler parameters are prefixed with “local”.
parm
is the name of the parameter.
work_item
is the work item currently being cooked by the scheduler.
default_value
is the value to return if the parameter cannot be found on the node or on the scheduler.
evaluateStringOverride(node, prefix, parm, work_item, default_value)
→ str
Returns the evaluated string value of a scheduler parameter, checking if the parameter has been overridden on the node passed to this method.
This method will return the overridden scheduler parameter value if it has been overridden by the node passed to the method. Otherwise, it will return the scheduler’s evaluated parameter value. If the parameter cannot be found on the node or the scheduler, the default value passed to this method will be returned.
The node
argument should be the node of the current work item being cooked by the scheduler.
The prefix
parameter should specify the prefix of the parameter. For example, all overridable Local scheduler parameters are prefixed with “local”.
parm
is the name of the parameter.
work_item
is the work item currently being cooked by the scheduler.
default_value
is the value to return if the parameter cannot be found on the node or on the scheduler.
expandCommandTokens(item_command, work_item)
→ str
Replace the special PDG tokens in the given command with scheduler-specific values.
There are several text tokens that can be used in paths and command strings. The purpose of these tokens is to provide a way of specifying replacements that need to be done for the particular scheduler. For a local-machine only scheduler, these would be absolute paths. For a farm scheduler they would have to be based on environment variables or some other mechanism for doing machine-specific substitutions. The tokens are:
-
__PDG_ITEM_NAME__
-
__PDG_DIR__
-
__PDG_TEMP__
-
__PDG_SCRIPTDIR__
-
__PDG_RESULT_SERVER__
-
__PDG_PYTHON__
-
__PDG_HYTHON__
-
__PDG_HFS__
formatTransferPath(local_path, default_type=pdg.fileTransferFile.TempDir, local=True, check_type=True)
→ str
Returns the destination path that local_path
will be copied to if it’s transfered by calling pdg.Scheduler.transferFile on this scheduler. When using this method files are always copied directly into the target directory, without preserving their directory structure.
local
determines whether or not the local or remote file path prefix should be used in the output.
check_type
determines whether or not the file’s extension should be used to determine which directory the file should be copied to. If check_type
is True
then the default_type
is only used if an extension-specific directory can’t be determined. For example, Python scripts and assets are transfered to the PDG script directory and PDG asset directory respectively. Otherwise, if check_type
is False
, then the default_type
argument will always determine the transfer destination.
formatTransferPath(root_path, local_path, sub_path="", default_type=pdg.fileTransferFile.TempDir, local=True, check_type=True)
→ str
Returns the destination path that local_path
will be copied to if it’s transfered by calling pdg.Scheduler.transferFile on this scheduler. Unlike the other formatTransferPath
variant, this method does preserve intermediate directories in the local file path. The local file path should contain the root_path
– any intermediate directories after the root path will be kept in the transfer path.
sub_path
specifies an optional sub directory within the target directory that should be added to the formatted path.
local
determines whether or not the local or remote file path prefix should be used in the output.
check_type
determines whether or not the file’s extension should be used to determine which directory the file should be copied to. If check_type
is True
then the default_type
is only used if an extension-specific directory can’t be determined. For example, Python scripts and assets are transfered to the PDG script directory and PDG asset directory respectively. Otherwise, if check_type
is False
, then the default_type
argument will always determine the transfer destination.
getPollingClient()
→ PDGNPollingClient
Returns the polling client for this scheduler’s MQ server, if applicable. Returns None if there is none; for example, if this scheduler is local.
isWorkItemReady(workitem_id, index)
→ pdg.workItemReadyResult
Returns pdg.workItemReadyResult.WorkItemReady
if the specified work item is ready to cook. If the work item has failed dependencies the WorkItemFailed
enum entry is returned, otherwise if the work item is still waiting then WorkItemBlocked
is returned.
The index
argument should be -1 if the item is not in a batch, or the batch index of the item if the work item is a sub item in a batch.
jobName(work_item)
→ str
Returns the standard farm job name for the work item. If the work item has a custom label set, this function returns the label as-is. Otherwise, it returns a string using the format @pdg_name @pdg_index:4 @pdg_frame:2
.
localizePath(path)
→ str
Localizes a remote path with respect to the scheduler’s working directory.
This replaces __PDG_DIR__
, __PDG_TEMP__
and __PDG_SCRIPTDIR__
, with the absolute paths on the current machine.
delocalizePath(path)
→ str
Returns the given path to be based on the scheduler text tokens __PDG_DIR__
if possible.
onWorkItemAddOutput(workitem_id, index, path, tag, checksum, active_only=False)
Called to indicate a work item has a new output file. The path
is
the path to the file as a string, tag
is the fie tag string and checksum
is an integer value.
index
should be -1 if the item is not in a batch, or the batch index of
the item if the work item is a sub item in a batch.
If active_only
is True
, the scheduler will ignore calls to this method
that correspond to work items that have already finished cooking, or are not
active in the scheduler.
onWorkItemAddOutputs(workitem_id, index, paths, tag, checksums, active_only=False)
Called to indicate a work item has an array of new output file. paths
is
the array of paths to the file as a string, tag
is the fie tag string or
array of tag strings, and checksums
is an array of integer values.
index
should be -1 if the item is not in a batch, or the batch index of
the item if the work item is a sub item in a batch.
If active_only
is True
, the scheduler will ignore calls to this method
that correspond to work items that have already finished cooking, or are not
active in the scheduler.
This method is more efficient than calling pdg.SchedulerBase.onWorkItemAddOutput multiple times.
onWorkItemAppendLog(workitem_id, index, log_data, log_type)
Appends the specified log data to the end of the work item’s log. The log_type
should be set to one of the enum values from pdg.workItemLogType.
index
should be -1 if the item is not in a batch, or the batch index of
the item if the work item is a sub item in a batch.
onWorkItemCanceled(workitem_id, index)
Called top indicate a work item has been canceled.
index
should be -1 if the item is not in a batch, or the batch index of
the item if the work item is a sub item in a batch.
onWorkItemFailed(workitem_id, index)
Called to indicate a work item has failed.
index
should be -1 if the item is not in a batch, or the batch index of
the item if the work item is a sub item in a batch.
onWorkItemFileResult(workitem_id, index, result, tag, checksum, active_only=False)
This method is deprecated. Use pdg.SchedulerBase.onWorkItemAddOutput instead.
onWorkItemInvalidateCache(workitem_id, index)
Invalidates the cache of the work item, without changing the output file list. This also causes downstream work items to recook instead of using cache files on the next cook. It has the same effect as calling pdg.WorkItem.invalidateCache directly on the work item.
onWorkItemSetAttribute(workitem_id, attribute_name, values)
This method is deprecated. Add an attribute using pdg.SchedulerBase.onWorkItemSetStringAttrib and others.
Called to set/changed an attribute value on a work item.
values
is an array of str
, float
or int
values, and
attribute_name
is the string name of the attribute.
onWorkItemSetCookPercent(workitem_id, index, cook_percent)
Sets the cook percent of the specified work item.
cook_percent
: the new completion percentage of the work item. 0
means not started and 100
means it is complete.
index
should be -1 if the item is not in a batch, or the batch index of
the item if the work item is a sub item in a batch.
onWorkItemSetCustomState(workitem_id, index, custom_state)
Sets the custom state string of the specified work item.
index
should be -1 if the item is not in a batch, or the batch index of
the item if the work item is a sub item in a batch.
onWorkItemSetStringAttrib(workitem_id, index, attribute_name, value, [attrib_index])
Called to set/changed an attribute value on a work item.
attribute_name
is the name of the attribute.
If attrib_index
is passed, it is the index within the attribute to be
set. In that case value
should be a string. Otherwise
value
should be a list of strings.
index
should be -1 if the item is not in a batch, or the batch index of
the item if the work item is a sub item in a batch.
onWorkItemSetIntAttrib(workitem_id, index, attribute_name, value, [attrib_index])
Called to set/changed an attribute value on a work item.
attribute_name
is the name of the attribute.
If attrib_index
is passed, it is the index within the attribute to be
set. In that case value
should be an integer. Otherwise
value
should be a list of values.
index
should be -1 if the item is not in a batch, or the batch index of
the item if the work item is a sub item in a batch.
onWorkItemSetFloatAttrib(workitem_id, index, attribute_name, value, [attrib_index])
Called to set/changed an attribute value on a work item.
attribute_name
is the name of the attribute.
If attrib_index
is passed, it is the index within the attribute to be
set. In that case value
should be a real number. Otherwise
value
should be a list of values.
index
should be -1 if the item is not in a batch, or the batch index of
the item if the work item is a sub item in a batch.
onWorkItemSetFileAttrib(workitem_id, index, attribute_name, value, [attrib_index])
Called to set/changed an attribute value on a work item.
attribute_name
is the name of the attribute.
If attrib_index
is passed, it is the index within the attribute to be
set. In that case value
should be a pdg.File. Otherwise
value
should be a list of pdg.File.
index
should be -1 if the item is not in a batch, or the batch index of
the item if the work item is a sub item in a batch.
onWorkItemSetDictAttrib(workitem_id, index, attribute_name, value, [attrib_index])
Called to set/changed an attribute value on a work item.
attribute_name
is the name of the attribute.
If attrib_index
is passed, it is the index within the attribute to be
set. In that case value
should be a pdg.Dictionary. Otherwise
value
should be a list of pdg.Dictionary.
index
should be -1 if the item is not in a batch, or the batch index of
the item if the work item is a sub item in a batch.
onWorkItemSetPyObjectAttrib(workitem_id, index, attribute_name, object_repr)
Called to set/changed an attribute value on a work item.
object_repr
should be a string repr
of a python object, and
attribute_name
is the string name of the attribute.
index
should be -1 if the item is not in a batch, or the batch index of
the item if the work item is a sub item in a batch.
onWorkItemStartCook(workitem_id, index, clear_outputs=False)
Called to indicate a work item has begun cooking. This method has no effect if the work item has already completed cooking.
index
should be -1 if the item is not in a batch, or the batch index of
the item if the work item is a sub item in a batch.
If clear_outputs
is True
, any existing output files added to the work item from previous cooks will be clear.
onWorkItemSucceeded(workitem_id, index)
Called to indicate a work item has successfully finished.
index
should be -1 if the item is not in a batch, or the batch index of
the item if the work item is a sub item in a batch.
getStatusURI(work_item)
→ str
Called to return the status URI for the specified work item. This appears in the MMB detail window of a work item. It can be formatted to point to a local file with file:///
or a web page with 'http://'.
getLogURI(work_item)
→ str
Returns the log URI for the specified work item. This appears in the MMB detail window of a work item, and is also available with the special @pdg_log
attribute. It can be formatted to point to a local file with file:///
or a web page with 'http://'.
submitAsJob(graph_name, node_name)
→ str
Submits the current network as a stand-alone job with this scheduler. The job will typically be a hython
process the loads the current hip
file and cooks the display node of the topnet
that the submitting scheduler was in. The scheduler that is used to cook the topnet
is decided by the topnet
Default TOP Scheduler parm. Or by the TOP Scheduler Override
parm on the TOP nodes.
runOnMainThread(wait, function, work_item=None, *args, **kwargs)
→ object
or pdg.PyDeferredTask
Runs the specified callable object function
on the main thread. If wait
is True
this method will block until the function has finished executing, otherwise it will return immediately. The return value produced by the function
is returned by this method when wait
is True
. If wait
is False
this method will return a pdg.PyDeferredTask handle that refers to the task that will run on the main thread. It’s possible to wait on the task at any point by calling pdg.PyDeferredTask.wait on the handle.
Extra positional and key word arguments passed are passed to the function
when it get invoked.
The method looks for a special work_item
keyword argument to determine which work item should be made active while function
is evaluating. If that argument is provided, the work item will be used for any parameter evaluations performed by the function.
Only one function can run on the main thread at a time. Each time this method is called the function
is added to a queue of pending main thread calls, which are processed by the main thread while the PDG graph is cooking. If the graph is canceled any queued operations that have not yet begun to execute are also canceled. Scripts that are waiting on this method will be unblocked, with a return result of None
.
For example, putting following code in a Python Script TOP will create work items that displays a Qt message that blocks the cook from continuing, until someone clicks on the OK button in the Qt dialog:
def show_dialog(message, text): from hutil.Qt import QtWidgets dialog = QtWidgets.QMessageBox() dialog.setWindowTitle(message) dialog.setText(text) dialog.exec_() work_item.node.scheduler.runOnMainThread( True, show_dialog, "Cook PDG Message", "Click 'OK' to continue cooking.")
transferFile(local_path, default_type=pdg.fileTransferType.TempDir, check_type=True)
→ bool
Transfers a local file into the scheduler’s working directory. If local_path
is a file it is copied directly into the destination directory, and intermediate paths are not preserved. Returns True
on success, and throws an exception if an error occurs while transfering the file.
check_type
determines whether or not the file’s extension should be used to determine which directory the file should be copied to. If check_type
is True
then the default_type
is only used if an extension-specific directory can’t be determined. For example, Python scripts and assets are transfered to the PDG script directory and PDG asset directory respectively. Otherwise, if check_type
is False
, then the default_type
argument will always determine the transfer destination.
transferFile(root_path, local_path, sub_path="", default_type=pdg.fileTransferType.TempDir, check_type=True)
→ bool
Transfers a local file into the scheduler’s working directory. Intermediate paths between the root_path
and local_path
are preserved. Returns True
on success, and throws an exception if an error occurs while transfering the file.
The option sub_path
argument specifies a sub directory in the target directory that should be added to the destination path.
check_type
determines whether or not the file’s extension should be used to determine which directory the file should be copied to. If check_type
is True
then the default_type
is only used if an extension-specific directory can’t be determined. For example, Python scripts and assets are transfered to the PDG script directory and PDG asset directory respectively. Otherwise, if check_type
is False
, then the default_type
argument will always determine the transfer destination.
workingDir(local)
→ str
Returns the local absolute path to the working directory if local
is True
, otherwise returns the remote
version of that path, which is specific to this scheduler. This is equivalent to __PDG_DIR__
.
tempDir(local)
→ str
Returns the local absolute path to the temp directory if local
is True
, otherwise returns the remote
version of that path, which is specific to this scheduler. This is the directory which will contain the pdgtemp
subdirectory, which is used to store transient files for the cook, such as log files and script files.
scriptDir(local)
→ str
Returns the local absolute path to the temporary script directory if local
is True
, otherwise returns the remote
version of that path, which is specific to this scheduler. This is a convenience method to return the equivalent of tempDir(local) + "/scripts"
logDir(local)
→ str
Returns the local absolute path to the temporary logs directory if local
is True
, otherwise returns the remote
version of that path, which is specific to this scheduler. This is a convenience method to return the equivalent of tempDir(local) + "/logs"
preExecProcess()
→ None
Should be called in a local subprocess just before child is executed (Posix only). Creates a new session and assigns with the groupid equal to the new pid, for use will os.killpg. This ensures that the entire subprocess tree can be cleaned up. Also ensures that the subprocess lifetime is tied to the parent with a SIGTERM being sent to the subprocess when the parent dies.
postExecProcess(pid)
→ None
maxThreads()
→ int
Returns the number of logical CPU cores available to the process. This will be equal to the core count of the local machine unless modified by hou.setMaxThreads() or the environment variable HOUDINI_MAXTHREADS
.
dataDir(local)
→ str
Returns the local absolute path to the temporary data directory if local
is True
, otherwise returns the remote
version of that path, which is specific to this scheduler. This is a convenience method to return the equivalent of tempDir(local) + "/data"
. This directory holds serialized work items, usually in JSON format to be used by job scripts.
setAcceptInProcess(accept_in_process)
Sets whether or not the scheduler accepts in-process work items. By default this is False
, except for the In-Process Scheduler TOP.
setWorkingDir(local_path, remote_path)
→ None
Sets the values of the local and remote working directory paths. See workingDir
.
setTempDir(local_path, remote_path)
→ None
Sets the values of the local and remote temp directory paths. See tempDir
.
workItemResultServerAddr()
→ str
Returns the network endpoint for the work item result server, in the format <HOST>:<PORT>, this is equivalent to the __PDG_RESULT_SERVER__
command token, and the job environment variable $PDG_RESULT_SERVER. This will typically be an XMLRPC API server.
setWorkItemResultServerAddr(addr)
→ None
Sets the value to be returned by workItemResultServerAddr
startService(service)
→ bool
The PDG Service Manager will call this method to start a service using this scheduler. The scheduler must provide an implementation for this method if it supports PDG services.
The method should return True
to indicate success, and False
for failure.
stopService(service)
→ bool
The PDG Service Manager will call this method to stop a service that was started by this scheduler. The scheduler must provide an implementation for this method if it supports PDG services.
The method should return True
to indicate success, and False
for failure.
workItemDataSource
: pdg.workItemDataSource
Property
The /tops/pdg/Py.pdg.WorkItemDataSource.html setting for this scheduler, which determines how item data is transferred to a remote worker machine.
Methods from pdg.TypeInstance ¶
type
: pdg.BaseType
Property
The pdg.BaseType object used to construct this instance
typeName
: str
Property
The type name of the type
object.