Hi guys,

I am trying to set up TOP-net to cache on the farm with Deadline on Windows with Build 17.5.258.

The deadlinescheduler seems to be configured correctly, but it always raises an error

Invalid BinaryConnection state transition from 4 to Failed. (FranticX.Database.DatabaseConnectionException)

The CommandLine submitter in Deadline is enabled and the paths are mapped. Although there is no “python.exe” existing in “$HFS/bin” with H17.5.258, therefore I mapped it to “$HFS/python27/python.exe” (expanded)

We are using certificates for authentification with deadline, might this be the issue?

Does someone have any suggestion how to fix this?

Thanks and cheers

Not really an amazing answer here, but we had many troubles with deadline there and a lot of things have been ironed out when we tried on 17.5.293. Might worth a try just in case

Yes, please update to the latest Houdini version, if possible. Otherwise, I need some further information:

Are you able to launch the Deadline Monitor from your workstation?

Are you able to use Deadline command from your machine outside of PDG? To verify, try using the Deadline Mantra plugin from within Houdini. Or open up a shell, and execute the deadline command to do something that queries the repository:

E.g.:

How are you specifying the repository and certificate data on the TOP Deadline scheduler?

There is a thread on the Deadline forums for the same error. Perhaps something in there can help? https://forums.thinkboxsoftware.com/t/repository-connection-error/24379/5 [forums.thinkboxsoftware.com]

Thanks you for your replies!

SirAlucard
Not really an amazing answer here, but we had many troubles with deadline there and a lot of things have been ironed out when we tried on 17.5.293. Might worth a try just in case

Switched to 17.5.293 to check if it is working. Unfortunately not.

seelan
Are you able to launch the Deadline Monitor from your workstation?

Yes, this is working absolutely fine. I am able to see all submitted jobs and am able to modify them. Using the same repository as in the Deadline scheduler. Also the Deadline ROP is working just as it should. Only the TOP Deadline submitter is causing trouble.

seelan
How are you specifying the repository and certificate data on the TOP Deadline scheduler?

It is a network adress, so I set it exactly as in Deadline but with an additional backslash to resolve the server path

Everything else I tried to set just as decribed in various posts in the PDG section. See attachments.

The jobs are not getting scheduled in Deadline Monitor. Got the same error from Houdini console
Invalid BinaryConnection state transition from 4 to Failed. (FranticX.Database.DatabaseConnectionException)
Is there anything configured falsely?


Thank you for your support!

If you are using a certificate, then you'll need to also specify the certificate in the Repository path on the TOP Deadline scheduler, like so: \\srvvalt\DeadlineRepository10:{port};{path to certificate}:{passphrase}

But if the Deadline ROP just works (without specifying the repository path), then you should be able to leave the Repository field empty in the TOP Deadline node.

seelan
But if the Deadline ROP just works (without specifying the repository path), then you should be able to leave the Repository field empty in the TOP Deadline node.

Thank you seelan!

That did the trick!
Just left the Repository field empty and the jobs got sent to Deadline!

Amazing!

Everything works!
Files get written to the right paths and so on.

Now every task is listed as a separate job in Deadline. Is there a chance to group them together on submission?

The empty repository field just means to use the installation setup of Deadline, which handles the certificate issue as well since presumably you already have it setup. I'll be improving the UX of this so it will be more intuitive.

As for the separate jobs issue, I am working on a solution that will make every work item a task, scheduled under a single PDG job. This is very close to being completed, but does mean a significant change to the behaviour and the TOP Deadline node itself. For the better . Anyway, will post on this forum when it is ready.

RedParrot
Is there a chance to group them together on submission?

I found it myself. I left the “Job Batch Name” Parameter empty. Typed in the job name and the tasks get grouped.
Cheers!

seelan
The empty repository field just means to use the installation setup of Deadline, which handles the certificate issue as well since presumably you already have it setup. I'll be improving the UX of this so it will be more intuitive.

Would be great to have a hint in the scheduler and/or in the help file. Because I believed the Repository parameter is mandatory in order to connect to a database. Have not thought that far that it might use the Deadline standard settings if left empty.

seelan
As for the separate jobs issue, I am working on a solution that will make every work item a task, scheduled under a single PDG job. This is very close to being completed, but does mean a significant change to the behaviour and the TOP Deadline node itself. For the better . Anyway, will post on this forum when it is ready.

That would be great to have them as task. But it is not that bad to have them as separate jobs, in spite of Deadline Monitor gets pretty quickly flooded with job entries.


Thank you so much!
Cheers

RedParrot
Would be great to have a hint in the scheduler and/or in the help file. Because I believed the Repository parameter is mandatory in order to connect to a database. Have not thought that far that it might use the Deadline standard settings if left empty.

Its in the tooltip and online documentation:

RedParrot
That would be great to have them as task. But it is not that bad to have them as separate jobs, in spite of Deadline Monitor gets pretty quickly flooded with job entries.


Thank you so much!
Cheers

Yes I should have mentioned the batch setting as well. But the task feature that will be added will improve performance when scheduling jobs. Deadline's commandline has a delay that adds up when scheduling multiple work items. Plus with this new feature a slave will be able to execute multiple work items at a time, instead of 1 work item per slave as is currently.

Hi. Adding to this thread. I am trying to render USD from PDG.

The jobs are going through to deadline but am getting this error. I can use the command line to render in deadline fine. Just not submit jobs through PDG. Has anyone had any experience with this? Best Mark

=======================================================
Error
=======================================================
Error: FailRenderException : PDGDeadline exception: Traceback (most recent call last):
File "C:\ProgramData\Thinkbox\Deadline10\workers\ca-mw10aah-3\plugins\5fd92de3a3722906a47c3068\PDGDeadline.py", line 341, in RenderTasks
exitCode = self.RunProcess( executable, arguments, startupDir, -1 )
Exception: Executable "python.exe" is not rooted, and does not exist in the current directory or in the PATH.
at Deadline.Plugins.DeadlinePlugin.RunProcessAsUser(String executable, String arguments, String startupDirectory, Int32 timeoutMilliseconds, String userName, String domain, String password, Boolean useSu, Boolean preserveEnvironment, Boolean setHomeVariable)
at Deadline.Plugins.DeadlinePlugin.RunProcess(String executable, String arguments, String startupDirectory, Int32 timeoutMilliseconds)

at Deadline.Plugins.DeadlinePlugin.FailRender(String message) (Python.Runtime.PythonException)
File "C:\ProgramData\Thinkbox\Deadline10\workers\ca-mw10aah-3\plugins\5fd92de3a3722906a47c3068\PDGDeadline.py", line 351, in RenderTasks
self.FailRender('PDGDeadline exception: {}'.format(traceback.format_exc(1)))
at Python.Runtime.Dispatcher.Dispatch(ArrayList args)
at __FranticX_GenericDelegate0Dispatcher.Invoke()
at Deadline.Plugins.DeadlinePlugin.RenderTasks()
at Deadline.Plugins.DeadlinePlugin.DoRenderTasks()
at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)
at Deadline.Plugins.PluginWrapper.RenderTasks(Task task, String& outMessage, AbortLevel& abortLevel)

=======================================================
Type
=======================================================
RenderPluginException

=======================================================
Stack Trace
=======================================================
at Deadline.Plugins.SandboxedPlugin.d(DeadlineMessage brn, CancellationToken bro)
at Deadline.Plugins.SandboxedPlugin.RenderTask(Task task, CancellationToken cancellationToken)
at Deadline.Slaves.SlaveRenderThread.c(TaskLogWriter aiw, CancellationToken aix)

=======================================================
Log
=======================================================
2020-12-15 21:44:35: 0: Loading Job's Plugin timeout is Disabled
2020-12-15 21:44:35: 0: SandboxedPlugin: Render Job As User disabled, running as current user 'admin'
2020-12-15 21:44:37: 0: nogui was not added to the CommandLineParser.
2020-12-15 21:44:37: 0: Executing plugin command of type 'Initialize Plugin'
2020-12-15 21:44:37: 0: INFO: Executing plugin script 'C:\ProgramData\Thinkbox\Deadline10\workers\ca-mw10aah-3\plugins\5fd92de3a3722906a47c3068\PDGDeadline.py'
2020-12-15 21:44:37: 0: INFO: *********** PDGDeadline InitializeProcess
2020-12-15 21:44:37: 0: INFO: About: PDG Plugin for Deadline
2020-12-15 21:44:37: 0: INFO: The job's environment will be merged with the current environment before rendering
2020-12-15 21:44:37: 0: Done executing plugin command of type 'Initialize Plugin'
2020-12-15 21:44:37: 0: Start Job timeout is disabled.
2020-12-15 21:44:37: 0: Task timeout is disabled.
2020-12-15 21:44:37: 0: Loaded job: PDG TASKS (5fd92de3a3722906a47c3068)
2020-12-15 21:44:37: 0: Executing plugin command of type 'Start Job'
2020-12-15 21:44:37: 0: DEBUG: S3BackedCache Client is not installed.
2020-12-15 21:44:37: 0: INFO: Executing global asset transfer preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\ca-mw10aah-3\plugins\5fd92de3a3722906a47c3068\GlobalAssetTransferPreLoad.py'
2020-12-15 21:44:37: 0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer...
2020-12-15 21:44:37: 0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py...
2020-12-15 21:44:37: 0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2020-12-15 21:44:37: 0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2020-12-15 21:44:37: 0: Done executing plugin command of type 'Start Job'
2020-12-15 21:44:37: 0: Plugin rendering frame(s): 3
2020-12-15 21:44:37: 0: Executing plugin command of type 'Render Task'
2020-12-15 21:44:37: 0: INFO: StartFrame: 3
2020-12-15 21:44:37: 0: INFO: Startup Directory: E:/UH_construction_2021/UH_projects/deadlineTest/pdgtemp/12676
2020-12-15 21:44:37: 0: INFO: Looking for task file: E:/UH_construction_2021/UH_projects/deadlineTest/pdgtemp/12676/job_514f2aabefbd44cf84ecd1ea7a6774e7\task_3.txt
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable HOUDINI_PATH to E:/UH_construction_2021/UH_projects/deadlineTest/pdgtemp/12676;&
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_JOBID to DL_JOB_ID
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable HOUDINI_MAXTHREADS to 0
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_JOBID_VAR to PDG_JOBID
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_TEMP to E:/UH_construction_2021/UH_projects/deadlineTest/pdgtemp/12676
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_DIR to E:/UH_construction_2021/UH_projects/deadlineTest
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable HFS to C:/PROGRA~1/SIDEEF~1/HOUDIN~1.426
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_RESULT_CLIENT_ID to 078f7065aade4142874ddf8832b7b68c
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_RESULT_SERVER to
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_SCRIPTDIR to E:/UH_construction_2021/UH_projects/deadlineTest/pdgtemp/12676/scripts
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_MQ_CONN_FILE to E:/UH_construction_2021/UH_projects/deadlineTest/pdgtemp/12676/job_514f2aabefbd44cf84ecd1ea7a6774e7/pdgmq_server.txt
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_HTTP_PORT to None
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_DISABLE_TIMESTAMPS to 1
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_USE_PDGNET to 1
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_RELEASE_SLOT_ON_POLL to 0
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_HFS to C:/PROGRA~1/SIDEEF~1/HOUDIN~1.426
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_SHARED_TEMP to E:/UH_construction_2021/UH_projects/deadlineTest/pdgtemp/12676
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_BREAKAWAY_FROM_JOB to 1
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_PYATTRIB_LOADER to
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_JOBUSE_PDGNET to 1
2020-12-15 21:44:37: 0: INFO: Setting os PATH with HFS: C:\Program Files\Thinkbox\Deadline10\bin\Library\bin;C:\Program Files\Thinkbox\Deadline10\bin\Library\bin;C:\Program Files\Thinkbox\Deadline10\bin\Library\bin;C:\Program Files\Thinkbox\Deadline10\bin\;;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\TortoiseSVN\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\Pulse Secure\VC142.CRT\X64\;C:\Program Files (x86)\Pulse Secure\VC142.CRT\X86\;C:\Docs\shared_programs\ffmpeg-N-100405-gbf4b9e933f-win64-gpl-shared-vulkan\bin;;C:/PROGRA~1/SIDEEF~1/HOUDIN~1.426/bin
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PATH to C:\Program Files\Thinkbox\Deadline10\bin\Library\bin;C:\Program Files\Thinkbox\Deadline10\bin\Library\bin;C:\Program Files\Thinkbox\Deadline10\bin\Library\bin;C:\Program Files\Thinkbox\Deadline10\bin\;;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files\TortoiseSVN\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\Pulse Secure\VC142.CRT\X64\;C:\Program Files (x86)\Pulse Secure\VC142.CRT\X86\;C:\Docs\shared_programs\ffmpeg-N-100405-gbf4b9e933f-win64-gpl-shared-vulkan\bin;;C:/PROGRA~1/SIDEEF~1/HOUDIN~1.426/bin
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_TEMP to E:/UH_construction_2021/UH_projects/deadlineTest/pdgtemp/12676
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_SHARED_TEMP to E:/UH_construction_2021/UH_projects/deadlineTest/pdgtemp/12676
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_SCRIPTDIR to E:/UH_construction_2021/UH_projects/deadlineTest/pdgtemp/12676/scripts
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_DIR to E:/UH_construction_2021/UH_projects/deadlineTest
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_HFS to C:/PROGRA~1/SIDEEF~1/HOUDIN~1.426
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable HFS to C:/PROGRA~1/SIDEEF~1/HOUDIN~1.426
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PYTHON to
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_JOBID to 5fd92de3a3722906a47c3068
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_JOB_BATCH_NAME to PDG simpleUSD 2020-12-15 21:42:57.374000
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_ITEM_NAME to usdrender1_14
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_INDEX to 3
2020-12-15 21:44:37: 0: INFO: Submit as job: False
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_SUBMIT_AS_JOB to False
2020-12-15 21:44:37: 0: INFO: Waiting for MQ connection file E:/UH_construction_2021/UH_projects/deadlineTest/pdgtemp/12676/job_514f2aabefbd44cf84ecd1ea7a6774e7/pdgmq_server.txt
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_RESULT_SERVER to 192.168.0.13:50621
2020-12-15 21:44:37: 0: INFO: PDG_RESULT_SERVER: 192.168.0.13:50621
2020-12-15 21:44:37: 0: INFO: Setting Process Environment Variable PDG_HTTP_PORT to 50622
2020-12-15 21:44:37: 0: INFO: PDG_HTTP_PORT: 50622
2020-12-15 21:44:37: 0: INFO: Task Executable: python.exe
2020-12-15 21:44:37: 0: INFO: Task Arguments: "E:/UH_construction_2021/UH_projects/deadlineTest/pdgtemp/12676/scripts/usdrender.py" "--hfs" "C:/PROGRA~1/SIDEEF~1/HOUDIN~1.426"
2020-12-15 21:44:37: 0: INFO: Invoking: Run Process
2020-12-15 21:44:37: 0: INFO: File "python.exe" is not rooted, checking current directory
2020-12-15 21:44:37: 0: INFO: File "python.exe" is not rooted and is not in the current directory, checking PATH
2020-12-15 21:44:37: 0: INFO: File "python.exe" is not rooted, is not in the current directory, and does not exist in PATH
2020-12-15 21:44:37: 0: Done executing plugin command of type 'Render Task'

=======================================================
Details
=======================================================
Date: 12/15/2020 21:44:41
Frames: 3
Elapsed Time: 00:00:00:06
Job Submit Date: 12/15/2020 21:42:59
Job User: admin
Average RAM Usage: 9389780992 (28%)
Peak RAM Usage: 9389780992 (28%)
Average CPU Usage: 4%
Peak CPU Usage: 10%
Used CPU Clocks (x10^6 cycles): 4515
Total CPU Clocks (x10^6 cycles): 112863

=======================================================
Worker Information
=======================================================
Worker Name: ca-mw10aah-3
Version: v10.1.11.5 Release (a9bdcb38a)
Operating System: Windows 10 Education
Running As Service: No
Machine User: admin
IP Address: 192.168.0.13
MAC Address: 04:92:26:5A:F6:7E
CPU Architecture: x64
CPUs: 12
CPU Usage: 14%
Memory Usage: 8.7 GB / 31.9 GB (27%)
Free Disk Space: 646.663 GB (60.146 GB on C:\, 325.690 GB on D:\, 260.827 GB on E:\)
Video Card: NVIDIA GeForce GTX 1080 Ti

With that path mapping, try setting the Python Job Parm on Deadline Scheduler TOP to \$PYTHON instead.

Hi Chris

That worked perfectly. Thank you, and have a great Christmas! Best Mark

Hi Again

So I am able to render fine. But.. If I use the "submit graph as job" option so I can close Houdini down the job goes to Deadline, but just sits there not rendering anything. Is there something else I need to do to get this to work? Best Mark

Here is my verbose log:

14:23:50 Not copying plugins folder because it exists!
14:23:50 Setting job directory: E:/UH_construction_2021/UH_projects/deadlineTest/pdgtemp/11564/job_0b67c7790889433380cb02ba4a46c1f8
14:23:50 Job file=E:/UH_construction_2021/UH_projects/deadlineTest/pdgtemp/11564/job_0b67c7790889433380cb02ba4a46c1f8/pdg_dl_job.txt
Plugin file=E:/UH_construction_2021/UH_projects/deadlineTest/pdgtemp/11564/job_0b67c7790889433380cb02ba4a46c1f8/pdg_dl_plugin.txt
14:23:50 Task 0 file: E:/UH_construction_2021/UH_projects/deadlineTest/pdgtemp/11564/job_0b67c7790889433380cb02ba4a46c1f8/task_0.txt
14:23:50 Submit As Job: Deadline command =
14:23:52 Submit job id: 5fda187804db5346d0738a99

Mark Wallman
but just sits there not rendering anything. Is there something else I need to do to get this to work? Best Mark

What does it look like in the Deadline monitor? Does the first task have any output? It could be that there aren't enough matching workers to take additional tasks?

Hi

My Deadline worker says it is working. And the monitor says is is rendering. It just does not do anything. If I do not "submit graph as job" it all works. Best

You need at least 2 workers, one to run the houdini PDG cook and the other to execute the work item jobs which are created. They can be on the same machine, because the Houdini cook is generally not very expensive cpu-wise.

Hi Chris. That make perfect sense. Thank you. I am doing test on a single machine right now so only have one worker. I will do tests on our farm instead.

Thanks again!

Best