So, finally getting Hqueue going at home, I'm quite impressed with the API and feature set.

Before I invest too much time writing up some more complex dependency dispatching and a PySide UI, has anyone scaled Hqueue up to a farm size with 1000's of clients and workstations?

Obviously you'd need the MySQL database activated on a dedicated server with boatloads of RAM and networking, but the xmlrpc interface worries me a bit from a performance standpoint. Just curious if anyone has really hammered it successfully?

Cheers,

Peter B

during our test (granted 3 or so years ago) we found it very slow on submissions.

a render with 2000 frames would take several minutes to submit because of the way created json objects to submit to the database. a other issue that a task could not have multiple parents meaning you couldn't do a proper DAG submission.

I hope this is different now, but we decided to go for gridengine.

Hey Lars,

Good info, the slow job submission would be killer for sure.

Looking at the API (but not yet trying it) it looks like multiple parents are possible. That would be a dealbreaker for sure of course.

BTW, I notice “parent” and “child” are used backwards in the docs. In all render queues I've ever used, a Parent job must complete before a Child, but the API docs make it sound like the opposite in Hqueue. Perhaps I'm reading it wrong though

pbowmar
BTW, I notice “parent” and “child” are used backwards in the docs. In all render queues I've ever used, a Parent job must complete before a Child, but the API docs make it sound like the opposite in Hqueue. Perhaps I'm reading it wrong though

I too found it backwards

Hi guys,

I used Hqueue at work on a small farm ( 3 nodes only ) and mostly to launch python jobs and I find that yes, with big jobs ( a lot of frames ) it can be quite slow to submit job …

And the parent / child job relationship is quite tricky as well, there is not clear dependence between 2 different jobs, you have to use one job and create children job on that particular job, which is sometimes what you don't want.
As far as I know there is no way you can link 2 jobs with a parent / child relationship easily.

For simple tasks, Hqueue is a perfect tool, with a great python API, for big farms and complex jobs, I wouldn't go for it …

Cetras
For simple tasks, Hqueue is a perfect tool, with a great python API, for big farms and complex jobs, I wouldn't go for it …

Out of interest, what would you go for ? What are people using these days ?


Thanks,
Drew