Forums 3rd Party Houdini Docker - Containerize & automate Houdini installs

Houdini Docker - Containerize & automate Houdini installs

   6366   10   4
User Avatar
aaronsmithtv
Member
18 posts
Joined: Feb. 2018
Offline
 
Hey everyone!

I'm super excited to share with you a project I've been working on called Houdini-Docker [github.com]; An open-source automation utility designed to make containerizing Houdini simple & easy.

Here's why you might want to check out Houdini-Docker:

  • It provides an alternative to the current standard workflow, which involves manual installation and Dockerfile configuration.
  • It offers a trustworthy source of Houdini installation images for those who need security, on Docker Hub [hub.docker.com]; As all workflow runs are publicly visible for transparency.
  • Images generated by Houdini-Docker are approximately 65% smaller than the most popular Houdini installation images, saving storage space and improving deployment speed.
  • Keep up-to-date with the latest production builds within future Dockerfiles, without the need for executing pipeline scripts.
  • Daily checks for the latest production builds, so new builds will always be available on Docker Hub within 24 hours.

All standard base images come with a full Houdini installation built on Debian 11 slim. The PATH environment variable is set for easy access to processes such as hython and hserver.

Quick-Start:
To get started with the most recent production build, just execute the following: (requires a Docker installation)

1. Pull the latest production build image (19.5.569-base as of 30/03/2023)
docker pull aaronsmithtv/hbuild:latest

2. Run a temporary container from the pulled image
docker run -it --rm --name houdini-container aaronsmithtv/hbuild:latest

3. Start a sesinetd instance (The command line Houdini license server), and verify that you have license entitlements with sesictrl (The command line equivalent of HKey)
sesinetd
sesictrl redeem

You should receive a prompt to enter your SideFX account credentials, and upon logging in, a list of license entitlements. Type q to quit the license utility and exit to close the container.

If you have specific requirements, such as another Linux distro, or you have more granular requirements for packages, builds, etc, simply forking the repository will enable the auto-build GitHub actions workflow to work with whatever additions you make.

I want to use Houdini-Docker, but I don't know what for!
Many studios will likely have their own CICD [about.gitlab.com] workflows that implement similar processes - but for those who are undergoing the task of deploying Houdini in some way, for example to a cloud platform, or as a license sever, it can be difficult to maintain a tool such as this (especially when many products are being juggled). Houdini-Docker aims to alleviate the difficulties in automating a highly customizable Houdini build process using Docker, while Hbuild is a repository for pre-built Houdini installation images based on Debian.

Here are some ideas for projects you can try out using a Houdini-Docker image:
  • A non-graphical Houdini instance as a web server using hwebserver [www.sidefx.com]
  • A containerized license utility
  • A containerized hqueue client

I'd love to hear your thoughts, feedback, and any ideas you might have. If you need any specific builds just let me know, and I'll add them to the Docker Hub repository. PRs and issues on GitHub are totally welcome!

You can find Houdini-Docker on GitHub here: https://github.com/aaronsmithtv/houdini-docker [github.com]
You can find the image repository Hbuild here: https://hub.docker.com/r/aaronsmithtv/hbuild [hub.docker.com]
Edited by aaronsmithtv - March 31, 2023 12:20:16
 Report
User Avatar
aaronsmithtv
Member
18 posts
Joined: Feb. 2018
Offline
 
For anyone interested, I have also open sourced and added the Python driver I use for accessing the SideFX Web API, sesiweb 0.1.0 [github.com], to pypi.

Import it into your scene by doing the following:

pip install --upgrade sesiweb

And then import the session class as follows: 

from sesiweb import SesiWeb

You can find more information in the sesiweb GitHub README [github.com], but if anyone wants to start quickly, here is a similar script to what I use in Houdini-Docker, using sesiweb 

from sesiweb import SesiWeb
from sesiweb.model.service import ProductBuild

sesi_secret = "your_secret_key"
sesi_id = "your_client_id"

sw = SesiWeb(sesi_secret, sesi_id)

# Get the most recent Houdini product builds
build = {"product": "houdini", "platform": "linux"}

# Get the latest Houdini build
build = sw.get_latest_build(prodinfo=build, only_production=False)

# Get the download URL, filename and hash of the build
build_dl = sw.get_build_download(
	prodinfo=ProductBuild(**build.dict())

print(build_dl)

sesiweb has full typehint support, input validation using Pydantic, custom exceptions and class test coverage. Enjoy!
 Report
User Avatar
drew
Member
120 posts
Joined: July 2005
Offline
 
Hi Aaron,

Really interested in this work, and timely as I'm currently trying to automate building a hqueue farm. In the past I've done it with cobbled together scripts and files shared via NFS mounts, but now that hqueue has python3 support it's time to modernise.

At the moment I get this error if I try and run houdini with your container. Hython and hscript work fine. I'm running on Ubuntu 22.04. (For my purposes I'll mostly be running headless tasks.)

root@539d5b0c2631:/opt/hfs19.5# houdini
/opt/hfs19.5/bin/houdini-bin: error while loading shared libraries: libsmime3.so: cannot open shared object file: No such file or directory

Cheers,
Drew
 Report
User Avatar
aaronsmithtv
Member
18 posts
Joined: Feb. 2018
Offline
 
drew
Hi Aaron,

Really interested in this work, and timely as I'm currently trying to automate building a hqueue farm. In the past I've done it with cobbled together scripts and files shared via NFS mounts, but now that hqueue has python3 support it's time to modernise.

At the moment I get this error if I try and run houdini with your container. Hython and hscript work fine. I'm running on Ubuntu 22.04. (For my purposes I'll mostly be running headless tasks.)

root@539d5b0c2631:/opt/hfs19.5# houdini
/opt/hfs19.5/bin/houdini-bin: error while loading shared libraries: libsmime3.so: cannot open shared object file: No such file or directory

Cheers,
Drew

Hey Drew, thanks for checking out my project, I hope it will help you automate your hqueue farm

I suppose I hadn't thought too much about the GUI libs, I designed the image itself around saving space to run Hython and the other headless Houdini processes. Now I'm looking at the libs though, it seems that they don't take up much space at all (around 6mb) and not installing them prevents you from launching houdini (clearly a bad idea on my part!)

Run this command in your container for now to get it running, and I'll create an issue on GitHub:

apt-get update && apt-get install -y libglu1 libsm6 bc wget libnss3 libxcomposite1 libxrender1 libxrandr2 libfontconfig1 libxcursor1 libxi6 libxtst6 libxkbcommon0 libxss1 libpci3 libasound2
 Report
User Avatar
graham
Member
1922 posts
Joined: Nov. 2006
Offline
 
Hey Aaron, thanks for doing all this!

I've been building my own image for spinning up automated testing of my projects and thought I'd give your stuff a shot to reduce stuff I need to manage!

Overall I didn't really have any issues at all getting my stuff set up that weren't really my own. My only comment I suppose would be the lack of python in the image as I needed Python for a bunch of stuff. Given the amount of stuff necessary for it, it's probably not the most friendly for image size so it can make sense leaving it out by default. Install it easy so it's not a huge issue.

Another thing I ran across was trying to use your sesiweb package inside the container to help set up licensing (as opposed to my own to reduce stuff I'd need to vendor). Despite the project listing python-3.7+ it looks like you're using some of the type hint unioning that was added in 3.10 so it won't run under 3.9. It seems that 3.9 is also the latest release supported in Debian 10's repo so you're left with compiling from src to get newer versions. Not a big deal but just wanted to highlight it.

Thanks again for all the work!
Graham Thompson, Technical Artist @ Rockstar Games
 Report
User Avatar
aaronsmithtv
Member
18 posts
Joined: Feb. 2018
Offline
 
graham
Hey Aaron, thanks for doing all this!

I've been building my own image for spinning up automated testing of my projects and thought I'd give your stuff a shot to reduce stuff I need to manage!

Overall I didn't really have any issues at all getting my stuff set up that weren't really my own. My only comment I suppose would be the lack of python in the image as I needed Python for a bunch of stuff. Given the amount of stuff necessary for it, it's probably not the most friendly for image size so it can make sense leaving it out by default. Install it easy so it's not a huge issue.

Another thing I ran across was trying to use your sesiweb package inside the container to help set up licensing (as opposed to my own to reduce stuff I'd need to vendor). Despite the project listing python-3.7+ it looks like you're using some of the type hint unioning that was added in 3.10 so it won't run under 3.9. It seems that 3.9 is also the latest release supported in Debian 10's repo so you're left with compiling from src to get newer versions. Not a big deal but just wanted to highlight it.

Thanks again for all the work!

Hey Graham, it's great to see that you're happy using sesiweb and Houdini-Docker, even with all of the bugs haha! I managed to get a free moment today and fixed the compatibility issues you were having in 0.1.1 so if you'd like to use the latest pypi wheel on Debian 10 you are free to do so

drew
Hi Aaron,

Really interested in this work, and timely as I'm currently trying to automate building a hqueue farm. In the past I've done it with cobbled together scripts and files shared via NFS mounts, but now that hqueue has python3 support it's time to modernise.

At the moment I get this error if I try and run houdini with your container. Hython and hscript work fine. I'm running on Ubuntu 22.04. (For my purposes I'll mostly be running headless tasks.)

root@539d5b0c2631:/opt/hfs19.5# houdini
/opt/hfs19.5/bin/houdini-bin: error while loading shared libraries: libsmime3.so: cannot open shared object file: No such file or directory

Cheers,
Drew

This has now been fixed in 0.1.1! You should be able to use the GUI version of Houdini from the latest hbuild image, or Houdini-Docker installation now.
 Report
User Avatar
Datagen
Member
1 posts
Joined: Dec. 2021
Offline
 
Hi Aaron!
Thank you for your development!
I faced some requirement/idea: Houdini HQueue server as application for "Truenas scale". That will be great to move the server form one of clients machines.

Thanks again
Andrey
 Report
User Avatar
aaronsmithtv
Member
18 posts
Joined: Feb. 2018
Offline
 
Datagen
Hi Aaron!
Thank you for your development!
I faced some requirement/idea: Houdini HQueue server as application for "Truenas scale". That will be great to move the server form one of clients machines.

Thanks again
Andrey

Hey Andrey, Houdini Docker creates a ready-to-use, base Houdini installation process using Debian. Provided that you're able to support containerization through Docker then in theory the process should be compatible with any version of Linux, including TrueNAS SCALE.

You could begin by changing around some install instructions in the core Dockerfile to install only HQueue, creating your ideal image that way. What is your intended approach for using HQueue and Docker? I am happy to lend assistance if I can know a little more about what you're trying to use Houdini Docker for
 Report
User Avatar
NicTanghe
Member
212 posts
Joined: Dec. 2016
Offline
 
Hello evryone.

I was looking for a seperate hqueserver aswell as a seperate license server image aswell.
Has annyone made some already ?
preferably with yamls ?
 Report
User Avatar
aaronsmithtv
Member
18 posts
Joined: Feb. 2018
Offline
 
NicTanghe
Hello evryone.

I was looking for a seperate hqueserver aswell as a seperate license server image aswell.
Has annyone made some already ?
preferably with yamls ?

I think one of the forks of the repo may already doing it, but you can always just change the installation in the Dockerfile: https://github.com/aaronsmithtv/Houdini-Docker/blob/main/hinstall/Dockerfile [github.com]

And use these instructions on how to run the image: https://github.com/aaronsmithtv/Houdini-Docker/tree/main/hinstall [github.com]

If you feel like you need to tailor the installation to any specific needs, a lot of the secrets are held inside houdini.install, which is the standard x86 Houdini for Linux installation package, downloaded through the web API. It's definitely easier than it looks
 Report
User Avatar
NicTanghe
Member
212 posts
Joined: Dec. 2016
Offline
 
Hello !
my automated gh build go`s say it succeeds on gh but then if i check the dockerhub.
builds i think is stil empty and all i c is an add for automated builds.

the logs still had the secrets as **** not sure that indicates a blank value or that it is keeping them secret.

also no tags show up

Run export PYTHONPATH="${PYTHONPATH}:$(pwd)"
INFO:root:Starting Houdini download client service
INFO:root:Latest build found: `20.5.410-base`
INFO:root:Tag does not exist in repository, beginning download/push process...
INFO:root:Building Docker image...
INFO:root:Built Docker image `sha256:2ed523b725e6`, pushing to Docker hub...

The push refers to repository [docker.io/***/***] -  (Image ID: )
Preparing -  (Image ID: 6f2e2a7eb8c6)
Preparing -  (Image ID: b99d01c6daf9)
INFO:root:Pushed Docker image `20.5.410-base` in `***/***`.
 -  (Image ID: )
The push refers to repository [docker.io/***/***] -  (Image ID: )
Preparing -  (Image ID: 6f2e2a7eb8c6)
Preparing -  (Image ID: b99d01c6daf9)
INFO:root:Pushed Docker image `latest` in `***/***`.
INFO:root:Finished HouDocker process, exiting...
Edited by NicTanghe - Nov. 30, 2024 16:49:07
 Report
  • Quick Links