The proper way of doing asyncio in Houdini 20's python node?

   1896   7   4
User Avatar
Member
4 posts
Joined: March 2017
Offline
In houdini 19.5, I can define async functions and invoke with asyncio.new_event_loop().run_until_complete(async_task())in python node

But in houdini 20, this yields error:

  File "C:\Program Files/Side Effects Software/Houdini 20.0.688/houdini/python3.10libs\haio.py", line 2101, in run_until_complete
    self.run_forever()
  File "C:\Program Files/Side Effects Software/Houdini 20.0.688/houdini/python3.10libs\haio.py", line 2079, in run_forever
    assert not self._isUIAvailable()
AssertionError

I've also tried to call asyncio.run(async_task())directly, and following error was raised:

  File "D:\Program Files\Side Effects Software\Houdini 20.0.688\python310\lib\asyncio\runners.py", line 33, in run
raise RuntimeError(
RuntimeError: asyncio.run() cannot be called from a running event loop

Is it still possible to do async things in python node for Houdini 20? And what's the proper way?
User Avatar
Member
13 posts
Joined: June 2017
Offline
Hello

I would be interested in that too! I have issues running the same asyncio code (that worked in Houdini 19.5) in Houdini 20.

In my case i was running the asyncio loop inside of a thread.

But in H20 this does not seem to be allowed anyomore?


Traceback (most recent call last):
  File "C:\PROGRA~1\SIDEEF~1\HOUDIN~1.724\python310\lib\threading.py", line 1016, in _bootstrap_inner
15:00:26 DEBUG:hatchet.base: Data: Start timers
    self.run()
  File "my_script.py", line 170, in run
    self._loop.create_task(self._async_start_heartbeat())
  File "C:\PROGRA~1/SIDEEF~1/HOUDIN~1.724/houdini/python3.10libs\haio.py", line 2165, in create_task
    check_thread()
  File "C:\PROGRA~1/SIDEEF~1/HOUDIN~1.724/houdini/python3.10libs\haio.py", line 112, in check_thread
    raise RuntimeError("Current thread is not the main thread")
RuntimeError: Current thread is not the main thread
User Avatar
Member
13 posts
Joined: June 2017
Offline
Hey @iiif

Just found something.

Apparently Houdini 20 ships with their own implementation of the asyncio event loop.

In Houdini 19.5 this feature could be enabled by setting the env variable:

HOUDINI_ASYNCIO=1

In Houdini 20 this seems to be enabled by default but can actually be turned off by setting the variable to 0.
After this my asyncio code worked again.

Please note that I don't know if this is a good idea...

The asyncio patch in the changelog for reference:
https://www.sidefx.com/changelog/?journal=&categories=&body=asyncio&version=&build_min=&build_max=&show_versions=on&show_compatibility=on&items_per_page= [www.sidefx.com]

The env variable:
https://www.sidefx.com/docs/houdini/ref/env.html [www.sidefx.com]

Apparently the nvidia omniverse connector also had issues with H20 and asyncio, but they managed to fix it, maybe worth looking in to what they did:
https://forums.developer.nvidia.com/t/houdini-20-and-connector-200-0-0-issue-with-node-parameter-menu/283025/4 [forums.developer.nvidia.com]
User Avatar
Member
2 posts
Joined: Nov. 2020
Offline
Forgive me if this doesn't immediately work as im typing on my phone from memory.

But i believe i got around this by running in a seperate thread while using:

asyncio.run_coroutine_threadsafe(async_function,event_loop)

I'm sure this is the code i used within a dataclass for handling asynchronous websocket communication, making sure to run the new thread with `asyncio.run_coroutine_threadsafe()`, did the job, but I'll confirm later.

def connect(self, server):
	server = 'localhost:8980'
        loop = asyncio.new_event_loop() 
        if not loop.is_running():
            threading.Thread(target=loop.run_forever, daemon=True).start()
        asyncio.run_coroutine_threadsafe(connect_async(server), loop)

async def connect_async(self, server):
User Avatar
Member
642 posts
Joined: June 2006
Offline
Good to know i am not alone
User Avatar
Member
13 posts
Joined: June 2017
Offline
Hey all,

so I found a way to make it work both for houdini 19.5 and Houdini 20 and above.

You can control this with the CREATE_THREAD variable. In 19.5 the event loop needs to run in a separate thread otherwise it's blocking the UI.

The scripts tries to include all sorts of asyncio stuff, like endless coroutines, async generators, executing a last coroutine before houdini shuts down as well as gracefully ending the loop.

Save this in a pythonrc.py file and include it in HOUDINI_PATH so the loop gets started when houdini boots up.

Note: sheduling a coroutine in the atexit event leads to seg fault in houdini 20 and above.

import threading
import asyncio
import atexit
import logging

logging.basicConfig(level=logging.DEBUG)
logger = logging.getLogger("hou-asyncio")
logger.setLevel(logging.DEBUG)

print("Hello from pythonrc")

CREATE_THREAD = False

def get_event_loop() -> asyncio.AbstractEventLoop:
    try:
        loop = asyncio.get_running_loop()
        logger.info("Take existing event loop: %s", loop)
    except RuntimeError:
        loop = asyncio.new_event_loop()
        logger.info("Created new event loop: %s", loop)
    return loop


async def async_start_heartbeat():
    while True:
        logger.info("Heartbeat")
        await asyncio.sleep(1.5)


async def async_generator():
    number = 0
    while True:
        yield number
        number += 1
        await asyncio.sleep(1)


async def async_use_async_generator():
    async for number in async_generator():
        logger.info("Received number: %i", number)


async def async_cancel_all_tasks(loop: asyncio.AbstractEventLoop):
    tasks = [
        t
        for t in asyncio.all_tasks(loop=loop)
        if t is not asyncio.current_task(loop=loop)
    ]
    for task in tasks:
        task.cancel()
    await asyncio.gather(*tasks, return_exceptions=True)


async def async_shutdown_tasks(loop: asyncio.AbstractEventLoop):
    logger.info("Cancelling all tasks...")
    await async_cancel_all_tasks(loop)

    logger.info("Shutting down async generators...")
    await loop.shutdown_asyncgens()


async def async_last_coroutine():
    logger.info("Houdini about to exit, executing last coroutine...")
    for i in range(3):
        logger.info("Last couroutine done in: %i", 3 - i)
        await asyncio.sleep(1)
    logger.info("Last coroutine done")


def on_exit(loop: asyncio.AbstractEventLoop, thread: threading.Thread = None):
    logger.info("Detected houdini exit event")

    future = asyncio.run_coroutine_threadsafe(async_last_coroutine(), loop)
    future.result()

    future = asyncio.run_coroutine_threadsafe(async_shutdown_tasks(loop), loop)
    future.result()

    logger.info("Stopping event loop...")
    loop.call_soon_threadsafe(loop.stop)

    logger.info("Closing event loop...")
    loop.call_soon_threadsafe(loop.close)

    if thread:
        thread.join()
        logger.info("Joined thread: %s", thread.native_id)


def _start_loop_target(loop: asyncio.AbstractEventLoop):
    asyncio.set_event_loop(loop)
    loop.run_forever()

def main():
    loop = get_event_loop()
    asyncio.set_event_loop(loop)

    # No need to start the event loop in Houdini?
    thread = None
    if not loop.is_running():
        if CREATE_THREAD:
            thread = threading.Thread(target=_start_loop_target, args=(loop,))
            thread.start()
            logger.info("Starting event loop: %s in thread: %s", loop, thread.native_id)
        else:
            logger.info("Starting event loop: %s", loop)
            loop.run_forever()

    logger.info("Scheduling main coroutines...")
    asyncio.run_coroutine_threadsafe(async_start_heartbeat(), loop)
    asyncio.run_coroutine_threadsafe(async_use_async_generator(), loop)

    atexit.register(on_exit, loop, thread)
    logger.info("Registered atexit callback")

if __name__ == "__main__":
    main()
User Avatar
Member
4 posts
Joined: March 2017
Offline
Hi Paul,

Thank you! Your advice helps a lot!
User Avatar
Member
73 posts
Joined:
Offline
@paulgolter, absolutely great info, thanks so much!
  • Quick Links