Sorry if this sounds weird, I copy pasted it from asking someone else on twitter :

hey I have a question about HEngine that I can't find the answer to

I have multiple HDAs in a level, and I want them to generate data in multiple "passes", and after each passes each HDA outputs data that might be relevant to other HDAs for the next pass
I would like those HDAs to not know about eachother, only about their inputs and outputs, which would be standardized data types (ie multiple HDAs will generate or take roads as input, but roads are all splines with the same data in the same format)
I want to be able to pop a new HDA into a level and have as little setup as possible to integrate it into that pipeline, and absolutely no modifying how the other HDAs actually work. Is that even possible with the stock HEngine plugin right now?

Here's possibly a better hypothetical example :
I have currently 2 HDAs in the scene, a landscape HDA which generates the landscape in multiple passes (initial landscape, stamping roads and other arbitrary stamps in, erosion, stamping some post erosion stamps in)
I also have a road HDA, which after pass 1 of the terrain, reads its height and projects down a spline onto it
Some HDA, either the road HDA or some other HDA (road merger or something) takes all the roads and merge them where they intersect. It then generates a road stamp used to smooth the errain around the road.
Now, I've made a city HDA which generates buildings, roads, and a terrain stamp to flatten the terrain under the buldings
I want to be able to pop that into a scene, and have the terrain and road merger HDA automatically be aware of the road and stamps generated by the city HDA (but not the HDA itself) and take them into account for road merging, stamping, etc like they already did eachother

If any of you got a good answer, or could direct me towards someone who would (especially if it's someone working at houdini or something) I would appreciate it

Sounds like PDG is the solution here. PDG can act as a manager for your HDAs, knowing which data to pass around, and when things may need to be recooked. PDG can manage arbitrary number of user inputs, so scaling it up or down is super easy. And it can automate so much of the work or managing existing HDAs.

There are many example files for PDG on roads and terrain which should help. https://www.sidefx.com/docs/houdini/tops/examples.html#roadbuilder [www.sidefx.com]

Houdini Engine has a slick integration of PDG as well, so while you might need to rethink your world tool in the context of PDG, all your HDAs and Unreal Engine workflows should more or less be the same.

CodyS
Sounds like PDG is the solution here. PDG can act as a manager for your HDAs, knowing which data to pass around, and when things may need to be recooked. PDG can manage arbitrary number of user inputs, so scaling it up or down is super easy. And it can automate so much of the work or managing existing HDAs.

There are many example files for PDG on roads and terrain which should help. https://www.sidefx.com/docs/houdini/tops/examples.html#roadbuilder [www.sidefx.com]

Houdini Engine has a slick integration of PDG as well, so while you might need to rethink your world tool in the context of PDG, all your HDAs and Unreal Engine workflows should more or less be the same.

I saw PDG when researching the topic and it definitely seems like it would be part of the solution, but at the same time most examples I've seen either had any HDAs it needed to process be setup internally in the top network, or other HDAs in the scene only ever acted on it as input, and never received data back from the HDA with the PDG,let alone did multiple back & forth in a row like I've described
Is that something PDG can actually do in HEngine? integrate external HDAs that were just dropped into the scene into its graph and control what data they generate and when?

Yeah, you are pretty much spot on. You would need to embed the HDA processors in the HDA(this is not editable in Unreal). And there would have to be some level of deconstruction to ensure your pipeline works inside the PDG network. So if you had a process of terrain>road>terrain>road, this could be done with PDG, but each trip would need to be its own unique process(either through HDA, or edited directly in the TOP network). As PDG flows linear, and can't jump back up in the process.

To answer your one question, you couldn't just plug in a building gen HDA or a road gen HDA and have PDG automatically know what to do. You would build this inside PDG as a building gen hda processor, and road gen HDA processor. And its inputs could be other HDAs, but it would be the output of those HDAs from my understanding.

And you can pass information as attributes in your PDG network! But if you had a road HDA with an attribute @lane_count = 6. This attribute will not be processed because the output will be a bgeo file that is passed to your next node, and the @lane_count won't transfer. But you can create and pass these attributes in your PDG network, to inform the next HDA it's intended parameters, being the bridge between these HDAs.



I guess the high level idea is thinking of it like this. There are kind of two main methods of working with HDAs without PDG.

1.You either build all your HDAs individually, (between 6-12) and drop each one down in your world, (creating 100s of HDA actors). These HDAs have no communication, and each need to be managed by hand. ie, Changing X parameter for certain conditions, for X number of HDA actors. And any updates/changes to the HDA need to be managed by hand on a case by case basic.

2.You make one massive HDA, which has those 6-12 HDAs embedded, and you edit these all directly in sops. Full communication! But processing time is slow. One curve move will requires a massive recompile for every dependency. Move a road over 5 feet, every road needs to be rebuild. Then every street light, car placement etc. On top of that, changes to a level can only really be done by one person.


PDG sort of brings the best of both worlds.
1. You connect all of these in a TOP network, as if it was 1 massive HDA network providing full HDA communication.
2. All user edits/parm changes can be handled by your PDG network, automating tedious management in your level.
3. Performance is better. If you move this 1 road curve, PDG identifies what will need to be changed, and what doesn't. And procedurally recook only specific items, speeding up processing time.
4. And this allows for multiple users to edit inputs. And HDA edits to happen independently.

If an HDA is a procedural 3d artist. PDG is a procedural world builder(in this context). Did I sell you on it yet?! hahaha

Here is a fun little tutorial that walks through a PDG network from front to back.
https://www.sidefx.com/tutorials/foundations-build-a-city-with-pdg/ [www.sidefx.com]

CodyS
Yeah, you are pretty much spot on. You would need to embed the HDA processors in the HDA(this is not editable in Unreal). And there would have to be some level of deconstruction to ensure your pipeline works inside the PDG network. So if you had a process of terrain>road>terrain>road, this could be done with PDG, but each trip would need to be its own unique process(either through HDA, or edited directly in the TOP network). As PDG flows linear, and can't jump back up in the process.

To answer your one question, you couldn't just plug in a building gen HDA or a road gen HDA and have PDG automatically know what to do. You would build this inside PDG as a building gen hda processor, and road gen HDA processor. And its inputs could be other HDAs, but it would be the output of those HDAs from my understanding.

And you can pass information as attributes in your PDG network! But if you had a road HDA with an attribute @lane_count = 6. This attribute will not be processed because the output will be a bgeo file that is passed to your next node, and the @lane_count won't transfer. But you can create and pass these attributes in your PDG network, to inform the next HDA it's intended parameters, being the bridge between these HDAs.



I guess the high level idea is thinking of it like this. There are kind of two main methods of working with HDAs without PDG.

1.You either build all your HDAs individually, (between 6-12) and drop each one down in your world, (creating 100s of HDA actors). These HDAs have no communication, and each need to be managed by hand. ie, Changing X parameter for certain conditions, for X number of HDA actors. And any updates/changes to the HDA need to be managed by hand on a case by case basic.

2.You make one massive HDA, which has those 6-12 HDAs embedded, and you edit these all directly in sops. Full communication! But processing time is slow. One curve move will requires a massive recompile for every dependency. Move a road over 5 feet, every road needs to be rebuild. Then every street light, car placement etc. On top of that, changes to a level can only really be done by one person.


PDG sort of brings the best of both worlds.
1. You connect all of these in a TOP network, as if it was 1 massive HDA network providing full HDA communication.
2. All user edits/parm changes can be handled by your PDG network, automating tedious management in your level.
3. Performance is better. If you move this 1 road curve, PDG identifies what will need to be changed, and what doesn't. And procedurally recook only specific items, speeding up processing time.
4. And this allows for multiple users to edit inputs. And HDA edits to happen independently.

If an HDA is a procedural 3d artist. PDG is a procedural world builder(in this context). Did I sell you on it yet?! hahaha

Here is a fun little tutorial that walks through a PDG network from front to back.
https://www.sidefx.com/tutorials/foundations-build-a-city-with-pdg/ [www.sidefx.com]

I think I get it now, and while I'm probably gonna look into it further if only because it's cool, it's probably not what I need

The main goal here for me was to be able to make new types of assets and have full control over where they are in the world and what they generate, while also having intercommunication between HDAs where relevant
The problem with PDG here is that every time I'd create a new HDA I'd have to edit the PDG graph to integrate it, expose a ton of parameters in the graph to be able to have as much control as I need (basically all the HDA params + world position and rotation + external inputs), and even then I'd probably still have to split that HDA into several HDAs, one for each "passes"

I'm guessing what I was going for just isn't possible right now? which I guess is fine, it's probably a very niche workflow if it exists at all yet

afaik unreal's PCG allows for this kind of intercommunication between PCG actors or whatever they're called (ie PCG actors generating their data in multiple steps, with each step generating some data that's then globally accessible for all other PCG actors for the next step), hopefully that gets some degree of compatibility with houdini eventually

I went down this route a couple of years back, trying to figure out how to communicate between HDAs inside UE - and ended up going with the one monolithic HDA with everything in it. (In the context of game world generation)

I had one gedankenexperiment I did not follow, but thought I could share for the lulz.

All the HDAs live inside a common session in Houdini (HARS). If you have two HDAs called HDA_Big and HDA_Small, in a fresh HARS session you can find them at /obj/HDA_Big1 and /obj/HDA_Small1, and from there you can communicate.
This naïve case of cause has a problem if you have several instances of the same HDA.

I think you could have an identifier string as a parameter on each HDA, and then you HDAs could use some python to find each other and form references for communication. These GUID parameters could be automatically set by any orchestrator-type script/BP/HDA when setting things up, or by dirty human hands in a simpler setup.
Dealing with asset execution order might make things "interesting".

I can not recommend this, but I would be interested in knowing if someone sets something like this up

eetu.