edward
I'm still trying to understand how using HDAs with embedded Instance objects (or embedded objects that contain packed primitives) are “unfriendly” or “impractical”.

Because even if the objects are contained within HDAs, I still need to place hundreds-of-thousands of objects. 20 unique buildings is easily hundreds-of-thousands of placements. Even if a building is converted to a single mesh, it exists somewhere as the full thing because I might want to destroy it using physics simulation at some point, which means it would be animated. That means I still need to manually place the objects when making it in the first place, and every time I place an object, finding the exact ‘Copy and Transform’ node to increment out of ~500 of them is hugely unfriendly and impractical.

I have already stated I don't think Houdini is capable of having the workflow I want because nodes don't scale well at all when doing a lot of manual work, they are suited to procedural work.

20 unique buildings means at most 20 placements when they're HDAs. Animating can happen at the geometry level (procedurally or non-procedurally). I get the feeling that you don't really understand Houdini workflows. Or perhaps you're not fully realizing all the parts that can still be procedural in Houdini while giving you the level of control that you need.

From everything that you've mentioned that you want to do, there's no need to do rewiring of nodes at all when placing down new objects. I recommend watching the Ghost Recon Wildlands GDC presentation to see some of the other possibilities on how to set things up in Houdini for large game levels.

…I have already stated I don't think Houdini is capable of having the workflow I want…

Of course Houdini isn't capable of doing what you want. Neither is any other software out there.

The reason being is because you simply want an enormous amount of objects with most or not all of them having the capacity to easily manipulate quickly to your liking, individually and uniquely, which means you have to do it manually for many different objects and/or sub-components of those objects.

… if you consider a house is made of ~8000 objects…. but there are a lot of things that can't be merged… For example, a humanoid is made of ~50 objects that are animated separately. 100 characters on screen is ~5000 objects. Then consider trees that can sway where each branch might be a separate object and so on. It all adds up very quickly….

If it weren't for the fact that you want to retain the capacity to manipulate uniquely many different objects you could simply use the copy and stamp or instancing nodes and use scripts to drive all those objects that can change each and every aspect of those objects with an algorithm that follows a pattern.

But I suspect this still would not be good enough for your intentions. What if you want to change some of those objects out of those pattern of changes seperately - you have to get in there to make the changes manually.

This is the kind of thinking I keep seeing from you. And that's not criticising, I think your simply ambitious.

Even if you could hire some software developers in which you think they are going to give you what you want, they won't be able to do it; Not with the current state of technology.

Maybe at some point with more development and access to AI technology, you would be able to get what you want:

In essence basically something that reads your mind as you look at your screen, upgrading and changing what you want without the need to manually pick out an object/s and using mouse/keyboard make the changes.

BabaJ
The reason being is because you simply want an enormous amount of objects with most or not all of them having the capacity to easily manipulate quickly to your liking, individually and uniquely, which means you have to do it manually for many different objects and/or sub-components of those objects.

I would challenge this. It's just as easy to transform sub-components at the SOP level (eg. with appropriately assigned ‘name’ attribute) on an as needed basis. I'm not seeing the dichotomy echoed in this thread where procedural means giving up on manually specifying things.

I would challenge this. It's just as easy to transform sub-components at the SOP level (eg. with appropriately assigned ‘name’ attribute) on an as needed basis. I'm not seeing the dichotomy echoed in this thread where procedural means giving up on manually specifying things.

I would agree with you.

However I was writting in the context of what I am assuming is the OPs intent.

And I definitely could be assuming wrong.

But I suspect what your saying could still be ‘cumbersome’ for the OP to do what they want.

I suggested earlier that he should maybe create a scaled down version of what he potentialy would like to do in a hip file and post it.

I think from there, better suggestions and solutions or their limitations could become more apparent.

Hi,
I second the request for a scaled down version of what he wants to do!
I am not sure i do 100% get the definition of task at hand.

I am dealing with loads of objects all as packed geo. I love it. Basically i only move points around, and when i want to edit something i unpack it, edit it and pack it again. great!

And its almost like working with (plm)xml in a definition when you reference packed geometry from disc. Having Structure and placement separated from geometry is great.

kind regards

Olaf

As requested. Instead of there being 3 ‘File’ nodes and ‘Copy and Transform’ nodes, there would be ~500. In the ‘File’ nodes the geometry is loaded as packed disk primitives.

To be clear, the placement of the geometry is manual using the edit node, deletions are done using the blast node.

The problem at hand is locating an individual ‘Copy and Transform’ node every time a copy is required is hard, this thread is the search for a faster way within Houdini.

There's a couple ways I can think of.

One is to think of a naming convention for your file/copy nodes together with a layout pattern that symbolically represents your scene.

That way you can look through your network quickly to find what you want.

This is something I have done for my own scenes and find it works well.

The other is, there should be a way to script( with python ) that if you select on object in your scene, that the corresponding copy node in your network becomes highlighted and zoomed in to the center of your network view.

I don't know what the python code would be for that or if it has that capacity.( I have a hunch it is possible, just need someone to chime in here who has more experience using Python with Houdini than I )

If not, you might have to dig into the HDK and develop that capacity there.( which I am sure you could through that )

Of course that's assuming you would like to do that ( the second option I mentioned ).

I haven't read this whole thread, so I apologize if I'm repeating someone else or missing something important…

You could make a digital asset with a multiparm on it, where each instance of the multiparm specifies a file name (and any other per-instance parms you may want to set on the copy SOP). Inside the HDA is a file/copy node pair inside a for loop. Each iteration through the loop pulls a different multiparm value from the parent asset.

This way you can pull in as many files as you want with a single top level node. Each file is one parm on the node. I've been told that it can be challenging getting the dependencies set up properly (to force a recook when the value of a single multiparm instance changes), but I know a similar technique has been made to work in other scenarios.

BabaJ
The other is, there should be a way to script( with python ) that if you select on object in your scene, that the corresponding copy node in your network becomes highlighted and zoomed in to the center of your network view.

This is the best suggestion so far.

Maybe even better, when a particular instance is clicked, Houdini would find out which ‘Copy and Transform’ node it is associated with, then when a hot-key is pressed, increment that node without even selecting it.

Edit: One problem with that is the copy would be wherever the original was placed which might be very far away.

I really don't think doing it at the SOP level is necessary for the task at hand. I've created just a simple asset in the attached .hip to show the possibilities along with a rough a video capture of the workflow.

edward
I really don't think doing it at the SOP level is necessary for the task at hand. I've created just a simple asset in the attached .hip to show the possibilities along with a rough a video capture of the workflow.

My method, as cumbersome as it is, enables the placement of ~400,000 instances while still getting around 30FPS in the viewport, where each instance can be manually placed individually.

I just tested your method. I set each asset to 1 copy so each one can be manually placed individually. The viewport could handle ~2000 instances while still getting around 30FPS.

That is why it is necessary to do this on the geometry level. Packed disk primitives do not work across different objects.

Each “asset” node is meant to represent ONE of your ~500 “file” nodes, ie. one input into the ‘merge1’ node of your example. So it's doing the exact same amount of instancing as in your example except without needing to wire each file into a merge. The Copy and Transform inside is the one that creates instances of the file.

Conversely, your test of using 2000 “asset” nodes is equivalent to your method of having 2000 “file” nodes. So if you want a fair comparison using your method, you need to do it your way with 2000 File plus Copy&Transform nodes wired into a merge.

I see what you mean. So to effectively select the ‘Copy and Transform’ node, just one of the instances would need to be clicked-on at the scene level, which would select the whole HDA which has the copy parameters in it.

This does suffer the new copy being created wherever the original was placed. Having to travel a few hundred metres if the scene is big to get the new copy, and then go back to place it would start to add up very quickly.

This is why creating a Python plugin might be the best solution. When clicking on an instance, the transform of that instance would then be known, so the new copy could be moved by the plugin to that position.

How that works depends how it holds up to my requirements though, and just a note, Unreal Engine 4 cannot handle these and I don't think any package can out-of-the-box:

1. Quick manual placement of instances.
   
2. Transform animation of instances, which would be translation, rotation, and scaling.
   
3. Rigid-body transform simulation of instances, again translation, rotation, and scaling.
   
4. Grouping of instances.
   
5. Hiding of groups of instances.
   
6. Copying of groups of instances.
   
7. Conversion of groups of instances to a single mesh.
   
8. Attachment of instances to skeletons, and animation of skeletons.
   

Houdini might have trouble with 1, 3, 6, and maybe 8, but I haven't looked into simulation or skeletons yet.

KalciferKandari
This does suffer the new copy being created wherever the original was placed. Having to travel a few hundred metres if the scene is big to get the new copy, and then go back to place it would start to add up very quickly.

You can customize this step however you want. I just happened to set the default to 2 copies when I made the asset, but you can easily make the HDA not do that.

I'm not understanding your “travel few hundred metres” problem, probably because you've never explained what you were trying to accomplish with your need for the “Copy and Transform” SOP in the first place. The “Copy and Transform” SOP lets you create copies procedurally with a transform relative to the previous copy. So the assumption here is that your copies of the original pieces of geometry can be placed procedurally. If instead, you need to place each instance also manually, then I don't see how your contrived example method can handle that at all. If you really need to place each instance manually, then I would probably do it something along lines of this instead:

• Have a “source” object that contains all the file SOPs for creating all the unique geometry packed primitives. This object is NOT meant to be displayed nor rendered. It just serves as the central place to create your geometry instances.
  
• In the Asset HDA, use a “Source” path parameter instead of a “File” path parameter. This “Source” path parameter contains the path to the file SOPs in your “source” object. Inside the HDA, you just use an Object Merge SOP to reference in source geometry. Ensure that Object Merge's “Transform” parameter is set to “None”.
  
• Use a “collection” object to merge everything together as demonstrated in my instancePlace.hip scene file.
  

This allows one to easily create instances with all the niceties that goes along with placing viewport objects. I should mention that I didn't quite make the Asset HDA in my demo as nice as it could be like give it a handle right away as soon as it's created in the viewport.

KalciferKandari
How that works depends how it holds up to my requirements though, and just a note, Unreal Engine 4 cannot handle these and I don't think any package can out-of-the-box:

To me, requirement 1 follows if you have 2. ie. I think Houdini is fine for these two. For 3, this is Houdini's bread and butter. Many (most?) large scale destruction sequences in Hollywood films are done using Houdini these days. 4 is easy, 5 is easy, 6 is easy (see my comment regarding name attributes above), 7 is demonstrated by the “collection” object.

For 8, skeletons in Houdini are no more than bone objects arranged in a hierarchy, and bone objects are just mostly the same as regular objects except with some additional parameters for the skin deformer and IK solver. However, your description is a bit vague and largely hinges on how you want to animate said skeletons.

edward
If instead, you need to place each instance also manually

Yes I do.

edward
If you really need to place each instance manually, then I would probably do it something along lines of this instead:
Have a “source” object that contains all the file SOPs for creating all the unique geometry packed primitives. This object is NOT meant to be displayed nor rendered. It just serves as the central place to create your geometry instances.
In the Asset HDA, use a “Source” path parameter instead of a “File” path parameter. This “Source” path parameter contains the path to the file SOPs in your “source” object. Inside the HDA, you just use an Object Merge SOP to reference in source geometry. Ensure that Object Merge's “Transform” parameter is set to “None”.
Use a “collection” object to merge everything together as demonstrated in my instancePlace.hip scene file.
This allows one to easily create instances with all the niceties that goes along with placing viewport objects. I should mention that I didn't quite make the Asset HDA in my demo as nice as it could be like give it a handle right away as soon as it's created in the viewport.

Finding it quite difficult to recreate this description. Can't find a ‘Source’ path parameter, or any documentation on ‘Collection’ nodes, or is that just a normal ‘Geometry’ node?

edward
Finding it quite difficult to recreate this description. Can't find a ‘Source’ path parameter, or any documentation on ‘Collection’ nodes, or is that just a normal ‘Geometry’ node?

No, I meant that the “source” and “collection” objects are just regular objects that you create where you put your source geometry and collect them together (respectively).

I've attached an example of where I was going above. In my first example, the instancing of a geometry was still occurring within the object which works fine. However, in order for the viewport to instance the same geometry across multiple objects, then we need to use Instance objects for this. So the attached file has an Asset HDA which does this.

Again, I'd like to stress that manually placing EVERY instance is a bad idea in general. I suggest seriously thinking about how to do things procedurally to get yourself 90% of the way there and then spend the last 10% manually tweaking where things should go.

Unless I'm missing something, the way to create new instances using this method is to copy and paste an asset node on the scene level, right? If that is the case, the viewport can only handle about 4000, which means the instances don't work properly. The viewport should be able to handle 400,000.

Is this some kind of quest to find out which software can display the most cubes in the viewport ?

http://forums.cgsociety.org/showthread.php?f=2&t=1438973&page=1&pp=15 [forums.cgsociety.org]
http://forum.isotropix.com/viewtopic.php?f=5&t=2920&view=unread#p11839 [forum.isotropix.com]

And I wholeheartedly agree with edward. This make no sense whatsoever in a production environment.
To solve a complex problem you divide it into smaller problems until you can solve them efficiently.
That's why you build HDAs or setups that help you place these millions of instances. Still want more control? Have more granular rules / attributes that drive the placement or have some parts placed manually.
Hand-placing 400k objects though sounds pretty crazy.

It is still unclear what you are trying to do in the sense of the big picture. What is your project goal ?
Is it just a benchmark ? If so, for what purpose ?

Here's some information on the animation:

• It's a rigid-body world.
  
• There is a set of ~500 meshes, let's call them ‘pieces’.
  
• Pieces cannot crack or be broken into smaller pieces.
  
• Everything that is solid is made of pieces.
  
• Groups of connected pieces are called ‘builds’, such as a house.
  
• A build should be converted to a single mesh for performance, and turned back to pieces for animation or simulation, such as a house collapsing.
  
• Characters are made of pieces as well.
  

For the scenes to have any substantial size a very large amount of objects will be used, even when converting builds into single meshes, so instances are absolutely necessary. To use not use them would scale the scenes down by at least 2 orders-of-magnitude.