FVG conducts independent research and development (IRAD) on technical computing challenges. Current focus is automation of CGI (generally VFX, modeling and animation). Our first project was to seamlessly connect Wolfram Research Mathematica (the Wolfram Language) to Houdini Engine, which we call the AFX engine. This provides a large number of technical computing capabilities to apply to VFX, animation and modeling. Here are AFX technical specs:

AFX is 20,000 lines of C and C++ code, seamlessly extends Wolfram Language by 260 new Houdini Engine primitive functions for VFX, Modeling and animation, all functions are completely thread safe and multiple AFX engines may be started and used simultaneously in or on multiple local or remote Mathematica sub-kernels. An effectively indefinite number of Houdini Engines can be started and used in parallel (based on licenses). AFX works flawlessly on all Windows 10, Mac OS and Linux platforms. AFX reduces Houdini Engine code application development by 93% from equivalent code written in C or C++ (this metric based upon recent tests where roughly seven lines of Wolfram Language AFX code consistently equates to 100 lines of C and C++ Houdini Engine code). Somewhat less for Python but we have no specific comparisons at this point, notwithstanding the full Wolfram Language is brought to bear. AFX has Wolfram Language documentation including every function having runtime documentation using “?" operator.

Now, we are moving to apply the AFX system to a problem of high difficulty in VFX, animation and modeling. Our initial thought process on this is to automate modeling with curves instead of with polygons or Subdivs. Currently we are looking at automatically creating an accurate 3D deformable model from a single image using algorithmic advanced image processing. Note: we exclude use of machine learning image training exemplars as others have done.

I am interested in suggestions for other challenges in VFX, modeling and animation. Thanks for your consideration.

VR

Lars Wood

awesome

Here is a technical white paper discussing the AFX research and development and why Wolfram Research Mathematica and SideFx Houdini were selected for the project over other technical computing and 3D platforms: http://fvg.com/objective.html [fvg.com]

This is fully generated by Mathematica AFX driving Houdini and Houdini Engine as a virtual machine http://fvg.com/validation.html [fvg.com] . The coordinate plot is the final input to the AFX NURBS curve generation process. When that is done the curves will be animated using AFX to sing to the audio track shown, with realistic head, facial and eye movement. I will introduce a slight breeze AFX physical effect so that the hair of the NURBS 2D character flows in the breeze. A suitable background will be created and AFX will then render the animation into a video with soundtrack. All algorithmically. This process will be completely automated running from Mathematica with Houdini virtual machine and could be deployed as a mobile app, web service, or as a node that takes an image as input. Next with AFX the 2D curve is convolved to 3D surfaces. I am no artist yet using AFX algorithmic prosthetics I can create art.

VR

Lars Wood

FVGDOTCOM
I am no artist yet using AFX algorithmic prosthetics I can create art.

That has nothing to do with art.

digitallysane: Art is in the eye of the beholder. Art is the expression or application of human creative skill and imagination. This is my form of art. Who appointed you to be the litmus for what is art or not art. Your words are intolerant toward those who hold different opinions from yourself. My USA partners are pleased with the results. There is still a lot of work to be done and I will post updates of my work just as anyone else is free to do here. In the US what constitutes art (free speech) is protected by our constitution fortunately.

The word “intolerant” is getting abused on this forum recently.

I followed the link in your post. There I found a bunch of photos and the results of some image processing / geometry generation applied on them. That is not art, is an example of an algorithm.

If I apply a Sobel filter on a folder of images using a batch process like ImageMagick, all I get is some filtered images, not art. Yeah, in the 80ies the result of some filtering operations was considered “digital art”, but we're in 2016 nowadays.

Instead of trying to utter such nonsense as “I'm an artist even if I'm not an artist”, maybe you should simply try to find some artists to work with you on your tools. It's painfully obvious from this thread and other posts of yours that, while your tech knowledge might be amazing and even innovative, you have absolutely no idea on how these tools are used to obtain an artistic result, which sometimes (not that often, unfortunately), can even be called art.

FVGDOTCOM
In the US what constitutes art (free speech) is protected by our constitution fortunately.

Yeah, so I'm free to say that what you post on that page is not art. Go figure.

digitallysane: No need to be verbally aggressive. As I wrote on the page, my process is part of the automated composition toward the goal of synthetic generation of NURBS curves and surfaces and consequent animation. As you said, this is 2016 and I am developing, experimenting with and applying new technology and methods to create art. Everything has a beginning (and an ending). I am starting with large scale NURBS curves and surfaces because that technology is the most appropriate technology to algorithmically exploit.

Established technologies have built in defenses against technologies that are disruptive to their status quo. This is why when Alexander Graham Bell invented the telephone, Western Union (telegraph technology) said this is not communication technology, since it was not telegraph based. It was necessary for Bell to find a niche, which in the case of the telephone was inter office communication within the same building, which telegraph was not well suited for. Using this beach head Bell slowly displaced the telegraph for long distance communications, including purchasing Western Union when the telegraph company was failing.

There is always push back from the established technology. You are simply pushing back, yet realize, we have just begun.

Note: Edwin Catmull is a computer scientist and president Pixar Animation Studios and Walt Disney Animation Studios. Go figure.

digitallysane
The word “intolerant” is getting abused on this forum recently.

That is not art, is an example of an algorithm.

art.

All CGI (VFX, modeling, animation, everything) are algorithms. Mine are just different algorithms applied to address CGI challenges.

In computer science, Houdini is classified as an example of a diagrammatic dataflow programming language. Therefore anything developed using Houdini is a program. All programs are algorithms. Ergo all Houdini modeling, VFX and animations are algorithms. Q.E.D.

Cool stuffs! Can expand on the virtual machines part; pros/cons to being able to do that?

Artye
Cool stuffs! Can expand on the virtual machines part; pros/cons to being able to do that?

Artye: Thanks! The importance of using Houdini as a virtual machine is rapid development and distribution of complex content. The AFX technology makes it so there are now three seamless and equal components to the SideFx Houdini platform. Think of the AFX technology as a triangle where the vertices of the triangle are Mathematica, Houdini and Houdini Engine. One can move dynamically and fluidly from Wolfram Research Mathematica to Houdini and to Houdini Engine based upon what you are trying to accomplish because the Mathematica environment contains an algorithmic representation of everything you see in Houdini and Houdini Engine (Houdini engine in the form of Houdini Digital Assets). Also, Mathematica connects directly to unreal engine and unity so you get that Houdini integration too since work you do using unreal or unity in Mathematica shows up in Houdini/Houdini Engine as AFX nodes.

Also and very important, from a technical director perspective, the frame work for many large complex shots are rapidly developed using the power of Mathematica's Wolfram Language. These complex shots are then distributed to less technical artists in the form of Houdini digital assets and Hip files. This aspect has resulted in ease of use and workflow performance beyond my expectations.

As an example, lets say you want to introduce strong artificial intelligence or complex computational fluid dynamics (or something even more technically ambitious) into Houdini Model creation, simulations, Mantra/Renderman rendering and so forth. You can now use the powerful Mathematica machine learning capabilities and mathematical functions to create these “intelligent” models, animations and visual effects rapidly algorithmically. Now since everything you do in Mathematica is reflected in Houdini and Houdini Engine digital assets, when you open up Houdini or a digital asset, there are new nodes connected to the procedural networks, which you can manipulate graphically, interacting back to Mathematica, which now runs as one or more distributed sub-kernels to Houdini. Houdini and Houdini Engine are two new and separate interfaces to the Mathematica Kernel and its Notebooks and vice versa.

Therefore, if you add functionality to the procedural network in Houdini or Houdini Engine, this shows up in Mathematica as Wolfram Language constructs, which you can choose to manipulate in Mathematica. What's really cool is that as a technical director (at least I think so having not been a technical director but understanding what they do), you can work with one or more non-technical artists in real time on their work by starting a Mathematica Wolfram Language user interface and connect to their Houdini session and interactively seeing and collaboratively manipulating what the artist sees algorithmically from Mathematica. Since Mathematica and the Wolfram Language are also cloud based, this can all be done remotely by connecting to the Wolfram cloud too, for example.

FVG first started the development of AFX since my passion is CGI modeling, animation and visual effects and so this was on my life bucket list to do (I usually work on things where CGI is secondary and not the primary motivation such as smart molecule design and development, see http://suprachem.com [suprachem.com] and this example of a smart molecule here http://fvg.com/drug-discovery.html [fvg.com] ). I first had the idea to do this working on very large scale integration integrated electronic circuit design (EDA see here http://fvg.com/vfx-synthesis.html [fvg.com] ). My VLSI work (and how industry does VLSI today) allows developing and interacting with a circuit design or simulation from either the algorithmic or the visual perspective, which enables optimal workflow. That is what my goal for AFX is and its working out to do this. Now we are applying AFX to some technical demonstrations.

VR

Lars Wood

I'm getting some Time Cube vibes here.

DaJuice; nonsense.

More here: http://fvg.com/cancer-mechanism.html [fvg.com] .

VR

Lars Wood

Re: the automated CGI part. It's a very interesting problem - i.e. a human is made of ~100 trillion cells, whilst your cgi representation would aim for what level of representation & complexity? Maybe your solution could ‘grow’ from some cells or could be ‘chiselled’ down from a block of clay until the object is complete, or, there could be an amorphous avatar that then is modified/weathered with attributes and afflictions gleaned from mathematica.

P.S. don't worry about the criticism - visual artists tend to do this a lot instead of learning things that truly are a challenge to themselves Why learn when you can attack!

Artye; You can tell the pioneers since they are the ones with the arrows sticking out of their backs: http://fvg.com/tenacity.html [fvg.com]

I am now implementing the final computational geometry algorithm to extract the curves from the 2D plotted image here (the blue on white plot of points): http://fvg.com/validation.html [fvg.com] . This will result in tens of thousands of sorted NURBS curve control points, which will create a deformable 2D model in Houdini.

Next step is to turn the curves into a differentiable manifold (a continuous surface) using more sophisticated computational geometry. This will involve tensor calculus. What I am working towards is a deformable 3D NURBS surface character model, which is generated from a single image. Once I have the NURBS model I will run some experiments in Houdini to see how easy it is to use. The technology can also create polygonal objects too. This approach is not limited to faces. Any captured image, (either raw or edited) or a bunch of images drawn by an artist could be used as input. An artist could sketch the characters and all the other assets in the animation on a watcom pad for example into photoshop or some other capture technology. The resulting images (jpegs work fine even with their lossy nature) would then be processed by the AFX algorithms into the content, ready to be fine tuned and animated. The idea as in hardware logic synthesis is to allow the designer/artist to focus on the content instead of the mechanics to get to content. If this works out it may be offered as a service on the internet where people submit 2D raw material and get back 2D/3D content assets.

Woooh Hooo!! It freaking worked!!! http://fvg.com/validation.html [fvg.com] The last image is the 2D NURBS result, which will now be turned into a Houdini asset that can be deformed and animated. It is composed of 5657 automatically generated NURBS curves with a total of 210,447 automatically generated control points. The color coding represents the individual NURBS curves. This is a completely automated process. Now I have to see how well it animates. Once I work the kinks out of the animation process and figure out what the optimal number of control points are I will feed that back into the algorithm so the next set of curves will be optimized that go into Houdini. Then I will move on to turning the 2D system into a 3D NURBS surface. Link to the screen shot below:

The NURBS curve 2D model plotted in Mathematica using Mathematica default line plot scaling, which incorrectly stretches the image.



The NURBS curve 2D model loaded into Houdini showing some of the NURBS curve network nodes (ortho). Scaled perfectly to the reference image.



Closeup of the NURBS curve 2D model loaded into Houdini (ortho). Scaled perfectly to the reference image.



Rendering the NURBS curve model is instantaneous with Mantra.



Houdini character and ocean animation currently in development (rendering using Mantra).

Thanks for the posts. What are all those nodes in the Network view? Are they separate processes from Mathematica.

Is that 5657 object nodes for representing each NURBS curve?

PS. I wonder how different the Trace SOP [sidefx.com] result would compare.

edward
Is that 5657 object nodes for representing each NURBS curve?

PS. I wonder how different the Trace SOP [sidefx.com] result would compare.

Sorry for not getting back sooner, long night and I crashed for the day.

Each object is a NURBS curve. The network structure is currently flat since I just wanted to see how it would look in Houdini. I will run the reference image against the trace SOP to to compare. Working on comparison with the trace SOP using the reference image now.

trace SOP

Using just the trace sop the initial result looses most of the detail (two spots for eyes otherwise no face, most of the hair missing, only edges) and the curves are impossible. basically does not work on the reference image. Before I post the trace result I will add a cop. Using the cop it looks like much more work than the five lines of AFX mathematica code I wrote, which is automatic. Alternatively, automatic AFX output can be used as cop input, producing high definition results with the trace but its non starter for the curves.

Notwithstanding, the effect I was striving for is exactly as I directed AFX to achieve. I did not want a mirror image of the original reference image. I wanted to achieve something that was based upon a deformable curve artistic interpretation of the reference image and that goal was achieved. The animation I have in mind will have the hair of the model moving in a breeze while it lip-syncs the audio track that I have selected: http://fvg.com/validation.html. [fvg.com] Lots of other AFX applications though for sure.