Hi,

I am currently setting up a 3D Scan Optimization Workflow, and I was wondering if there maybe is some kind of ONNX Model for inferring a roughness (and metalness) map from the diffuse und height map. I am also wondering how a program like "Materialize" or "3D Sampler" work.

OR

Any tips on how you create roughness maps for 3D Scans?

Any input is welcome!


Cheers,
Josh

This sort of thing usually doesn't require a neural network, just multiple pictures. Whether you'd actually get better results or speed things up for the amount of time you'd have to dump into training one is beyond me. Usually most of the simpler maps like roughness can be computed for most materials by just comparing shots from different angles. I've seen several techniques ranging from multiple shots lit by differently polarized flashes from different angles to the snapshots from multiple angles technique substance capture uses (which may use some type of neural net but probably not considering there's a javascript program that'll do roughly the same thing but maxes out at 4 images) to this neat paper which only requires one cell phone shot without flash and one with:

https://dl.acm.org/doi/pdf/10.1145/2766967 [dl.acm.org]

This page https://cpetry.github.io/NormalMap-Online/ [cpetry.github.io] can generate normals, displacement, ambient occlusion, and specular from either 4 images taken of the same spot but lit from 4 angles, or from a single height map. The site is downloadable from github if you want to peruse the code https://github.com/cpetry/NormalMap-Online [github.com]

Here's a single-photo neural net repository that's probably closer to what you're looking for, but the page with model downloads appears dead so you might have to look for it on archive.org
https://github.com/msraig/InexactSA?tab=readme-ov-file [github.com]

That's an older tensorflow model, but it can likely be converted into an ONNX model with the proper tools. Tensorflow is basically abandoned for GPU on Windows.