Bump Height, Displacement Map or Sculpting Details FROM a Normal Map?

[Infinite]

Bump Height, Displacement Map or Sculpting Details from Normal Map?

Does anyone know if this is possible? I have searched and search on Google and ZBrush forums.

Imagine it as reverse engineering. Is it possible to get sculpting details from a Tangent Space or Global Space normal map?

Imagine a photograph of someone, lighting them from different angles. Left, Right, Top and Bottom. Generating a Normal map from these different lighting conditions, using the RGB channel layers in Photoshop - http://zarria.net/nrmphoto/nrmphoto.html

The normal map is great to have but in real terms with regards to using those high resolution accurate bump details from the normal map they are effectively useless until render time, when the map is used. I'm trying to find out how to get access to that information.

The main goal to this is to find a way to get high detail bump or height/depth from a photograph, to use in conjuntion with Passive Stereo Photogrammetry.

Is there anyway to extract that height/depth information from those normal maps?

Relevant:

http://gl.ict.usc.edu/Research/ma08/

if anyone has any ideas it would be great to chat about it here.

[companioncube]

not sure if i understood right, but maybe this is what your looking for http://www.crazybump.com/

[Infinite]

not sure if i understood right, but maybe this is what your looking for http://www.crazybump.com/

Thanks AJ. I was using this yestarday it's a really great application but not quite what I am after.

[El Burritoh]

The normal map is great to have but in real terms with regards to using those high resolution accurate bump details from the normal map they are effectively useless until render time, when the map is used. I'm trying to find out how to get access to that information.

The main goal to this is to find a way to get high detail bump or height/depth from a photograph, to use in conjuntion with Passive Stereo Photogrammetry.




It sounds like you're wanting to show bump detail in realtime. If so, you could probably use a normal map to bake out bump detail to a grayscale image.

[Infinite]

It sounds like you're wanting to show bump detail in realtime. If so, you could probably use a normal map to bake out bump detail to a grayscale image.

Well yeah kind of. The end goal is to acquire true bump information from a photograph of a face for example. The current way I approach this is to use the HighpassFilter hack and use this as a displacement or bump image in ZBrush (which is pretty much what Crazybump does)

I just want to find a way to get access to true bump information.

I will try the bake idea it might work as a bump image :)

It must be impossible to convert normal map information to a displacement or be able to apply as a vector map to freeze the high resolution mesh and use that. It's just a trick on render time I guess.

EDIT*

No can't seem to get this to work.

There are some great test files from the UCT site - http://gl.ict.usc.edu/Research/FaceScanning/sample_face.zip

[MDB101]

Hey man, have yah tried vector displacement map? I gave it a shot in mental ray 3d max 2011. This gives a full blown true bump of your sculpt. Also gives self shadows, But does not calculate SSS. I think yah have to have your geo mesh to figure that out. But using VDM the right way can give that push. Imagine we can have some sorta normal map or vector map for specularity? LOL that would be cool to see!



I downloaded the script here

http://www.psychocore.com/

[gjpetch]

Pretty sure vector displacement won't work for this case, the colours in a normal map represent something quite different to the colours in a vector displacement map.
The nvidia normal map plugin for photoshop can convert normal maps to height maps. Not sure if that's any help?
The accuracy will not be high, but I think you could still get good results, since you'd only be extracting the details (isolated using a high pass filter) like pores and wrinkles, not the overall shape of the face.
Cool stuff, good luck with it man! I'd love to play around with that kind of stuff too; those Debevec papers are really inspiring.

[Infinite]

Pretty sure vector displacement won't work for this case, the colours in a normal map represent something quite different to the colours in a vector displacement map.
The nvidia normal map plugin for photoshop can convert normal maps to height maps. Not sure if that's any help?
The accuracy will not be high, but I think you could still get good results, since you'd only be extracting the details (isolated using a high pass filter) like pores and wrinkles, not the overall shape of the face.
Cool stuff, good luck with it man! I'd love to play around with that kind of stuff too; those Debevec papers are really inspiring.

Hey,

Yeah I have tried the Nvidia Normal Map plug-in but it freezes when you convert a normal map to height map. Crazybump works but it clearly just converts to a high pass filter hack like you can do in PS anyway, so either one is no good.

I wonder what Paul Debevec does in the those papers. Whether they are just applying normal maps on render time, or if they have some way to extrapolate the normal information and apply it to the mesh. I guess normal maps don't work that way at all.

Vector Displacement doesn't work.

It seems Normal Maps are only any good at render time,

[gjpetch]

I'm not familiar with the high pass filter hack method, out of curiosity, how do you do that?
If my memory serves then the nvidia plugin only works with square images, 512, 1024, 2048, whatever. And I think I've had problems using it on high resolution images before, so I've temporarily cropped images into quadrants, pushed those through the plugin, then stitched them back together.
I think the "Digital Emily" project used displacement maps based on normal maps acquired through Debevec's light stage.
http://gl.ict.usc.edu/Research/DigitalEmily/
Just taking a quick look, it says "a high-resolution version of the mesh is created and the vertices of each triangle are allowed to move forward and back until they best exhibit the same surface normals as the normal map."
So yeah, I guess it would be possible to write a script or simple tool that rotates faces or moves verts according to the normal maps, then you could extract displacement maps from that.

[Kel Solaar]

Pretty sure vector displacement won't work for this case, the colours in a normal map represent something quite different to the colours in a vector displacement map.

Exactly : Normal map describe the normal of the shaded point in tangent or wordspace at the UV coordinates whereas the vector displacement map represent the new shaded point position ( original point position + vector from the vector displacement map ) at the uv coordinates.

KS

[Infinite]

You can use the Highpass filter in Photoshop to generate a fake bump or displacement map, depending on how much use. You can then apply this fake bump in ZBrush or in a 3rdpart rendering application. It's great for certain things like pores wrinkles etc but it can't really figure out what is height and what is depth. This is pretty much what CrazyBump does when converting to Height Maps.


I think the "Digital Emily" project used displacement maps based on normal maps acquired through Debevec's light stage.
http://gl.ict.usc.edu/Research/DigitalEmily/
Just taking a quick look, it says "a high-resolution version of the mesh is created and the vertices of each triangle are allowed to move forward and back until they best exhibit the same surface normals as the normal map."
So yeah, I guess it would be possible to write a script or simple tool that rotates faces or moves verts according to the normal maps, then you could extract displacement maps from that.

Yes I am aware of that Emily project, as well as the other papers and notes on that site. These guys are incredible, very clever.

I wonder who I could contact to find out about such a script? This is the main goal of mine, to try and convert that normal map information into actual displaced geometry. Or even displacement maps.

It's hard to tell whether they use normal maps just at render time or if they are able to convert to displacement data.

[Infinite]

here's an example normal map from Paul Debevec's site. It doesn't work at all with the Nvidia plugin.

[Infinite]

Here's an Albedo diffuse map and another version with the Highpass filter applied.

[Kel Solaar]

I think it's because it's a worldspace normal map, from what I remember the Nvidia tool is taking tangent space ones no ( Used it years ago ) ?

KS

[Infinite]

I think it's because it's a worldspace normal map, from what I remember the Nvidia tool is taking tangent space ones no ( Used it years ago ) ?

KS

Yeah I think your right there Kel. Any ideas of how to extrapolate high res detail from a world space normal map? :drool:

[Kel Solaar]

Hehe nope :)

At the time I was searching for that I had good hope in Nvidia toy, but it never really worked for me. So currently I have no solution :rant:

KS

[gjpetch]

Yeah I think your right there Kel. Any ideas of how to extrapolate high res detail from a world space normal map? :drool:
Try running a high pass filter on the world space normal map. In this example I ran a high pass filter with a 6.1 radius, then I adjusted the levels on the blue channel (I moved the white point to 128, because the blue channel of normal maps is mostly white, rather than mostly grey).

You could definitely get better results than this if you experiment with different radius values in the high pass filter, and tweak the contrast a bit.

[gjpetch]

And a quick attempt at extracting a height map using the nvidia photoshop plugin. I had to paint out contrasty parts of the normal map before running the nvidia tool, otherwise the small details got overwhelmed by useless artifacts, and I had to run another high pass filter on the height map to counteract an overall gradient that the nvidia tool seems to produce.

[nghia59]

Hey Lee, the issue is not normal maps but the type of normal map that you are using. There are 3 types, world, tangent, and camera.

When you render a normal map from the nvidia filter it renders it using tangent to be placed on a 2D texture. That way it calculates it on the mesh and will update according to the position of the normals of a 3D model against its world space position.

What you are looking for is the camera based model. An example of this can be found in the mental ray ambient occlusion node by switching the type of output mode in the options. Use the amb occ node to be your diffuse on a surface shader (or constant in some packages) and render your model out with that.

I think the way Paul did it was that he's got a 3D scanner that will take samples from the entire object and spit out the data. Son in essence it's actually coming from a 3D object and not a flat image.

The cool thing about using camera based normals in rendering as a pass though is that you can use the color channels as a mask for lighting passes. After that is set up you can dial the hue in that map and relight your scene in 2D.

Hey BTW, you aren't Lee Perry from Ion Storm back in the day are you?

- Nghia

[nghia59]

Crazy bump is probably your best bet to extract 3D lighting data but there's a limitation to it's sampling thresholds as t sees things like the image outside of the object that you're trying to extract and bends the edges. The best it can do is relieve it like a coin.

2553 2554

[Infinite]

gjpetch: Thanks for your attempts. Believe it or not I have done all those tests. 100's of them and got exactly the same results. It's not really what I am after. Those methods are ok but again just hacks, on the 'face' of it they look like they work but they don't really. Some parts of the image are picked up to be height when they are depth and vise versa. Thanks for trying though, I appreciate your help.

But again we are back to square one. Once the normal map is created, I want to get access to the actual displaced information you see at render time. Which I think is impossible.

nghia59: Thanks, yeah I know about the different normal map types and their different functionalities.

When you render a normal map from the nvidia filter it renders it using tangent to be placed on a 2D texture. That way it calculates it on the mesh and will update according to the position of the normals of a 3D model against its world space position.

I didn't know that, thanks.

I think the way Paul did it was that he's got a 3D scanner that will take samples from the entire object and spit out the data. Son in essence it's actually coming from a 3D object and not a flat image.

The cool thing about using camera based normals in rendering as a pass though is that you can use the color channels as a mask for lighting passes. After that is set up you can dial the hue in that map and relight our scene in 2D.

See above and original post. I point to Paul's work and papers, as well as posted examples from his site.

Hey BTW, you aren't Lee Perry from Ion Storm back in the day are you?

No. :sw:

Crazy bump is probably your best bet to extract 3D lighting data but there's a limitation to it's sampling thresholds as t sees things like the image outside of the object that you're trying to extract and bends the edges. The best it can do is relieve it like a coin.

As I mentioned earlier, I have used Crazybump, and it's OK but again just runs cheap hacks. Not really what I am after.

Thanks for your comments again I really appreciate your time and help on this. Crazybump and the highpass filter hack are useful but not quite what I am after.

I have some access to a capture system that can lit an object from front, top, bottom, left, right and globally which is what I am designing and working with at the moment. But it's useless unless I can really utilize the generated normal maps.

See related work:

http://www.cgfeedback.com/cgfeedback/showthread.php?t=638


Re-lighting:

http://www.ir-ltd.net/uploads/Captures-01_flat.jpg

[Infinite]

Crazy bump is probably your best bet to extract 3D lighting data but there's a limitation to it's sampling thresholds as t sees things like the image outside of the object that you're trying to extract and bends the edges. The best it can do is relieve it like a coin.

2553 2554

And how do you think I can get access to that Normal Map information? I have already run these tests and got identical results.

The goal and purpose of this thread is to find a way to extract that information to actual mesh geometry. This is why I wonder what Paul and his team do? they must just use that information at render time only. But I have seen them extrapolate Displacement Maps from their captured data.

[gjpetch]

Infinite, pretty sure Debevec is using high pass filtering too. See this quote in the paper you linked to: "We extract the high frequency details of the estimated normals using high-pass filtering and then add these details to the smoothed geometric normals."
I guess the only difference is that he's isolating the specular from the diffuse using polarised filters, because specular gives more accurate normals than diffuse.

[Kel Solaar]

The key like Lee stated it many time is here :

We add in the skin texture detail by essentially embossing the specular normal map onto the 3D mesh.

And they do that like Lee quoted it :

By doing this, a high-resolution version of the mesh is created and the vertices of each triangle are allowed to move forward and back until they best exhibit the same surface normals as the normal map. Our lab first described this process on the web in some work involving Light Stage 2 back in 2001, though back then we were using normal maps built from the diffuse facial reflection observed in traditional light stage data. The result is a very high-resolution 3D scan, with different skins textures clearly observable in different areas of the face.

So basically the process is to get the normal map, then tessellate the surface until you get enough polys and then move those points along the original geometry normal until the normals of adjacent faces match the ones exhibited into the normal map.

It's not a basic 2d Process but an actual work on the high resolution mesh.

KS

[nghia59]

And how do you think I can get access to that Normal Map information? I have already run these tests and got identical results.

The goal and purpose of this thread is to find a way to extract that information to actual mesh geometry. This is why I wonder what Paul and his team do? they must just use that information at render time only. But I have seen them extrapolate Displacement Maps from their captured data.

Hey Lee, I think this might answer your question. Looking at your reference pics Paul might have used lighting passes like these...

http://zarria.net/nrmphoto/nrmphoto.html

If his scanner has a strobe on it while he's taking pictures it'll be a lot like these samples. Crap I had his email but can't seem to find the darn card. Now I'm going crazy looking for it. :banghead:

- Nghia

[Infinite]

Hey Lee, I think this might answer your question. Looking at your reference pics Paul might have used lighting passes like these...

http://zarria.net/nrmphoto/nrmphoto.html

If his scanner has a strobe on it while he's taking pictures it'll be a lot like these samples. Crap I had his email but can't seem to find the darn card. Now I'm going crazy looking for it. :banghead:

- Nghia

Thanks agains but I am already using this method. Roughly anyway. I am at this stage right now, my biggest problem is being able to auto warp/align the images together as there is some movement during the 6.7 second capture time for 6 images.

I even have my capture device setup with Circular Polarizers that are able to remove specularity from some of the lights, I just need to add linear polarizers now to each light source, oriented correctly.

The issue is.. extrapolating the high frequency data from the normal maps to apply to the low frequency captures :confused:

[Infinite]

Infinite, pretty sure Debevec is using high pass filtering too. See this quote in the paper you linked to: "We extract the high frequency details of the estimated normals using high-pass filtering and then add these details to the smoothed geometric normals."
I guess the only difference is that he's isolating the specular from the diffuse using polarised filters, because specular gives more accurate normals than diffuse.

It looks like adding a high pass filter to the specular normal map yields better results than when applied to an albedo colour map. That's interesting. Still a bit of a hack though, I thought they did more than that?

Again a BIG thanks to the guys posting here, it is much appreciated. I love talking about this stuff : )