Normals vs. contrast

[Testae]

Hello everybody,

at the moment I am working on my master's thesis with a RTI-related topic ("Reflectance Transformation Imaging of Transparent Materials") and therefore I of course have to think a lot about the results and information one can obtain with this technique. I came across this earlier but in the last days the following question crossed my mind again and again, especially as the new viewer is able to generate a normal map that can be viewed and examined: So imagine if you have a plain paper with text on it. Pictures are taken with a regular camera and you do your best lighting the whole thing equally or even use some kind of dome. And then you check the normals visualization in the viewer. What you will end up is a display of the paper, its wrinkles (if there are any) and structure (if your camera is good enough). But what you will also see is the text printed onto the paper. But the paint would have sunk into the paper when you printed it. And even if not, it should not build normals in the way they are visualized in the normal map. So this shows that the color sometimes affects the normals in quite anextensive way. Question is: If this is obviously the case, how do I know that an RTI of a painting for example really shows the paint layers and not the differences in color darkness in the normal visualization and the specular enhancement? I discussed the normals-and-color-problem a while ago with George already and then thought that it was due to the fact that I did not use a dome. But now I built a dome and especially contrasts of bright and dark colors still lead to the described issues. So I wonder what to think about this phenomenon?

Best,

Alex

[Taylor]

You make some interesting observations about how color apparently affects the normals, Alex.  I wonder if the printing method that was used to print the text on the paper could have any effect?  You mentioned that "the paint would have sunk into the paper...and even if not, it should not build normals in the way they are visualized in the normal map..."

 

Could you possibly run some controlled experiments using different inks, types and colors of paper, printing methods, and wavelengths of light (using filters on the lens or varying the light source)?  For example, restricting the wavelength to one in which the pigment color is transparent or has very little contrast with the paper could show whether the ink or the printing technique actually changes the surface normals independently of the color.  It wold be interesting to find out whether it's not the color of the ink per se, but the relative opacity or specularity of the ink that causes the paper to reflect light differently than the surrounding paper and affects the normals.  Paper coatings might also be affected in different ways by the ink. 

 

I could also speculate that since paper generally isn't opaque and has some translucency (i.e., it's not a perfectly diffuse Lambertian surface), other factors could come into play, such as varying patterns of light scattering within the paper fibers where the ink is deposited, or even shadows cast by the ink on the background behind the paper, but this could be stretching a bit if there's a simpler explanation for your observations.

[Testae]

Thank you for your interesting thoughts on that Taylor. I am afraid that I will not find the time to look into that topic more deeply as this is not a big aspect of my thesis but I gave it a try today with different printed papers (please see pictures below). One is a page from a book (lower left), the other is a journal's cover (lower right). I printed a lorem-ipsum using an ink jet printer (upper right) and a laser printer (upper left) and also wrote on the paper with a felt tip and a pencil. All of these do show up in the normals visualization and accordingly also in the specular enhancement mode. This of course should not happen as there is no change of the paper's surface normals just by printing on it. So I assume that color contrasts have a huge effect on the surface normals. This also carries a great danger of misinterpreting RTI-Files. I had some cases where I became very suspicious because for example the outlines of a painting looked very elevated in specular enhancement although they did not appear like that in the paint layer itself. One of the best "real life examples" was an RTI of an reverse glass painting I recorded lately (I hope this is the right term for those kind of paintings that are created on a glass plate instead of a canvas). So the pictures were taken through the glass depicting the bottom side of the paint layer. Thus the paint layer should have been absolutely flat and in respect of that one would expect a rather monochrome blue for the normal visualization. However the whole painting showed up in the normal map and appeared to have a three dimensional paint layer when viewing the file in SE-mode. So it looks to me as if I can not really trust the normals created from surface areas with various colors close to one-another. Is there an explanation for patterns only resulting from a color contrast to show up in the surface normal visualization?

 

Default

 

 

SE

 

normals 

[cdschroer]

This is an interesting topic.  I want to point out some research in this area, which while may not be directly answering this question, might still be of interest.

 

There has been work to look at the reproducibility of surface normals (which doesn't say anything aout how accurate they are) and also to lok at quantifying the accuracy using a target.  A link for this work was recently posted to the forums: http://forums.culturalheritageimaging.org/index.php?/topic/330-rti-calibration-with-a-spatial-target/

 

I'll note also that this work, as well as our own work at CHI, indicates that having a dome does not necessarily produce more accurate or more repeatable surface normals.  There are many factors affecting surface normal accuracy, and a well captured highlight RTI set is as good as a dome set, assuming all other factors are equal.

 

Maybe not as directly related - but I also recently saw some work presented on using spectral imaging with RTI, and the team was looking at manuscripts among other subjects.  The results for the project can be found here: http://palimpsest.stmarytx.edu/integrating/productsummary.html

 

There has also ben some work done on improving surface normal accuracy, since there are known issues, specifically with regard to shiny materials or where there is shadowing in the image set.  This work has ben going on for a few years at Simon Fraser University under Professor Mark Drew.  Tom Malzbender was involed in some of the early work.  There are several papers, but the most recent one is the masters thesis of Mingjing Zhang - which you can find here:

http://troy.lib.sfu.ca/record=b6372232

 

And finally, I wil note that my understanding and our eperience indicate that RTI does not work on transparent subjects, unless your goal is to shoot through the transparent substance to image something underneath (such as a documnt mounted under glass) I'd be interested to know more about the RTI transpareny work, if there is more that can be shared at this time.

 

Carla

[leszekp]

While I haven't imaged any fully transparent items (like glass bottles), I've had some success with semi-transparent material, specifically projectile points knapped out of semi-transparent obsidian. At a bare minimum, you need to have a uniformly-colored background material, preferably one with little to no texture.Here are links to three photos.

 

First is a standard photo of the item against a photo scale, lit from overhead, where the scale is visible through the artifact.

 

 

The second shows the normals view derived from the 2nd-order RTI, and the scale is still visible through and around the artifact.

 

 

The third photo is a 2nd-order normals image where the artifact was on a more uniformly-colored background.

 

Each small division is 1 mm.

[Taylor]

Thanks for the links to other research, Carla, and for the images, Alex and leszekp.  It looks like some further tests would be useful, and I'll try some experiments myself and post the results if they seem helpful. 

 

Since the topic of Alex's research is RTIs of transparent objects, I'll post a few images from my first RTI project, captured during the CHI class I took in January 2012.  The subject is a pendant carved from rutilated quartz, a translucent mineral with opaque inclusions of rutile (TiO2), which appear as fine spindles within the pendant.

 

1st image:  RTI default view.

2nd image:  RTI normal map

3rd image:  PTM normal map

4th image:  PTM detail with Image Unsharp Masking showing tool marks

 

The rutile inclusions are visible in the RTI normal map (2nd image), even though they're on the inside of the pendant and shouldn't affect the normal vectors.  Since there's no slider for "gain" or removing color from the image, I wonder if it's possible that the RTI "normal map" is simply a blend of the RGB color information and the normal vectors?  The normal map for the PTM file (3rd image) has much higher contrast and the rutile fibers are much less visible.  This suggests, although it doesn't prove, that the color contrasts visible in the RTI normal map due to the rutile fibers (2nd image) have something to do with the rendering algorithm in the RTIViewer, rather than errors in the normal vectors themselves. 

 

The 4th image shows that very fine tool marks on the surface of the carved pendant can be effectively captured using RTI, even though translucent objects are sometimes difficult to capture.

[Bender]

Alex,

 

I've done similar experiments with high contrast writing on paper as well, and get similar results. I think there are 2 factors at work here. First of all, as Taylor suggests, the process of printing does change the surface somewhat. Some geometrical deformation seems unavoidable, especially when using the enhancement methods that are sensitive enough to bring out even faint indented writing without ink. 

 

The more important factor though is that the computation of the normals is not error free, even in the best photometric stereo methods. The normals are computed from some assumption on the form of the reflectance function, polynomial in the case of PTMs. In the PTM viewer, we simply find the maximum of the polynomial fit to the reflectance function to establish the normal direction, a reasonable approximation for surfaces close to Lambertian. I believe this is also done in the RTI viewer currently, although work is in progress on more advanced techniques. However, even for a perfect Lambertian surface, this is only an approximation, as is the assumption that your surface is Lambertian to start with. Certainly ink is a much less diffuse surface than paper is, so differences in normal estimation will be introduced. Even self-shadowing from microfacets will effect reflectance measurements that are used for normal estimation.

 

For a more advanced method of estimating surface normals from RTI input, take a look at the paper by Mingjing that Carla references as well as the earlier work by Mark Drew et al. This technique works quite well in practice. There are dozens if not hundreds of papers on variants of photometric stereo out there though, it's a subfield of its own.

 

On the topic of transparency, I would concur with Carla's comments. The enhancement techniques in RTI and PTM were designed to be used with opaque surfaces, so all bets are off in that case. 

 

Tom

[Testae]

Thanks a lot for your replies and links everybody! Especially Tom's explanation helped me a lot in understanding why these phenomenon occurs. I think it is very important to know that there is a chance of these errors to occur so one has that in mind when it comes to looking at RTI files of paintings and other polychrome surfaces.

For the transparent materials: I discovered the possibility of using RTIs on transparent materials "accidently" when looking at a RTI of a painted glass bottle. In the areas where there was nothing else but glass some fine scratches showed up in the specular enhancement mode that were nearly invisible when looking at the glass directly let alone in photos of the object. After some other tests I can say that in some cases it is possible to show details of structures on or in transparent materials using RTI better than this would be possible using plain photography. Based on what I found out so far the most important thing is to use plain monochromatic background with no structure, just as leszekp mentioned. If this background is black, nearly all the light is reflected by the transparent object's surfaces. And this helps to show details on and in the glass or transparent polymer. Of course the result does not show one surface of the object but all surfaces (so if you compare the information in a regular RTI to a photo, the information in a RTI from a transparent object is somehow rather an x-ray *wink* ). But every structure that reflects the light does show up, e.g. surface irregularities, scratches, bubbles and so on. So I think if one understands where the information one gets with this data is coming from (i.e. not the first surface of the object but every reflecting surface plus there is an interaction of various reflections from the different surfaces) RTI is quite a helpful tool when it comes to investigating transparent objects. Not because it accurately visualizes everything but because it shows things that would otherwise remain barely visible.
Here are some examples of polymer materials (not the best results I achieved so far but I just had that file sitting around):

 

First of all a pet packing. Although there are some errors a good amount of the surface normals looks quite "possible" in my opinion.

 

 

 

This is a transparent PE bag. As you can see, the SE-mode clearly helps to show the surfaces structure of the wrinkled plastic.

 

 

 

Green transparent plastic. The scratches appear more obvious in the SE-Mode (please note that in this case a white background was used. A black background would result in better images with clearer surface normals.

 

 

Last one is a PS box. Here the SE-mode can help to show the fine scratches in the surface, while in the default mode especially the rough and big scratches are visible.

 

 

I would love to hear your opinions on these.

 

Alex

[Taylor]

For what it's worth, I've attached a comparison of normals processed using both the PTM and RTI (hemispherical harmonics, or HSH) algorithms in both visible (RGB) and infrared (940 nm) wavebands:

 

Comparison of normals.pdf

 

The quick-and-dirty setup includes 3 different felt-tip marker colors (two different blacks and one orange) in horizontal lines on 3 different surfaces:  black gaffer's tape and two colors of post-it notes, yellow and magenta.  I used felt-tip markers in an attempt not to make an impression on the paper surfaces, but this wasn't completely successful, as you can see (the orange line on the post-its left a deeper impression, because I went over it a few times).

 

There are some obvious errors in the normals outside the ink lines that probably have more to do with some inconsistency in my quick-and-dirty highlight method.  I didn't use a string, I just estimated the distance from the light source to the surface during the image captures.  I should also mention that the visible RGB and IR light sources were different:  I used a Canon speedlight for the RGB images and a 60-watt incandescent bulb in a reflector for the IR images, due to the longer exposure times required.  The PTMs and RTIs were processed using the same data set for each waveband (visible RGB and IR).

 

All three surfaces (gaffer's tape and post-it notes) are attached to a notebook that has a matte-black plastic cover with a slight curvature from left to right.  The matte-black plastic notebook cover still has some specularity, however.  You'll notice the normals on the notebook cover are very close to the normals of the black lines, creating an effect of "windowing" through the post-it notes, especially in the visible-range RTI (Figure 1f).   In the IR waveband, only the orange ink was relatively transparent in the PTM and RTI images (Figures 2a-f), while the black inks were still visible.  

 

The lines on black gaffer's tape were intended to show the extent to which color contrasts or other ink and surface properties had an effect on the normals.  Although there's not much contrast, all three lines are visible in the RGB default views (Figures 1a and 1d) and they are more or less visible also in the normal views (Figures 1c and 1f).  This part of the experimental setup seemed inconclusive to me, but comments are welcome.  Maybe some better-controlled experiments would be worth doing.

 

Consistent with Tom's comments above, I think the effect on the normals has more to do with specular properties of the ink rather than the color contrast, although the effect is generally more pronounced on the lighter surfaces where there is more contrast.  This could have more to do with contrasts in specularity than color, however.   The gaffer's tape has more specularity than the post-it notes, which seem closer to Lambertian surfaces.  The normals on the ink follow very closely the normals on the matte surface of the notebook, which isn't what you'd expect if the color contrast of the ink was affecting the normals.

 

Comparing PTM and RTI normals in Figures 1c and 1f, however, there is almost no effect on the PTM normals in the magenta post-it on the right of Figure 1c, even though the lines have a lot of contrast in the default (Fig. 1a) and specular enhancement (Fig. 1b) views.  The normals in the PTMs (Figs. 1c and 2c) seemed to be less affected by the contrast of the ink on the post-its than the normals in the RTIs (Figs, 1f and 2f), and are more affected in the visible RGB captures (Figs. 1c and 1f) than in the IR captures (Figs. 2c and 2f). 

 

Finally, I'll note that color contrasts are quite visible in the normal views (see the scale for contrast), but this isn't clearly just the normal directions.  It appears that the normal views are blends of the both pseudo-color normals with RGB data.  There seems to be less RGB data in the PTM normal views (Figs. 1c and 2c) than in the RTI normal views (Figs. 1f and 2f).  Can anyone confirm if this is the case?

 

Testae, your observation that RTI can be effective at least on semi-transparent materials is consistent with the experience I had with the rutilated quartz pendant above, if the surface isn't highly polished and if there's enough surface texture to diffusively reflect a significant amount of light from the surface, without too much specularity.  I'd be happy to make my PTM and RTI files available if you'd like to use them.

[Bender]

Testae,

 

Interesting stuff! It does make sense that in the transparency case you will have contributions to the reflectance function at each pixel from any surface that crosses that pixel ray, in addition to contributions from the opaque background surface. Keeping the background dark would minimize the effects from that background. Its kind of a trick to keep the fact that the subject is transparent from mattering. RTI does seem useful in the transparency case since it can potentially 'amplify' the visibility any surface structure, like bubbles, etc. in the medium as you point out. Your examples make a strong case for helping see the first reflecting surface better, do you have any examples of internal surface structure being visible that is otherwise difficult to discern?

[cdschroer]

Taylor -

 

Nice job with the tests! I think that is useful additional data.

[leszekp]

Here's someone who's using RTI to document cracks, scratches and bubbles in transparent glass bottles.

 

Original website (in German): http://www.recad.de/Dokumentation/dokumentation.html

Googlefied into English: https://translate.google.com/translate?sl=auto&tl=en&js=y&prev=_t&hl=en&ie=UTF-8&u=http%3A%2F%2Fwww.recad.de%2FDokumentation%2Fdokumentation.html&edit-text=&act=url

[Testae]

leszekp,

 

this is my website, glad you found it...

As mentioned above I worked on the topic of documenting glass, transparent polymers and other transparent objects with RTI in my Master's thesis. I will soon have it ready to hand it out to anybody who is interested in the topic (just some little layout corrections etc.) but in the meantime you may want to check out this poster from a conference in Berlin: https://de.dariah.eu/documents/10180/472725/48_Dittus-RTI_transparenter_Materialien_Poster.pdf/2b300533-d145-4dc5-b424-4411dde85551

It is in German I am afraid, but you can see in the pictures how RTI can improve the documentation of cracks, bubbles, engravings etc.

[leszekp]

Thanks for the link, Testae - look forward to your thesis (and trying to revive my grade-school German). And congratulations on your award!