Jump to content


  • Content Count

  • Joined

  • Last visited

  • Days Won


jedavis last won the day on February 23 2015

jedavis had the most liked content!

Community Reputation

11 Good

About jedavis

  • Rank
  1. You can think of this as a sparse data interpolation problem. We measure N lighting directions, and we have to fit M polynomial coefficients. The data is noisy, not properly bandpass filtered, has outliers, etc. If you think back to your freshman calculus or other numerical class, in an ideal world you need at least M knowns, to estimate M unknowns, so we have N>M. If this isn't true, then we have an overfitting problem. Of course the data isn't perfect, so as a rule of thumb, lets say we need (N/2)>M. It was asked above what overfitting is? See the image below for a visual/math way to think of this problem. In practice, it means that you will see "noise" appear when you put the lighting direction at any direction other than the ones you sampled. This is the extra wiggling in the plot on the right that isn't real data. This happens when you have too many polynomial terms. Now on the question of PTM vs HSH, what are we changing, we are changing the choice of polynomial. They are both polynomials, but maybe one as a term for xy and the other has a term for x^2. The original PTM paper defined 6 terms. The plot given above from the Zhang et al paper has a definition that allows a variable number of terms. This is the first time I've seen PTM defined with a variable number of terms, and I think all existing fitters and viewers use the 6 term definition. HSH are Hemi-spherical Harmonics. They are also polynomials, but have a historical mathematical definition which includes 4 terms, 9 terms, 16 terms, etc. The question is which polynomial terms are best? Well, that depends on your data. We did some experiments in the 2007 time range of just trying random polynomial terms to see if we could do better than PTM or HSH, and indeed we could, but it was dependent on the images we tested, and we abandoned that research before finding a new set of terms which was always better. The plot above says that for whichever set of images was tested, the extended definition of PTM was better than HSH. In terms of real choices of tools you can use, you can have PTM-6 or HSH-4, HSH-9, HSH-16, there aren't widely available tools for anything else. Since we dont really have compression built into any of the tools, you can expect the file sizes to roughly scale with the number of terms. You can also expect to fit the data better with more terms. Since matte surfaces are more flat and specular surfaces have a bump at the highlight, then thinking about the plot above, we can see that more terms let us represent the bump of the specular highlight better. The last point is about how to evaluate error. In papers we like our nice plots. We generally use some metric that comes down to a number. It might be RSME or some perceptually driven metric, but it always comes down to a "quality number". This is a gross simplification. In practice none of these methods know what the object *really* looks like between light directions, because we didnt capture an image there. So we are making up what it looks like. Its the space between the samples in the plot above, is it straight? or curved? or has a wiggle? We just dont know. It might be true that the "quality number" thinks the wiggle has lowest error, but viewer A likes the straight fit and viewer B likes the curved fit. This is why I said earlier in this thread that you would have to look and see what you like. In the image processing papers people often use Structual Simimlarity (SSIM) when they want a human perceptual number, but its still just a "quality number" which still grossly simplifies the situation. In my experience its primarily a feel good for researchers to claim they are doing the right thing, but its not substantially different than RSME. http://en.wikipedia.org/wiki/Structural_similarity
  2. You have the basic idea right. In a normal image, each pixel stores the RGB color value for that pixel. If we want a relightable image, we could just store the RGB color value for each of 50 lighting directions in each pixel, and then look up the right one when we want to draw the picture. But this wouldn't let us interpolate in the color in between the lighting directions we actually took pictures of. So we fit a polynomial to the 50 RGB values instead. This polynomial has 6 terms and thus 6 coefficients in PTMs, and either 9 or 16 coefficients in the most common RTIs. Spherical harmonics are just a specific set of polynomials. So really PTM and RTi are just doing exactly what you would do in excel if you wanted to make a plot from some scatter data points and told excel to fit a curve for you. When you use the 'normal' PTM or RTI render this is happening. One additional thing that is often used with PTM and RTI is to calculate the surface normal. The surface normal (the local orientation of the surface) can be used to calculate synthetic lighting that many people find useful for visualizing small scratches and features on objects. When you use this mode the picture is rendered via computer graphics and the PTM coefficients aren't used at all. There are a variety of rendering modes and some might combine both sets of data.So part of your confusion is there is a set of techniques often used together and the exact method depends on the rendering mode you chose.
  3. Is it available to try out? It turns out I've got some students in a class supposed to build an iOS viewer this qtr. We'd love to try yours. davis@cs.ucsc.edu Prof. James Davis - UCSC
  4. I would say "pre-commercially" available. The dome shown at relightable.com is one developed by myself and collaborators. I'm a professor at UCSC and longtime collaborator with CHI. My lab has been involved in various standardization and software building with CHI over the years. The intention of building the dome was to "make a standardized version available to organizations that can't afford a custom build". This is in response to my own experience that it was costing me $10-30K to build custom versions in my university lab and I was seeing lots of other labs repeat this. We have a manufacturing house that can build them for us, but we aren't really set up as a company. Its more like I'm a university professor, I saw a need based on my work with CHI, and just took a step forward. I'd say the current stage is "Check to see if the thing we are building matches a real need". We have versions of this dome at a couple of museums now trying to understand if its useful. We'd be thrilled to work with more people so that we make something people actually need. I had previously sent a note to other professors in computer science that I happen to know saying "I can build these things for about $5K each, and happy to ask for more built if anyone wants one." I'm happy to extend that to anyone in the cultural heritage world. At this stage thats barely covering costs (well at this stage its definitely not covering costs, my wife isn't thrilled, and I hope to rectify that soon). Its unclear at this point if there is enough interest to be on track to ever be "commercially available", but certainly if we can work out something financially sustainable I have students that would be interested to take it on. Please feel free to get in touch. For the moment, I think I can get one to anyone for $5K. However if you're interested and this is out of your budget, then I'd still be interested in a note with your usage and budget. I can obviously go back to the manufacturer and strip away features for a lower cost if there is a demand. davis@cs.ucsc.edu Prof. James Davis
  5. I think Carla's comments are accurate. I don't think I can comment on the practicalities of using them, but in terms of the mathematics, the number of coefficients controls the "possible" quality of the result. Using too few will lose detail of the reflectance, but using too many will overfit and produce incorrect results. In this sense you can think of PTM as about the same as if there were HSH of order 1.5. I think for practical purposes they are all about the same and you can just try them and see what you like.
  • Create New...