Learning local evidence for shading and reflectance

We address the important and unsolved problem of determining whether variations in image intensity are caused by changes in surface normal (shading) or reflectance (paint). A solution to this problem is necessary for machines to interpret images as people do and could have many applications. We take a learning-based approach. We generate a trainiiag set of synthetic images containing both surface normal and reflectance variations, and then label the variations at each position, scale, and orientation as to whether they are caused by shading or paint. The classification is done locally, using a feature vector of nonlinear filter responses. We fit a probability density model to the filter outputs using a mixture of factor analyzers. The resulting model indicates the probability based on local image evidence, that a pyramid coefficient at each orientation and scale is caused by shading or reflectance variations. Although the classification is done using a fixed lighting direction, we can solve for the correct lighting direction by rotating the image to the orientation, relative to the light source, that gives the most shape-like labelings. The labeling allows us to reconstruct two high passed images: one contains those parts of the input image caused by shading effects, while the other contains only those parts caused by reflectance changes. The resulting classifications compare well with human psychophysical performance on a test set of images, and show good results for test photographs.