Depth Edge Detection Using Multiflash Imaging                                         

                    A report by Gabriela Martínez, December 2005.
                   
This project is based on the paper: "Non-photorealistic Camera: Depth Edge and Stylized Rendering using Multi-Flash Imaging" [Ramesh et al. 2004]
                                                     
     INTRODUCTION

The problem of detecting edges in two dimensional images is well founded in computer vision. Several techniques have been proposed, for example Canny Edge Detector. The majority of the available techniques have been remained within the classical formulation problem: given a single image, how can one detect the edges of important features?.
Ramesh et al. take a different approach to solve this standard problem. They used a camera with multiple flashes that are strategically positioned to cast shadows along the depth discontinuities (depth edges) in the scene. The projective geometric relationship of the camera is exploited to detect depth edges and distinguish them from intensity edges due to material discontinuities.

     ALGORITHM

The method is based on two observations regarding epipolar shadow geometry.

           

First note that, a shadow of a depth edge pixel is constrained to lie along the epipolar ray passing trough that pixel. Second, the shadow is observed if and only if the background pixel is on the side of the depth edge opposite the epipole along the epipolar ray.

The algorithm needs a minimum of five images to detect depth edges of all orientations: Ambient image (taken without flash), and four images where the flashes are positioned above, below, left and right of the lens.

                                 
                                 

The basic algorithm is as follows:
  1. Capture ambient image I0
  2. Capture n pictures with a light source Ik+ for  n=1,2,3,4.
  3. Compute Ik=Ik+ - I0
  4. Compute the maximum intensity image. For all pixels x, Imax=maxk(Ik(x))  k=1,2,3,4. 
  5. For each flash image k, compute ratio image Rk.  Rk(x)=Ik/Imax(x) .
  6. For each ratio image, traverse epipolar rays from epipolar ray ek. Find pixels y with step edges with negative transition and mark them as a depth edge.
Create ratio images is useful because pixels lit by flash k the ratio image value is roughly 1.0; for pixels in shadowed regions, the values is close to 0.0. Thus, intensity profile shows a sharp negative transition at the depth edge as we traverse from non-shadowed foreground to shadowed background. This reduce the depth edge detection problem to an intensity step edge detection problem. Moreover, it is an easy case, since the constructed is virtually free of features except along depth edges. It is therefore trivial to use a gradient base method such as Sobel convolution kernel.
It is necessary to note that we only are interested in negative transition (regions where the gradient is negative) and mark those pixels. After marking those pixels in the four ratio images, we combine the edge maps from all the n images in order to obtain our final confidence map and finally threshold it to eliminate noise.
Once the depth edge detection is complete it is easy to detect texture edges because Intensity edges are boundaries between lit and shadowed regions due to ambient light source(s), rather than the flashes attached to the camera. So, texture edges are intensity edges of Imax images minus the depth edges. As we proceed previously, we perform a similar gradient based edge detection algorithm on the maximum intensity image Imax, and then subtract the depth edge image.

     IMPLEMENTATION

I implemented the algorithm in Matlab because Matlab's Processing Image Tool Box has all the function that we need to solve our problem.
The implementation is really simple and it has to be followed as it is described in the paper. To solve the intensity problem I used the Sobel kernel (it can be used Prewitt kernel too) generated by fspecial and apply it according to flash orientation using imfilter .
The thresholding method was implemented using Matlab's function bwlabel . The first step is mark each pixel in the confidence image that passes a low confidence value, and the second step is mark pixels that passes a high confidence value. Next, bwlabel compute all connected components of both images.       

     RESULTS
     COMMENTS

The algorithm presented by Ramesh Raskar et al. is easy to implement and it requires a little computation. The technique give us a robust classification to distinguish depth edges from texture edges.
An important contribution of the method is they exploit the epipolar relationship between light sources and cast shadows to extract geometric features from multiple images of the scene so instead of estimate the full 3D coordinates (create a 3D scene model) of points in the scene, and then look for depth edges, this technique reduces general 3D problems of depht edge recovery to one intensity step edge detection.

     REFERENCES