Early in 1900's a Russian photographer, Sergei Mikhailovich Prokudin-Gorskii, attempted to make color photos by taking three exposures through red, green and blue filters stacked one ontop of another. The goal of this project is to create a procedure to automatically align each of these images to create a single color image with as little errors in alignment as possible.
The most straightforward way to align these images is to have some heuristic, and take the best score from all possible ways to overlay the channels. This is very slow, we can do better. Since the images are already mostly aligned we can search through a smaller displacement window and use the best score in that window. For the heuristic we can take the summed squared errors of every pixel value and the lower the score the better the displacement. Below are the results from this method.
For larger images the basic method is too slow, since the window we would need to search over can be hundreds of pixels. A better way would be to rescale the images and do the basic alignment on the downscaled image. This means if I rescaled an image down to 1/8th its original size, every pixel I cover will be 8 pixels in the original image. With this knowledge we can find the best displacement for each rescaled image, and propagate this displacment up our image pyramid to start the search from a more informed position. Not only does this greatly increase performance but it also greatly increases the quality of the image as shown below.
Using the raw pixel values has a major downfall, what if a large portion of the image is either red, green, or blue? In the case of our friend Emir here, his clothes are mostly blue which means that we might have some problems attempting to align his picture. One solution is to do edge dectection on the image. Edge dectection finds the edges of objects in an image and outlines them. Using the edges of our subject we can much better align his body and the background to their true displacement. I used canny edge dectction, found in the skimage library, for simple and quick edge detection. While this did hurt performance, in the end it led to greatly increased picture quality.
From all the examples from above the borders of our image dont particularly look very nice, in order to fix this I implemented automatic cropping. The basic idea is to use edge detection on the borders of an image (about the outer 10% of the image) and find straight lines along the edge and crop the image there. In order to find the best cropping I use several values to tweak the edge detection to be smoother or more granular, and I take the maximum ammount of cropping and use that in the final result.
