Due Date: 11:59pm on Tuesday, September 18, 2012
Overview
This projectexplores EulerianVideo Magnification to reveal temporal image variations that tendto be difficult tovisualize. This videobriefly describes the technique, and shows a variety of visual effectsattainable with it. For example, one can amplify blood flow andlow-amplitude motion with Eulerian Magnification without needing to performimagesegmentation orcomputing opticalflow.
The primary goal of this assignment is to amplify temporal colorvariations in two videos(faceand baby2),and amplify color or slow-amplitude motion in your own custom short video sequence that you record yourself. For this you will readthe paperabout Eulerian Video Magnification and implement the approach.
How can you amplify image variations that are hard to see with thenaked eye? The insight is that some of these hard-to-see changes occur atparticular temporal frequencies that we can augment using simplefilters in the frequency domain! For example, to magnify pulse we canlook at pixel variations with frequencies between 0.4 and 4Hz, whichcorrespond to 24 to 240 beats per minute.
The Eulerian magnification process is straight forward:
(1) An image sequence is decomposed into different spatial frequency bandsusing Laplacian pyramids
(2) The time series corresponding to the value of a pixel on all levels of the pyramid are band-passfiltered to extract frequency bands of interest
(3) The extracted band-passed signals are multiplied by a magnification factor and this result is added to the original signals
(4) The magnified signals that compose the spatial pyramid are collapsed to obtain the final output
If the input video has multiple channels (e.g., each frame is a color image in RGB color space), then we can process each channel independently. The YIQ color space is particularly suggested for Eulerian magnification since it allows to easily amplify intensity and chromaticity independently of each other (we can use rgb2ntsc and ntsc2rgb to move between RGB and YIQ in MATLAB).
The hard part of the process is to find the right parameters to getthe desired magnification effect. For example, one can change the sizeof the Laplacian pyramid, multiply the time series corresponding tothe value of a pixel by different scale factors at different levels ofthe pyramid, or attenuate the magnification when adding the augmentedband-passed signals to the original ones. The choice of the band-passfilter (e.g., the range of frequencies it passes/rejects, its order,etc.) can also influence the obtained results. This exploration ispart of the project, so you should start early!
Laplacian pyramid (20pts)
The first step to augment a video is to compute a Laplacian pyramidfor every single frame (see Szeliski's book, section 3.5.3). TheLaplacian pyramid was originally proposed by Burt and Adelson in their1983paper TheLaplacian pyramid as a compact image code, where they suggested tosample the image with Laplacian operators of many scales. This pyramidis constructed by taking the difference between adjacent levels of aGaussian pyramid, and approximates the second derivative of the image,highlighting regions of rapid intensity change.
Each level of the Laplacian pyramid will have different spatialfrequency information, as shown in the picture above. Notice that weneed to upsample one of the images when computing the differencebetween adjacent levels of a Gaussian pyramid, since one will have asize of wxh, while the other will have (w/2)x(h/2)pixels. Since the last image in the Gaussian pyramid does not containan adjacent image to perform the subtraction, then it just becomes thelast level of the Laplacian pyramid.
Notice that by doing the inverse process of constructing a Laplacianpyramid we can reconstruct the original image. In other words, byupsampling and adding levels of the Laplacian pyramid we can generatethe full-size picture. This reconstruction is necessary toaugment videos using the Eulerian approach.
Temporal filtering (40pts)
We consider the time series corresponding to the value of a pixel onall spatial levels of the Laplacian pyramid. We convert this timeseries to the frequency domain using the Fast Fourier Transform(fft in MATLAB), and apply a band pass filter to thissignal. The choice of the band-pass filter is crucial, and we recommenddesigning and visualizing the filter with fdatooland fvtool in MATLAB(see an example by MathWorks).
To make this process easier, we provide you with a butterworthBandpassFilter function togenerate a Butterworth band-pass filter of a particular order. Thisfunction was generated with fdatool, and is optional for you to use. You can download thefile, or use the code below for reference:
% Hd = butterworthBandpassFilter(Fs, N, Fc1, Fc2)
% Fs - sampling frequency (e.g., 30Hz)
% N - filter order (must be an even number)
% Fc1 - first cut frequency
% Fc2 - second cut frequency
% Hd - approximate ideal bandpass filter
function Hd = butterworthBandpassFilter(Fs, N, Fc1, Fc2)
h = fdesign.bandpass('N,F3dB1,F3dB2', N, Fc1, Fc2, Fs);
Hd = design(h, 'butter');
end
More details on the fdesign.bandpass parameters can befound here. Checkthe Eulerian VideoMagnification paperfor details on the parameters they used on the face andbaby2 videos. You will have to find the right parameters for the other video that you capture yourself, and process.
In order to filter the time series of the pixels fast, werecommend you perform this operation in the frequency domain, sincemultiplication is faster than convolution. But be carefulabout fft's output format when doing this! As explainedin thistutorial, the DC component of fftx = fft(x),for x a 1D signal, is the firstelement fftx(1) of the array. If x has aneven number of samples, then the magnitude of the FFT will besymmetric, such that the first (1+nfft/2) points are unique, and therest are symmetrically redundant. Theelement fftx(1+nfft/2) is the Nyquist frequency componentof x in this case. If the number of samplesof x is odd, however, the Nyquist frequency component isnot evaluated, and the number of unique points is (nfft+1)/2.
Also, if you decide to use the butterworthBandpassFilterfunction, then you will need to get the frequency components of the filterfor fast computation. This can be done by usingMATLAB's freqzfunction, by passing the filter and the length of the output that youwant (i.e., fftHd = freqz(Hd,NumSamples)). Again, be careful abouthow the frequency components are output by freqz.
Pixel change magnification (10pts)
After extracting the frequency band of interest, we need to amplify itand add the result back to the original signal.
Image reconstruction (30pts, which includes evaluation of your results)
After amplifying the signals, all that is left is to collapse theLaplacian pyramids into a single image per frame. Notice that we can attenuate the amplification to obtain different resuts, or we can low-pass filter the amplified signal to reduce effects on high frequency components of the images, such as borders. Two differentblood flow amplification effects on the face.mp4 image sequence are presented below,
Bells & Whistles (Extra Credit)
Try some special moves to increase your score:
- (5pts) Pick a time series corresponding to the value of a pixel on any pyramid level, and make a plot of power vs frequency after converting to the frequency domain (you can follow these steps for plotting). Show the plot in your project website and indicate the pyramid level of the pixel you chose, and its (x,y) location.
- (5pts) Using the same time series as above, make a plot of power vs frequency after band-pass filtering the signal in the frequency domain. Show the plot in your project website and indicate the pass band of the filter that you used.
- (15pts) Try two additional set of parameters on each of the processed videos (face, baby2, and the one you captured), and show the different results you obtained. Explain what is being augmented in each case (e.g., color variation or slow-amplitude motion).
- (10pts) Try to augment low-amplitude motion specifically in an another video provided by the authors of the Eulerian Video Magnification paper (see the Data section of the project website). Do not pick the face or baby2 video for this purpose, nor any other video you amplified before.
Deliverables
Use both words and images to show us what you've done (describe in detail your algorithm parameterization for each of your results).
Place all code in your code/ directory. Include a README describing the contents of each file.
In the website in your www/ directory, please:
- Include a brief description of the project, and explain how you constructed the Laplacian image pyramid.
- Show your result for amplifying blood flow on the face and baby2sequences, as well as your results for the additional sequence that youcapture and process. Your results should mimic the resultsobtained by the authors on the face and baby2 sequences (they shouldamplify similar pulsation rates, though colorizing may bedifferent). Upload your videos to YouTube, then embed them into yourproject web-page, and indicate the parameters you used in each case.
- Explain any difficulties and possible reasons for bad results.
- Include any bells & whistles and explain what parameters you used.
