Image Frequencies and Combinations

CS180, Joshua Liao

Background

Frequency domain analysis doesn't just exist for audio signals, but also for visual signals!
This project shows some of the interesting visual consequences when you play with the mathematics of visual signals.

Part 1.1: Filters

You can use finite difference filters as proxies for partial derivatives.
By convolving an image with the filters below, you get a matrix whose values represent the difference between adjacent pixels!
\( D_y = \begin{bmatrix} 1 & -1 \end{bmatrix}, D_y = \begin{bmatrix} 1 \\ -1 \end{bmatrix} \)

This is the original, greyscale 'Cameraman' image (white border artifacts included).

Convolution with \( D_x \)

Convolution with \( D_y \)

This is the magnitude of the gradient, derived from \( sqrt(D_x^2 + D_y^2) \).

By binarizing the gradient using a threshold value,
we can see where the foreground edges are (locations of high gradient).
(threshold qualitatively determined as 60 out of 255).

Part 1.2: Gaussians

Note that the finite difference filter is prone to noise in the edge image.
We can fix that by using a Gaussian filter to blur the image beforehand, eliminating noise, and then applying the finite difference filters.

We start with a blurred image by convolving with a Gaussian filter.
kernel size = 8 and sig = 6, ie. three standard deviations in all directions lie in the filter.

Convolution with \( D_x \). We can see the lines are "thicker"; this is because of blurring.
The image's original edges have a 'wider scope' of influence.

Convolution with \( D_y \). Similarly, thicker edges.

Magnitude of the gradient, derived from \( sqrt(D_x^2 + D_y^2) \).
Note that it looks smoother, and noise at the bottom of the image goes away.

Binarized gradient. This time, the threshold is 20.
Blurring averages out values, making high values / outliers less present, meaning that the threshold can be lower. There are less noisy edges found.

Convolution is an associative operation. Thus, you can convolve the dx, dy finite difference operators with the Gaussian used to blur, and then convolve with the original image to get the same result shown here.

Part 2.1: Sharpening

Gaussian filters blur the image— in other words, they remove high frequencies, or serve as a low-pass filter.
How do we get the high frequencies? We can just subtract the low frequencies! Sharpening is a technique where you add more high frequencies back into an image to "sharpen" its details.
FAQ: What are high frequencies in images? High frequencies mean signals that change more quickly.
In the visual domain, this is most easily visualized by edges, which represent sudden changes (as opposed to gradual change).

This is the original image of the Taj Mahal.

After convolution with a Gaussian, we can get a low-freq copy.

When you subtract the low frequencies from the original, you get high frequencies.

Original, for comparison.

Sharpened Taj Mahal: alpha = 1, gaussian kernel = 21, sigma = 21/6.

Similar to before, some clever maths allow us to represent the whole operation with one convolution, instead of subtracting large images.
Here's the same result using one convolution.

Bonus: Here's the raw high frequency image.
What's with the color artifacts? When you subtract the low frequencies, it's possible to get pixel values outside the normal bounds of [0, 255]. The image then looks very strange! For visualization, the high frequency is clipped to these bounds to look nicer. However, in the actual operation / math, this intermediate clipping is not executed until the final result.

What if you do clip in intermediate operations?
(If you subtract and end up negative, clip to zero, then add, you get a different result than if you skip the clipping.) Here's the sharpened Taj when you perform intermediate clipping on the high frequency image. It looks less sharp! This is probably because the raw pixel difference in the high frequency image was a negative value or over 255; the true range of variation/difference was lost when the values were clipped.

Here's a raccoon.

The high frequency parts of the raccoon.

Add them together for a sharper raccoon.

Here's a shiba.

The high frequency parts of the shiba.

Add them together for a sharper shiba.

Here's a cat.

The high frequency parts of the cat (and the bike).

Add them together for a sharper cat.

Bonus image of low frequency cat.

This is the result of re-sharpening a sharpened image!
The sharper cat image above this one was blurred, then subtracted away from the sharp cat to get a second degree of sharp detail. In some areas, this detail is pretty nice— for example, the fur on the cat is very detailed. However, the image as whole looks a bit fried.

Part 2.2: Hybrid Images

Hybrid images play with human perception by encoding one image at a high frequency, and then another image at a low frequency. Try looking at the following images from close up, observing the fine details encoded at high frequencies, then looking from far away, when the human perception no longer registers the higher frequencies and sees the low frequency image instead.
All hybrid images were generated by a len 25 Gaussian filter, low_sigma = 25/6, high_sigma = 25/8.

Here is Derek...

And here is Nutmeg...

Here is the hybrid image of Nutmeg on top of Derek. Close up, Derek has catlike features, but far away, you can see just Derek.

Bells and Whistles:
Let's examine how color works for hybrid images. Here, the low frequencies are colored, while the high frequencies are processed in greyscale. Coloring works here because low frequencies capture illumination and larger local region trends.

And here, the high frequencies are colored, while low frequencies are greyscale. Coloring doesn't work with only high frequencies, because high frequencies inherently represent rapid change in visual data.

Here is President Obama smiling.

Here is President Obama being stern.

Here is a hybrid of Obama. Close up, Obama appears stern. But as you go farther away, you can tell that he is really smiling. This image highlights another key implementation detail for hybrid images: when you select images, the image size and background should be similar— otherwise you get artifacts like seen here because stern Obama had to be rotated to match smiling obama.

Basketball.

Soccerball.

A basketball superimposed onto a soccerball.
This hybrid image is not quite as successful as Derek-Nutmeg. The reason for this failure is because when you make hybrid images, you need to consider perception as whole— and of course, while the shape fits here, the colors don't really work. The texture of the basketball can only be seen while close to the image, but the colors aren't carried over.

Let's look into the hybrid image process. Here is the low frequency portion, where the soccerball goes through a low-pass filter.

Here is a high-pass filtered basketball.
(Remember, these color artifacts come from not clipping in intermediate steps.)

This is the frequency representation of the original soccerball, using FFT.

This is the frequency representation of the basketball.

The FFT frequency representation of the low-pass'ed soccerball.
Note how many of the frequencies are now gone.

The FFT frequency representation of the high-pass'ed basketball.
Note how the amplitude graph is brighter now.

Finally, the FFT representation of the hybrid image.
You can track the diagonal line present in the high pass image FFT's showing up here.

Part 2.3, 2.4: Blending the Oraple

As described in the 1983 paper by Burt and Adelson. We show some results in image blending.
Specifically, using a Laplacian stack of two images and a mask, and then blending images at each level at the stack to create a seamless blended image.

The original apple!

The orange-nal orange!

Apple 0 in the Gaussian stack.

Apple 2 in Gaussian stack.

Apple 4 in Gaussian stack.

Orange 0 in the Gaussian stack.

Orange 2 in Gaussian stack.

Orange 4 in Gaussian stack.

Apple 0 in laplacian stack.

Apple 2 in laplacian stack.

Apple 4 in laplacian stack.

Orange 0 in laplacian stack.

Orange 2 in laplacian stack.

Orange 4 in laplacian stack.

Oraple 0 in laplacian stack. (This is unclipped)

Oraple 2 in laplacian stack. (This is unclipped)

Oraple 4 in laplacian stack. (This is unclipped)

The reconstructed Oraple, found by collapsing the Oraple laplacian stack!

More Splines

Here are some more image blends for entertainment.
Some of these images are not-quite-so seamless, for which we can only blame the lack of precision in finding two images that map onto each other well, along with an rough filter.

Here is an 'angy' cat.

Here is a dog looking out of a car.

The mask used to isolate the dog's face.

This is the result of blending the three above together.

Here is Obama.

Here is a cat.

The mask used.

This is the result of blending the three above together.