Transcript PowerPoint - People

Joint Noise Level Estimation from Personal Photo Collections
YiChang
1*,2
Shih
1
Kwatra
Vivek
Troy
1Google Research
1
Chinen
Hui
2MIT CSAIL
1
Fang
Sergey
*Internship work at Google
Goal
Contributions
 Given a set of face images from the same person, taken
under different lighting and cameras, estimate the noise
levels in each image
 Key observation: given two noisy images, the noise
levels are correlated if they share the same underlying
2
2
image content, since 𝜎1 − 𝜎2 = 𝑣𝑎𝑟[𝑰𝒏,𝟏 ] − 𝑣𝑎𝑟[𝑰𝒏,𝟐 ]
Overview
Starting from a face image collection:
 Preprocess: geometrically and photometrically align the
images with affine transform and color match
 We formulate the estimation as maximizing the joint
probability distribution between all images’ noise levels
 The joint distribution is conditioned on the pair-wise
2
2
relative noise levels {𝜌𝑖𝑗 |𝜌𝑖𝑗 ≜ 𝜎𝑖 − 𝜎𝑗 }. We use a twostage optimization that first estimates {𝜌𝑖𝑗 }, then {𝜎𝑖 }
 𝑰𝑛 = 𝑰𝑜𝑟𝑖𝑔 + 𝒏, i.i.d, zero mean. 𝜎 = noise level ≜ 𝑠𝑡𝑑[𝒏]
 This is difficult because we cannot decouple 𝒏 from 𝑰𝑛
1
Ioffe
 Two-stage optimization:
 Estimating {𝜌𝑖𝑗 }: We take a patch-based method. We
first find the patch correspondence between 𝑰𝒊 and 𝑰𝒋 ,
∗
then find the best estimated relative noise {𝜌𝑖𝑗 } from
the patch pairs.
∗
2
2
∗
 With {𝜌𝑖𝑗 }, estimate {𝜎𝑖 } by constraining 𝜎𝑖 − 𝜎𝑗 = 𝜌𝑖𝑗
Pair-wise Relative Noise 𝜌𝑖𝑗 Estimation
Results
Ground Truth Experiment and Comparison
 The two faces are not perfectly aligned
 We show one example below with estimated
noise levels and denoised result using BM3D +
our method for noise parameter
 Add synthetic Gaussian noise with different parameters
 Compare estimated noise levels and denoised result by BM3D
q
• 𝑐𝑝𝑞 = exp(−𝜅𝑝𝑞 𝒑1𝑝 − 𝒑2𝑞
2
Input
BM3D
Input
y=x Line (Ground Truth)
Liu et. al. (Mean)
Metric-Q (Mean)
Our Method (Mean)
BM3D
Mean PSNR (Metric-Q)
Mean PSNR (Our Method)
Mean PSNR(Best BM3D)
30
I2
∗
𝜌12
=
σ=23.2
σ=14.5
𝑝,𝑞 𝑐𝑝𝑞 𝜁𝑝𝑞
𝑝,𝑞 𝑐𝑝𝑞
), confidence that (𝑝, 𝑞) is a true correspondence
26
40.00
22
18
14
10
6
 More subjects
38.00
37.00
36.00
35.00
33.00
2
σ=38.2
39.00
34.00
2
σ=23.9
• For computational efficiency, we selected the best 5 𝑞 s for each 𝑝
41.00
Mean PSNR
I1
 Compute pair-wise relative noise by aggregating 𝜁𝑝𝑞 :
q
Estimated Noise Sigma
 We break down the image into patches, and
estimate the patch-wise relative noise
levels 𝜁𝑝𝑞 by 𝜁𝑝𝑞 ≜ 𝑣𝑎𝑟[𝒑1𝑝 ] − 𝑣𝑎𝑟[𝒑2𝑞 ]
p
7
12
0.00
17
Synthetic noise
10.00
15.00
20.00
Mean Noise Sigma
True Noise Sigma
Clean image
5.00
Our method
Metric-Q
Best BM3D
Absolute Noise Level Estimation with Global Optimization
 We estimate {𝜎𝑖 } conditioning on

2
𝜎𝑖
2
𝑖≠𝑗 𝑤𝑖𝑗 𝜎𝑖
∗
{𝜌𝑖𝑗 }
2
𝜎𝑗
=argmin
−
−
𝑤𝑖𝑗 : similarity between two faces
∗ 2
𝜌𝑖𝑗
User Study
2
𝜎𝑖
∗
𝜌𝑖𝑗
 Solving a linear system
2
𝜎𝑗
 The system is under-determined, up to adding
a constant number.
- option 1: assign some images to be zero noise
- option 2: assuming the collection contains clean
images, assign the least noisy one to be zero. We use this one for evaluations
 Based on BM3D denoised result, decide which
one is preferable
 Ran on 71 images, each is evaluated by 3 users
Input image
Our method
Metric-Q
35%
24%
41%
σ=46.3
σ=35
Metric-Q
σ
PSNR
19.25
22.33
20.31
34.65
27
34.53
23
34.65
Selected References
C. Liu, R. Szeliski, S. Kang, C. Zitnick, and W. Freeman. Automatic estimation and removal of
noise from a single image. IEEE Transactions on Pattern Analysis and Machine Intelligence,
30(2), 2008
X. Zhu and P. Milanfar. Automatic parameter selection for denoising algorithms using a noreference measure of image content. IEEE Transactions on Image Processing, 19(12), 2010.
Acknowledgements
Ours
Equally good
σ=9.1
σ=27
We thank MIT Graphics and Vision group for helpful discussion. We would like to thank the
volunteers who participated in the user study.