Transcript Computer Vision - RWTH Aachen University

Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Computer Vision
Binary Image Analysis
23.10.2008
Bastian Leibe
RWTH Aachen
http://www.umic.rwth-aachen.de/multimedia
leibe@umic.rwth-aachen.de
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Binary Images
• Just two pixel values
• Foreground and background
• Regions of interest
Slide credit: Kristen Grauman
B. Leibe
2
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Uses: Industrial Inspection
R. Nagarajan et al. “A real time marking inspection scheme
B. Leibe
for semiconductor industries“, 2006
3
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Uses: Document Analysis, Text Recognition
Handwritten digits
Natural text (after detection)
Scanned documents
B. Leibe
4
Source: Till Quack, Martin Renold
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Uses: Medical/Bio Data
Source: D. Kim et al., Cytometry 35(1), 1999
B. Leibe
5
Uses: Blob Tracking & Motion Analysis
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Frame Differencing


Source: Kristen Grauman
Background Subtraction


Source: Tobias Jäggli
B. Leibe
6
Uses: Shape Analysis, Free-Viewpoint Video
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Visual Hull Reconstruction
Silhouette
Blue-c project, ETH Zurich
Medial axis
B. Leibe
7
Uses: Intensity Based Detection
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
• Looking for dark pixels…
fg_pix = find(im < 65);
Slide Credit: Kristen Grauman
B. Leibe
8
Uses: Color Based Detection
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
• Looking for pixels within a certain color range…
fg_pix = find(hue > t1 & hue < t2);
Slide Credit: Kristen Grauman
B. Leibe
9
Issues
• How to demarcate multiple
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
regions of interest?


Count objects
Compute further features per object
• What to do with “noisy”
binary outputs?


Holes
Extra small fragments
Slide Credit: Kristen Grauman
B. Leibe
10
Outline of Today’s Lecture
• Convert the image into binary form
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer

Thresholding
• Clean up the thresholded image

Morphological operators
• Extract individual objects

Connected Components Labeling
• Describe the objects

Region properties
B. Leibe
11
Image Source: D. Kim et al., Cytometry 35(1), 1999
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Thresholding
• Grayscale image  Binary mask
• Different variants

One-sided

Two-sided

Set membership
1, if F i, j   T
FT i, j   
0, otherwise
1, if T1  F i, j   T2
FT i, j   
0, otherwise
1, if F i, j  Z
FT i, j   
0, otherwise
B. Leibe
12
Image Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/
Selecting Thresholds
• Typical scenario
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer

Separate an object from a distinct background
• Try to separate the different grayvalue distributions



Partition a bimodal histogram
Fit a parametric distribution (e.g. Mixture of Gaussians)
Dynamic or local thresholds
• In the following, I will present some simple methods.
• We will see some more general methods in Lecture 6…
B. Leibe
13
A Nice Case: Bimodal Intensity Histograms
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Ideal histogram,
light object on
dark background
Actual observed
histogram with
noise
Source: Robyn Owens
B. Leibe
14
Not so Nice Cases…
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
• How to separate those?
Two distinct modes
Overlapping modes
Multiple modes
• Threshold selection is difficult in the general case



Domain knowledge often helps
E.g. Fraction of text on a document page ( histogram quantile)
E.g. Size of objects/structure elements
Source: Shapiro & Stockman
B. Leibe
15
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Global Binarization [Ohta’79]
• Search for the threshold T that minimizes the withinclass variance within of the two classes separated by T
2
 within
(T )  n1 (T ) 12 (T )  n2 (T ) 22 (T )
where
n1 (T )  I ( x , y )  T ,
n2 (T )  I ( x , y )  T 
• This is the same as maximizing the between-class
variance between

2
between
(T )    
2
2
within
(T )
 n1 (T )n2 (T )1 (T )   2 (T )
2
B. Leibe
16
Algorithm
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Precompute a cumulative grayvalue histogram h.
For each potential threshold T
1.) Separate the pixels into two clusters according to T
2.) Look up n1, n2 in h and compute both cluster means
2
3.) Compute  between
Choose
2
T   arg max T [σ between
(T)]
B. Leibe
17
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Local Binarization [Niblack’86]
• Estimate a local threshold within a small neighborhood
window W
TW
TW  W  k   W
Effect:
where k [-1,0] is a user-defined parameter.
• Improved version to suppress background noise for
document binarization [Sauvola’00]

 W

TW  W 1  k  
 1
 R


Useful for text,
but not for larger objects
where R is the dynamic range of  and k > 0.
 Typical values: R=128 for 8-bit images and k 0.5.
B. Leibe
18
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Effects
Global threshold selection
(Ohta)
Original image
Local threshold selection
(Niblack)
B. Leibe
19
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Additional Improvements
• Document images often contain a smooth gradient
Try to fit that gradient with a polynomial function
Original image
Fitted surface
Shading compensation
Source: S. Lu & C. Tan, ICDAR’07
B. Leibe
Binarized result
20
Surface Fitting
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
• Polynomial surface of degree d
f ( x, y ) 
d
b
i  j 0
i, j
i
xy
j
• Least-squares estimation, e.g. for d=3 (m=10)
1

1


1
x0 y0
x1 y1
 
xn y n
Ab  I
Solution with
x02 x0 y0  y03   b0   I 0  pseudo-inverse:
  
2
3 
x1 x1 y1  y1   b1   I1  b  ( AT A) 1 AT I

,
         
    Matlab (using SVD):
2
3 
xn xn yn  yn  bm   I n 
bI \A
B. Leibe
21
Surface Fitting
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
• Iterative Algorithm
1.) Fit parametric surface to all points in region.
2.) Subtract estimated surface.
3.) Apply global threshold (e.g. with Ohta method)
4.) Fit surface to all background pixels in original region.
5.) Subtract estimated surface.
5.) Apply global threshold (Ohta)
Initial
guess
Refined
guess
6.) Iterate further if needed…
• The first pass also takes foreground pixels into account.


This is corrected in the following passes.
Basic assumption here: most pixels belong to the background.
B. Leibe
22
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Result Comparison
Original image
Global (Ohta)
Local (Sauvola)
Polynomial
+ Global
Source: S. Lu & C. Tan, ICDAR’07
B. Leibe
24
Outline of Today’s Lecture
• Convert the image into binary form
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer

Thresholding
• Clean up the thresholded image

Morphological operators
• Extract individual objects

Connected Components Labeling
• Describe the objects

Region properties
B. Leibe
25
Image Source: D. Kim et al., Cytometry 35(1), 1999
Cleaning the Binarized Results
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
• Results of thresholding often still contain noise
• Necessary cleaning operations


Remove isolated points and small structures
Fill holes
 Morphological Operators
B. Leibe
26
Image Source: D. Kim et al., Cytometry 35(1), 1999
Morphological Operators
• Basic idea
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer


Scan the image with a structuring element
Perform set operations (intersection, union)
of image content with structuring element
Matlab:
>> help strel
• Two basic operations


Dilation (Matlab: imdilate)
Erosion (Matlab: imerode)
• Several important combinations

Opening (Matlab: imopen)
Closing (Matlab: imclose)

Boundary extraction

B. Leibe
27
Image Source: R.C. Gonzales & R.E. Woods
.
Dilation
• Definition
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer


“The dilation of A by B is the set
of all displacements z, such that
ˆ and A overlap by at least one
( B)
z
element”.
B1
A
A  B1
ˆ is the mirrored version of B,
(( B)
z
shifted by z)
• Effects
If current pixel z is foreground, set all
pixels under (B)z to foreground.
 Expand connected components
 Grow features
 Fill holes

B. Leibe
B2
A  B2
28
Image Source: R.C. Gonzales & R.E. Woods
Erosion
• Definition
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer

“The erosion of A by B is the set
of all displacements z, such that
( B) z is entirely contained in A”.
B1
A
1
• Effects
If not every pixel under (B)z is
foreground, set the current pixel z
to background.
 Erode connected components
 Shrink features
 Remove bridges, branches, noise

B. Leibe
B2
2
29
Image Source: R.C. Gonzales & R.E. Woods
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Effects
Dilation with circular
structuring element
Original image
Erosion with circular
structuring element
B. Leibe
30
Image Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/
Opening
• Definition
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer

Sequence of Erosion and Dilation
• Effect
is defined by the points that
are reached if B is rolled around
inside A.
 Remove small objects,
keep original shape.

B. Leibe
31
Image Source: R.C. Gonzales & R.E. Woods
Effect of Opening
• Feature selection through size
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
of structuring element
Opening with small
structuring element
Original image
Thresholded
Image Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/ B. Leibe
Opening with larger
structuring element
32
Effect of Opening
• Feature selection through shape
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
of structuring element
Input Image
B. Leibe
Opening with circular
structuring element
33
Image Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/
Closing
• Definition
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer

Sequence of Dilation and Erosion
• Effect
is defined by the points that
are reached if B is rolled around
on the outside of A.
 Fill holes,
keep original shape.

B. Leibe
34
Image Source: R.C. Gonzales & R.E. Woods
Effect of Closing
• Fill holes in thresholded image
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
(e.g. due to specularities)
Original image
Thresholded
Closing with circular
structuring element
Size of structuring
element determines
which structures are
selectively filled.
B. Leibe
35
Image Source: http://homepages.inf.ed.ac.uk/rbf/HIPR2/
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Example Application: Opening + Closing
Original image
Opening
Closing
Structuring
element
Eroded image
Dilated image
B. Leibe
36
Source: R.C. Gonzales & R.E. Woods
Morphological Boundary Extraction
• Definition
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer

First erode A by B, then subtract
the result from the original A.
 ( A)  A  ( A  B)
• Effects

If a 33 structuring element is used,
this results in a boundary that is
exactly 1 pixel thick.
B. Leibe
37
Source: R.C. Gonzales & R.E. Woods
Morphology Operators on Grayscale Images
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
• Dilation and erosion typically performed on binary
images.
• If image is grayscale: for dilation take the neighborhood
max, for erosion take the min.
Original
Slide credit: Kristen Grauman
Dilated
B. Leibe
Eroded
38
Outline of Today’s Lecture
• Convert the image into binary form
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer

Thresholding
• Clean up the thresholded image

Morphological operators
• Extract individual objects

Connected Components Labeling
• Describe the objects

Region properties
B. Leibe
39
Image Source: D. Kim et al., Cytometry 35(1), 1999
Connected Components Labeling
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
• Goal: Identify distinct regions
Binary image
Sources: Shapiro & Stockman, Chandra
Connected components
labeling
B. Leibe
40
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Connected Components Examples
Source: Pinar Duygulu
B. Leibe
41
Connectedness
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
• Which pixels are considered neighbors?
4-connected
Source: Chaitanya Chandra
8-connected
B. Leibe
42
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Sequential Connected Components
Slide credit: J. Neira
B. Leibe
43
Sequential Connected Components (2)
• Process the image from left to
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
right, top to bottom:
1.) If the next pixel to process is 1
i.) If only one of its neighbors
(top or left) is 1, copy its label.
ii.) If both are 1 and have the
same label, copy it.
iii.) If they have different labels
 Copy the label from the left.
 Update the equivalence table.
iv.) Otherwise, assign a new label.
• Re-label with the smallest of equivalent
labels
Slide credit: J. Neira
B. Leibe
44
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Application: Blob Tracking
Absolute differences from frame to frame
Slide credit: K. Grauman
B. Leibe
45
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Thresholding
Slide credit: K. Grauman
B. Leibe
46
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Eroding
Slide credit: K. Grauman
B. Leibe
47
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Application: Segmentation of a Liver
Slide credit: Li Shen
B. Leibe
48
Outline of Today’s Lecture
• Convert the image into binary form
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer

Thresholding
• Clean up the thresholded image

Morphological operators
• Extract individual objects

Connected Components Labeling
• Describe the objects

Region properties
B. Leibe
49
Image Source: D. Kim et al., Cytometry 35(1), 1999
Region Properties
• From the previous steps, we can
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
obtain separated objects.
• Some useful features can be
extracted once we have connected
components, including





Area
Centroid
Extremal points, bounding box
Circularity
Spatial moments
B. Leibe
50
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Area and Centroid
• We denote the set of pixels in a region by R
• Assuming square pixels, we obtain

Area:
1
A
( x , y )R

Centroid:
x
y
Source: Shapiro & Stockman
1
A
1
A
x
( x , y )R
y
( x , y )R
B. Leibe
51
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Circularity
• Measure the deviation from a perfect circle
R
( x, y )
Circularity:
C
R

where  R and  R2 are the mean and variance of the distance from the centroid of
the shape to the boundary pixels (xk,yk).

Mean radial distance:
R 

1
K
K 1
 ( x , y )  ( x, y)
k
k 0
k
Variance of radial distance:
 
2
R
Source: Shapiro & Stockman
K 1
1
K
  (x , y )  (x, y)   
k 0
2
k
B. Leibe
k
R
52
Invariant Descriptors
• Often, we want features independent of
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
location, orientation, scale.
a1, a2 , a3 ,
Slide credit: Kristen Grauman
b1, b2 , b3 ,
B. Leibe
Feature space
distance
53
Central Moments
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
• S is a subset of pixels (region).
• Central (j,k)th moment defined as:
 jk 
j
k
(
x

x
)
(
y

y
)

( x , y )S
• Invariant to translation of S.
• Interpretation:




0th central moment:
2nd central moment:
3rd central moment:
4th central moment:
Slide credit: Kristen Grauman
area
variance
skewness
kurtosis
B. Leibe
54
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Moment Invariants
• Normalized central moments
 pq
pq
 pq   ,

1
00
2
• From those, a set of invariant moments can be defined
for object description.
1  20  02
2  ( 20   02 ) 2  4112
3  (30  312 ) 2  (3 21  03 ) 2
4  (30  12 ) 2  ( 21  03 ) 2
(Additional invariant
moments 5, 6, 7
can be found in the
literature).
• Robust to translation, rotation & scaling,
but don’t expect wonders (still summary statistics).
B. Leibe
55
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Moment Invariants
5  (30  312 )(30  12 )(30  12 ) 2  3(21  03 ) 2 

 (321 03 )(21  03 ) 3(30  12 ) 2  (21  03 ) 2
)(
6  (20 02 30  12 )  (21  03 )
 411 (30  12 )(21  03 )
2
2


7  (321 03 )(30  12 )(30  12 ) 2  3(21  03 ) 2 

 (312 30 )(21  03 ) 3(30  12 ) 2  (21  03 ) 2
B. Leibe

56
Axis of Least Second Moment
• Invariance to orientation?
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer

Need a common alignment
Axis for which the
squared distance to 2D
object points is minimized
(maximized).

Compute Eigenvectors of 2nd moment matrix (Matlab: eig(A))
  20

 11
11 
 v11 v12  1 0   v11 v12 
T
 VDV  

  0   v

02 
v
v
v
2   21
22 
 22 22  
B. Leibe
T
57
Summary: Binary Image Processing
• Pros
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer



Fast to compute, easy to store
Simple processing techniques
Can be very useful for constrained scenarios
• Cons




Hard to get “clean” silhouettes
Noise is common in realistic scenarios
Can be too coarse a representation
Cannot deal with 3D changes
Slide credit: Kristen Grauman
B. Leibe
58
References and Further Reading
• More on morphological operators can be found in
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer

R.C. Gonzales, R.E. Woods,
Digital Image Processing.
Prentice Hall, 2001
• Online tutorial and Java demos available on

http://homepages.inf.ed.ac.uk/rbf/HIPR2/
B. Leibe
59
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Questions ?
B. Leibe
60
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Matlab Intro: Everything is a matrix
Tutorial adapted from W. Freeman, MIT
61
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Matlab Intro: Matrix Index
Tutorial adapted from W. Freeman, MIT
62
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Matlab Intro: Manipulating Matrices
Tutorial adapted from W. Freeman, MIT
63
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Matlab Intro: Manipulating Matrices
Tutorial adapted from W. Freeman, MIT
B. Leibe
64
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Matlab Intro: Scripts and Functions
• Scripts are m-files containing MATLAB statements
• Functions are like any other m-file, but they accept
arguments
• Name the function file the same as the function name
Tutorial adapted from W. Freeman, MIT
B. Leibe
65
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Matlab Intro: Try to Code in Matrix Ways
B. Leibe
66
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Matlab Intro: Important Commands
•
•
•
•
•
•
•
whos
 List variables in workspace
help
 Get help for any command
lookfor
 Search for keywords
clear/clear x  Erase a variable/all variables
save
 Save the workspace
load
 Load a saved workspace
keyboard
 Enter debugging mode (until dbquit)
B. Leibe
67
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Morphology in Matlab
•
•
•
•
•
•
•
N = hist(Y,M)
IM2 = imerode(IM,SE);
IM2 = imdilate(IM,SE);
IM2 = imclose(IM, SE);
IM2 = imopen(IM, SE);
L = bwlabel(BW,N);
STATS = regionprops(L,PROPERTIES) ;




'Area'
'Centroid'
'BoundingBox'
'Orientation‘, …
Slide credit: Kristen Grauman
Augmented Computing
and Sensory
PerceptualVision
WS 08/09
Computer
Questions ?
B. Leibe
69