Transcript Computer Vision - RWTH Aachen University

Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Computer Vision – Lecture 19
Tracking with Linear Dynamic Models
26.01.2012
Bastian Leibe
RWTH Aachen
http://www.mmp.rwth-aachen.de
leibe@umic.rwth-aachen.de
Many slides adapted from S. Lazebnik, K. Grauman
Announcements
• Last lecture next Tuesday: Repetition

Summary of all topics in the lecture
“Big picture” and current research directions
Opportunity to ask questions

Please use this opportunity and prepare questions!
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer


2
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Course Outline
•
•
•
•
•
•
•
Image Processing Basics
Segmentation & Grouping
Object Recognition
Local Features & Matching
Object Categorization
3D Reconstruction
Motion and Tracking


Motion and Optical Flow
Tracking with Linear Dynamic Models
• Repetition
B. Leibe
10
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Recap: Estimating Optical Flow
I(x,y,t–1)
I(x,y,t)
• Given two subsequent frames, estimate the apparent
motion field u(x,y) and v(x,y) between them.
• Key assumptions



Brightness constancy: projection of the same point looks the
same in every frame.
Small motion: points do not move very far.
Spatial coherence: points move like their neighbors.
Slide credit: Svetlana Lazebnik
B. Leibe
11
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Recap: The Brightness Constancy Constraint
I(x,y,t–1)
I(x,y,t)
• Brightness Constancy Equation:
I ( x, y, t  1)  I ( x  u( x, y), y  v( x, y), t )
• Linearizing the right hand side using Taylor expansion:
I ( x, y, t 1)  I ( x, y, t )  I x  u( x, y)  I y  v( x, y)
• Hence, I x  u  I y  v  It  0
Spatial derivatives
Slide credit: Svetlana Lazebnik
Temporal derivative
B. Leibe
12
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Solving the Aperture Problem
• Use all pixels in a KK window to get more equations.
• Least squares problem:
• Minimum least squares solution given by solution of
Recall the
Harris detector!
Slide adapted from Svetlana Lazebnik
B. Leibe
13
Recap: Iterative Refinement
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
• Estimate velocity at each
pixel using one iteration of
LK estimation.
• Warp one image toward the
other using the estimated
flow field.
• Refine estimate by repeating
the process.
estimate
update
Initial guess:
Estimate:
x
x0
estimate
update
Initial guess:
Estimate:
x
x0
Initial guess:
Estimate:
estimate
update
• Iterative procedure


Results in subpixel accurate localization.
Converges for small displacements.
Slide adapted from Steve Seitz
B. Leibe
x0
x
14
x0
x
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Recap: Coarse-to-fine Estimation
u=1.25 pixels
u=2.5 pixels
u=5 pixels
Image 1
u=10 pixels
Gaussian pyramid of image 1
Slide credit: Steve Seitz
Image 2
Gaussian pyramid of image 2
B. Leibe
15
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Recap: Coarse-to-fine Estimation
Run iterative L-K
Warp & upsample
Run iterative L-K
.
.
.
Image 1
Image 2
Gaussian pyramid of image 1
Gaussian pyramid of image 2
Slide credit: Steve Seitz
B. Leibe
16
Topics of This Lecture
• Tracking with Dynamics
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer



Detection vs. Tracking
Tracking as probabilistic inference
Prediction and Correction
• Linear Dynamic Models
• The Kalman Filter



Kalman filter for 1D state
General Kalman filter
Limitations
• Practical Issues


Obtaining the observations
Data association
B. Leibe
18
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Detection vs. Tracking
…
t=1
Slide credit: Kristen Grauman
t=2
t=20
B. Leibe
t=21
19
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Detection vs. Tracking
…
t=1
t=2
t=20
t=21
• Detection

We detect the object independently in each frame and can
record its position over time, e.g., based on blob’s centroid or
detection window coordinates.
Slide credit: Kristen Grauman
B. Leibe
20
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Detection vs. Tracking
…
t=1
t=2
t=20
t=21
• Tracking with dynamics:

We use image measurements to estimate the object position,
but also incorporate the position predicted by dynamics, i.e.,
our expectation of the object’s motion pattern.
Slide credit: Kristen Grauman
B. Leibe
21
Tracking with Dynamics
• Key idea
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer

Given a model of expected motion, predict where objects will
occur in next frame, even before seeing the image.
• Goals


Restrict search for the object
Improved estimates since measurement noise is reduced by
trajectory smoothness.
• Assumption: continuous motion patterns



Camera is not moving instantly to new viewpoint.
Objects do not disappear and reappear in different places.
Gradual change in pose between camera and scene.
Slide adapted from S. Lazebnik, K. Grauman
B. Leibe
22
General Model for Tracking
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
• The moving object of interest is characterized by an
underlying state X
• State X gives rise to measurements or observations Y
• At each time t, the state changes to Xt and we get a new
observation Yt
X1
X2
Y1
Y2
Slide credit: Svetlana Lazebnik
…
B. Leibe
Xt
“Hidden”
state
Yt
Observed
measurements
23
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
State vs. Observation
• Hidden state : parameters of interest
• Measurement : what we get to directly observe
Slide credit: Kristen Grauman
B. Leibe
24
Tracking as Inference
• The hidden state consists of the true parameters we
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
care about, denoted X.
• The measurement is our noisy observation that results
from the underlying state, denoted Y.
• At each time step, state changes (from Xt-1 to Xt ) and
we get a new observation Yt.
• Our goal: recover most likely state Xt given


All observations seen so far.
Knowledge about dynamics of state transitions.
Slide credit: Kristen Grauman
B. Leibe
25
Steps of Tracking
• Prediction: What is the next state of the object given
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
past measurements?
PX t Y0  y0 ,, Yt 1  yt 1 
• Correction: Compute an updated estimate of the state
from prediction and measurements.
PX t Y0  y0 ,, Yt 1  yt 1, Yt  yt 
• Tracking can be seen as the process of propagating the
posterior distribution of state given measurements
across time.
B. Leibe
26
Simplifying Assumptions
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
• Only the immediate past matters (“Markov assumption”)
PX t X 0 ,, X t 1   PX t X t 1 
Dynamics model
• Measurements depend only on the current state
PYt X 0 , Y0 , X t 1, Yt 1, X t   PYt X t 
Observation model
Slide credit: Svetlana Lazebnik
X1
X2
Y1
Y2
B. Leibe
…
Xt
Yt
27
Tracking as Induction
• Base case:
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer


Assume we have an initial prior that predicts state in absence of
any evidence: P(X0)
At the first frame, correct this given the value of Y0=y0
P( y0 | X 0 ) P( X 0 )
P( X 0 | Y0  y0 ) 
 P( y0 | X 0 ) P( X 0 )
P( y0 )
Posterior prob.
of state given
measurement
Slide credit: Svetlana Lazebnik
Likelihood of Prior of
measurement the state
B. Leibe
28
Tracking as Induction
• Base case:
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer


Assume we have initial prior that predicts state in absence of
any evidence: P(X0)
At the first frame, correct this given the value of Y0=y0
• Given corrected estimate for frame t:


Predict for frame t+1
Correct for frame t+1
predict correct
Slide credit: Svetlana Lazebnik
B. Leibe
29
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Induction Step: Prediction
• Prediction involves representing PX t y0 ,, yt 1 
given PX t 1 y0 ,, yt 1 
PX t y0 ,, yt 1 
  PX t , X t 1 y0 ,, yt 1 dXt 1
of total
probability
  P XLaw
|
X
,
y
t
t 1
0 , , yt 1 P X t 1 | y0 , , yt 1 dX t 1
P  A    P  A, B  dB
  P X t | X t 1 P X t 1 | y0 ,, yt 1 dXt 1
Slide credit: Svetlana Lazebnik
B. Leibe
30
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Induction Step: Prediction
• Prediction involves representing PX t y0 ,, yt 1 
given PX t 1 y0 ,, yt 1 
PX t y0 ,, yt 1 
  PX t , X t 1 y0 ,, yt 1 dXt 1
  P X t | X t 1 , y0 ,, yt 1 P X t 1 | y0 ,, yt 1 dXt 1
on Xt–1


  P XConditioning
|
X
P
X
t
t 1
t 1 | y0 , , yt 1 dX t 1
P  A, B  P  A | B P  B 
Slide credit: Svetlana Lazebnik
B. Leibe
31
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Induction Step: Prediction
• Prediction involves representing PX t y0 ,, yt 1 
given PX t 1 y0 ,, yt 1 
PX t y0 ,, yt 1 
  PX t , X t 1 y0 ,, yt 1 dXt 1
  P X t | X t 1 , y0 ,, yt 1 P X t 1 | y0 ,, yt 1 dXt 1
  P X t | X t 1 P X t 1 | y0 ,, yt 1 dXt 1
Independence assumption
Slide credit: Svetlana Lazebnik
B. Leibe
32
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Induction Step: Correction
• Correction involves computing PX t y0 ,, yt 
given predicted value PX t y0 ,, yt 1 
PX t y0 ,, yt 
P yt | X t , y0 ,  , yt 1 P X t | y0 ,  , yt 1 

P yt | y0 ,  , yt 1 
P yt | X t P X t Bayes
| y0 , rule
, yt 1 

P yt | y0 ,  , yPt 1B| A P  A
P  A | B 

P yt | X t P X t | y0 ,P B, yt 1 
 P y
t
| X t P X t | y0 ,  , yt 1 dX t
Slide credit: Svetlana Lazebnik
B. Leibe
33
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Induction Step: Correction
• Correction involves computing PX t y0 ,, yt 
given predicted value PX t y0 ,, yt 1 
PX t y0 ,, yt 
P yt | X t , y0 ,  , yt 1 P X t | y0 ,  , yt 1 

P yt | y0 ,  , yt 1 
P yt | X t P X t | y0 ,  , yt 1 

P yt | y0 ,  , yt 1 

P yt |Independence
X t P X t | y0assumption
,  , yt 1 
(observation yt depends only on state Xt)
 P y
t
| X t P X t | y0 ,  , yt 1 dX t
Slide credit: Svetlana Lazebnik
B. Leibe
34
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Induction Step: Correction
• Correction involves computing PX t y0 ,, yt 
given predicted value PX t y0 ,, yt 1 
PX t y0 ,, yt 
P yt | X t , y0 ,  , yt 1 P X t | y0 ,  , yt 1 

P yt | y0 ,  , yt 1 
P yt | X t P X t | y0 ,  , yt 1 

P yt | y0 ,  , yt 1 

P yt | X t P X t | y0 ,  , yt 1 
 P y
t
| X t P X t | y0 ,  , yt 1 dX t
Conditioning on Xt
Slide credit: Svetlana Lazebnik
B. Leibe
35
Summary: Prediction and Correction
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
• Prediction:
P X t | y0 ,  , yt 1    P X t | X t 1 P X t 1 | y0 ,  , yt 1 dX t 1
Dynamics
model
Slide credit: Svetlana Lazebnik
B. Leibe
Corrected estimate
from previous step
36
Summary: Prediction and Correction
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
• Prediction:
P X t | y0 ,  , yt 1    P X t | X t 1 P X t 1 | y0 ,  , yt 1 dX t 1
Dynamics
model
• Correction:
P X t | y0 ,, yt  
Observation
model
Predicted
estimate
P yt | X t P X t | y0 ,, yt 1 
 P y
t
Slide credit: Svetlana Lazebnik
Corrected estimate
from previous step
| X t P X t | y0 ,, yt 1 dXt
B. Leibe
37
Topics of This Lecture
• Tracking with Dynamics
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer



Detection vs. Tracking
Tracking as probabilistic inference
Prediction and Correction
• Linear Dynamic Models
• The Kalman Filter



Kalman filter for 1D state
General Kalman filter
Limitations
• Practical Issues


Obtaining the observations
Data association
B. Leibe
38
Notation Reminder
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
x ~ N (μ, Σ)
• Random variable with Gaussian probability distribution
that has the mean vector μ and covariance matrix Σ.
• x and μ are d-dimensional, Σ is dd.
d=2
d=1
If x is 1D, we
just have one Σ
parameter:
the variance σ2
Slide credit: Kristen Grauman
B. Leibe
39
Linear Dynamic Models
• Dynamics model
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer

State undergoes linear tranformation Dt plus Gaussian noise

xt ~ N Dt xt 1 ,  dt
n1
nn

n1
• Observation model

Measurement is linearly transformed state plus Gaussian noise

yt ~ N M t xt ,  mt
m1
Slide credit: S. Lazebnik, K. Grauman
mn
B. Leibe

n1
40
Example: Randomly Drifting Points
• Consider a stationary object, with state as position.
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer

Position is constant, only motion due to random noise term.
xt  pt
pt  pt 1  
 State evolution is described by identity matrix D=I
xt  Dt xt 1  noise  I pt 1  noise
“Random Walk”
Slide credit: Kristen Grauman
B. Leibe
41
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Example: Constant Velocity (1D Points)
Measurements
States
time
Slide credit: Kristen Grauman
B. Leibe
42
Figure from Forsyth & Ponce
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Example: Constant Velocity (1D Points)
• State vector: position p and velocity v
(greek letters
pt  pt 1  (t )vt 1  
 pt 
denote noise
xt   
vt  vt 1  
terms)
v
 t
1 t   pt 1 
xt  Dt xt 1  noise  
 noise



0 1   vt 1 
• Measurement is position only
 pt 
yt  Mxt  noise  1 0   noise
 vt 
Slide credit: S. Lazebnik, K. Grauman
B. Leibe
43
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Example: Constant Acceleration (1D Points)
Slide credit: Kristen Grauman
B. Leibe
45
Figure from Forsyth & Ponce
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Example: Constant Acceleration (1D Points)
• State vector: position p, velocity v, and acceleration a.
pt  pt 1  (t )vt 1  
(greek letters
 pt 
denote noise


vt  vt 1  (t )at 1  
xt   vt 
terms)
 at 
at  at 1  
1 t
xt  Dt xt 1  noise  0 1
0 0
0   pt 1 
t   vt 1   noise
1   at 1 
• Measurement is position only
 pt 




yt  Mxt  noise  1 0 0  vt   noise
a 
B. Leibe  t 
Slide credit: S. Lazebnik, K. Grauman
46
Example: General Motion Models
• Assuming we have differential equations for the motion
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer

E.g. for (undampened) periodic motion of a spring
d2 p
 cp
2
dt
• Substitute variables to transform this into linear system
p1  p
• Then we have
 p1,t 


xt   p2,t 
 p3,t 
dp
p2 
dt
d2 p
p3  2
dt
p1,t  p1,t 1  (t ) p2,t 1  
p2,t  p2,t 1  (t ) p3,t 1  
p3,t  cp1,t 1  
B. Leibe
 1 t
Dt   0 1
 c 0
0
t 
0 
48
Topics of This Lecture
• Tracking with Dynamics
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer



Detection vs. Tracking
Tracking as probabilistic inference
Prediction and Correction
• Linear Dynamic Models
• The Kalman Filter



Kalman filter for 1D state
General Kalman filter
Limitations
• Practical Issues


Obtaining the observations
Data association
B. Leibe
49
The Kalman Filter
• Optimal method for tracking linear dynamical models
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
under the assumption of Gaussian noise
• The predicted/corrected state distributions are Gaussian


You only need to maintain the mean and covariance.
The calculations are easy (all the integrals can be done in closed
form).
Slide adapted from Svetlana Lazebnik
B. Leibe
50
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
The Kalman Filter
Know corrected state from
previous time step, and all
measurements up to the
current one.
 Predict distribution over
next state.
Receive measurement
Time update
(“Predict”)
Measurement update
(“Correct”)
PX t y0 ,, yt 1 
Mean and std. dev.
of predicted state:
 ,

t
PX t y0 ,, yt 
Time advances: t++

t
Slide credit: Kristen Grauman
Know prediction of state,
and next measurement.
Update distribution
over current state.
Mean and std. dev.
of corrected state:
 ,

t
B. Leibe

t
51
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Kalman Filter for 1D State
Want to
represent
and update

Px y ,, y   N 

) 
Pxt y0 ,, yt 1   N  , ( )

t
t
0
t
B. Leibe

t
 2
t
, (
 2
t
52
Propagation of Gaussian densities
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Shifting the mean
Bayesian combination
Slide credit: Svetlana Lazebnik
Increasing the variance
1D Kalman Filter: Prediction
• Have linear dynamic model defining predicted state
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
evolution, with noise
X t ~ N dxt 1, d2 
• Want to estimate predicted distribution for next state

PX t y0 ,, yt 1   N  , ( )
• Update the mean:

t
  d

t

t 1
• Update the variance:
 2
t

for derivations,
see F&P Chapter 17.3
( )    (d )
 2
t
Slide credit: Kristen Grauman
2
d
B. Leibe
 2
t 1
54
1D Kalman Filter: Correction
• Have linear model defining the mapping of state to

Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
measurements:
Yt ~ N mxt , m2

• Want to estimate corrected distribution given latest
measurement:

PX t y0 ,, yt   N t , ( t )2
• Update the mean:

 2
 2



m
y
(


t
m
t
t )
t 
2
2
 2
 m  m ( t )
• Update the variance:
 ( )
( )  2
 m  m ( )
 2
t
Slide credit: Kristen Grauman
B. Leibe
2
m
 2
t
2
 2
t
55
Derivations: F&P Chapter 17.3
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Prediction vs. Correction
 2
 2



m
y
(


t
m
t
t )
t 
2
2
 2
 m  m ( t )
2
 2

(

 2
m
t )
( t )  2
 m  m2 ( t ) 2
• What if there is no prediction uncertainty ( t  0) ?
 

t

t
( )  0
 2
t
The measurement is ignored!
• What if there is no measurement uncertainty ( m  0) ?
yt
 
m
( )  0

t
 2
t
The prediction is ignored!
Slide credit: Kristen Grauman
B. Leibe
56
position
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Recall: Constant Velocity Example
measurements
state
time
State is 2D: position + velocity
Measurement is 1D: position
Slide credit: Kristen Grauman
B. Leibe
57
Figure from Forsyth & Ponce
Constant Velocity Model
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
o true state
x measurement
* predicted mean estimate
+ corrected mean estimate
bars: variance estimates
before and after
measurements
Slide credit: Kristen Grauman
B. Leibe
58
Figure from Forsyth & Ponce
Constant Velocity Model
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
o true state
x measurement
* predicted mean estimate
+ corrected mean estimate
bars: variance estimates
before and after
measurements
Slide credit: Kristen Grauman
B. Leibe
59
Figure from Forsyth & Ponce
Constant Velocity Model
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
o true state
x measurement
* predicted mean estimate
+ corrected mean estimate
bars: variance estimates
before and after
measurements
Slide credit: Kristen Grauman
B. Leibe
60
Figure from Forsyth & Ponce
Constant Velocity Model
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
o true state
x measurement
* predicted mean estimate
+ corrected mean estimate
bars: variance estimates
before and after
measurements
Slide credit: Kristen Grauman
B. Leibe
61
Figure from Forsyth & Ponce
Kalman Filter: General Case (>1dim)
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
• What if state vectors have more than one dimension?
PREDICT
CORRECT

t

t

t 1


t
K t   M M t  M   mt
xt  Dt xt1
T
t
“Kalman gain”
  Dt  D  dt
T
t

T
t
xt  xt  Kt yt  M t xt
t  I  Kt M t t


1
“residual”
More weight on residual
when measurement error
covariance approaches 0.
Less weight on residual as
a priori estimate error
covariance approaches 0.
for derivations,
see F&P Chapter 17.3
Slide credit: Kristen Grauman
B. Leibe
62
Kalman Filter: Basic Algorithm
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
• Initialize initial prediction
x0  x0 , 0  0
• For each time step t
1. Perform data association of observations to tracks
– E.g. select the observation with the highest prediction posterior
(= lowest Mahalanobis distance to predicted position)
yt  yt , j with j  arg min  x  yt ,i   

t 1
T

1

t 1
i

x
 t 1  yt ,i 
2. Correct estimate from previous step using new measurement
x  x  Kt  yt  M x  ,

t

t
t  I  Kt M t t

t t

t


t
K t   M M t  M   mt
T
t
T
t

1
3. Make prediction for next time step
xt  Dt xt1
t  Dt t1DtT  d
t
B. Leibe
63
Summary: Kalman Filter
• Pros:
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer




Gaussian densities everywhere
Simple updates, compact and efficient
Very established method, very well understood
Optimal for linear dynamic models under Gaussian noise
• Cons:


Unimodal distribution, only single hypothesis
Restricted class of motions defined by linear model
Slide adapted from Svetlana Lazebnik
B. Leibe
64
Why Is This A Restriction?
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
• Many interesting cases don’t have linear dynamics

E.g. pedestrians walking

E.g. a ball bouncing
B. Leibe
65
Ball Example: What Goes Wrong Here?
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
• Assuming constant acceleration model
Prediction t1
Prediction t2
• Prediction is too far
from true position to
compensate…
Prediction t3
• Possible solution:



Prediction t4
Keep multiple different
motion models in parallel
Decide for one at each time step
I.e. would check for
bouncing and continue
Prediction t5
B. Leibe
Correct
prediction
66
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Tracking – Difficulties
Video
B. Leibe
67
Figure from Michael Breitenstein
Resources: Kalman Filter Web Site
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
http://www.cs.unc.edu/~welch/kalman
• Electronic and printed references




Book lists and recommendations
Research papers
Links to other sites
Some software
• News
• Java-Based KF Learning Tool



On-line 1D simulation
Linear and non-linear
Variable dynamics
Slide adapted from Greg Welch
B. Leibe
68
Remarks
• Try it!
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer

Not too hard to understand or program
• Start simple


Experiment in 1D
Make your own filter in Matlab, etc.
• Note: the Kalman filter “wants to work”


Debugging can be difficult
Errors can go unnoticed
• Try it yourself in exercise 7.3!
69
Slide adapted from Greg Welch
Topics of This Lecture
• Tracking with Dynamics
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer



Detection vs. Tracking
Tracking as probabilistic inference
Prediction and Correction
• Linear Dynamic Models
• The Kalman Filter



Kalman filter for 1D state
General Kalman filter
Limitations
• Practical Issues


Obtaining the observations
Data association
B. Leibe
70
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Obtaining the Observations
• Up to now, I’ve only talked about the tracking part.
• How do we get the object measurements?
• Some possible approaches



Background modelling
Tracking lines & contours
Tracking-by-detection
B. Leibe
71
Background Modeling for Tracking
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
• Segmentation based tracking
• Object separation

Specialized models for each
foreground object and for
the background
• Model separation


Appearance
Motion (Tracking)
• Bayesian per-pixel classification
Slide adapted from Daniel Roth
B. Leibe
72
Background Color Model
• Gaussian mixture for each pixel
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer


Model “common“ appearance
variation for each pixel separately.
Update the mixtures over time
Gaussian
Mixture
– Adapt to lighting changes, etc.


Easiest when an empty scene is
first seen for some frames.
But can also be applied for online
learning.
• De-facto standard for many tracking applications


With fixed-camera scenarios
With limited background motion
C. Stauffer, E. Grimson, Learning Patterns of Activity Using Real-Time Tracking,
IEEE Trans. PAMI, 22(8):747-757, 2000.
B. Leibe
73
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Example Results with Background Modelling
B. Leibe
74
Video by Daniel Roth
Tracking Lines
• Track lines on the object
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
given a 3D model
• Basic idea



Only have to search along
the gradient direction.
Sufficient to sample only
a few locations
Very fast search possible
B. Leibe
75
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Example Application: Face Tracking
L. Vacchetti, V. Lepetit, P. Fua, Stable Real-Time 3D Tracking Using Online and
Offline Information, IEEE Trans. PAMI, 26(10), pp. 1385-1391, 2004.
B. Leibe
76
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Tracking by Detection
• Spacetime trajectory analysis



Collect detections in spacetime volume (ground position + time)
Link up detections to form physically plausible ST trajectories
Select set of ST trajectories that best explain the data
B. Leibe
77
[Leibe, Cornelis, Cornelis, Van Gool, CVPR’07]
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Tracking by Detection
3D through SfM
Object detections
Spacetime trajectories
• Spacetime trajectory analysis



Collect detections in spacetime volume (ground position + time)
Link up detections to form physically plausible ST trajectories
(independent Kalman filters in 3D world coordinates)
Select set of ST trajectories that best explain the data
[Leibe, Cornelis, Cornelis, Van Gool, CVPR’07]
78
Spacetime Trajectory Analysis
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
pedestrian detection
car detections
own vehicle
B. Leibe
79
[Leibe, Cornelis, Cornelis, Van Gool, CVPR’07]
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
Mobile Tracking Through Crowds
[Ess, Leibe, Schindler, Van Gool, CVPR’08]
80
Data Association
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
• So far, we’ve assumed the
entire measurement to be
relevant to determining the
state.
• In reality, there may be
uninformative measurements
(clutter) or measurements may
belong to different tracked
objects.
• Data association: task of
determining which
measurements go with which
tracks.
Slide credit: Svetlana Lazebnik
B. Leibe
81
Data Association
• Simple strategy: only pay attention to the measurement
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
that is “closest” to the prediction
Slide credit: Svetlana Lazebnik
B. Leibe
82
Data Association
• Simple strategy: only pay attention to the measurement
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
that is “closest” to the prediction
Doesn’t always work…
Slide credit: Svetlana Lazebnik
B. Leibe
83
Data Association
• Simple strategy: only pay attention to the measurement
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
that is “closest” to the prediction.
• More sophisticated strategy: keep track of multiple
state/observation hypotheses


Can be done with particle filtering
But leads to state space explosion (many particles needed)
• This is a general problem in computer vision, there is no
easy solution.
• More sophisticated approaches available



Multi Hypothesis Tracking (MHT)
Joint Probabilistic Data Association Filters (JPDAF)
Can typically only handle few states or become very expensive
B. Leibe
84
References and Further Reading
• A very good introduction to tracking with linear dynamic
Augmented Computing
and Sensory
PerceptualVision
WS 11/12
Computer
models and Kalman filters can be found in Chapter 17 of

D. Forsyth, J. Ponce,
Computer Vision – A Modern Approach.
Prentice Hall, 2003
B. Leibe
85