Lingbing Jiang

Ting Liu

10

What we do

1. Optical flow estimation

2. Transformation restoration

3. Image sequence stabilization

9

Optical Flow

Classical formulation:

$E(\mathbf{u},\mathbf{v})=E_D(x,y,u_{x,y},v_{x,y})+\lambda E_S(x,y,u_{x,y},v_{x,y})$

$E_D(x,y,u_{x,y},v_{x,y})=\sum_{x,y}f_D\left(I_1(x,y)-I_2(x+u_{x,y},y+v_{x,y})\right)$

$\begin{align} E_S(x,y,u_{x,y},v_{x,y})=&\sum_{x,y}[f_S(u_{x,y}-u_{x+1,y})+f_S(u_{x,y}-u_{x,y+1})\nonumber\label{eq:of_smooth_term}\\ &+f_S(v_{x,y}-v_{x+1,y})+f_S(v_{x,y}-v_{x,y+1})] \end{align}$

8

Optical Flow

Add non-local term:

$\begin{align} E(\mathbf{u},\mathbf{v})=&E_D(x,y,u_{x,y},v_{x,y})+\lambda E_S(x,y,u_{x,y},v_{x,y})\nonumber\\ &+\lambda_2 E_C(\mathbf{u},\mathbf{v},\mathbf{\hat{u}},\mathbf{\hat{v}})+\lambda_3 E_{NL}(\mathbf{u},\mathbf{v},\mathbf{\hat{u}},\mathbf{\hat{v}}) \end{align}$

$E_C=\Vert\mathbf{u}-\mathbf{\hat{u}}\Vert^2+\Vert\mathbf{v}-\mathbf{\hat{v}}\Vert^2$

$E_{NL}=\sum_{x,y}\sum_{(x',y')\in N_{x,y}}\left(|\hat{u}_{x,y}-\hat{u}_{x',y'}|+|\hat{v}_{x,y}-\hat{v}_{x',y'}|\right)$

$\hat{u}^{(k+1)}_{x,y}=\text{median}(N^{(k)}\cup D)$

$\hat{\mathbf{u}}^{(0)}=\mathbf{u}\qquad\hat{u}^{(1)}_{x,y}\approx\text{median}(N^{(0)}\cup\{u_{x,y}\})$

D. Sun, S. Roth and M.J. Black. Secrets of optical flow estimation and their principles, CVPR 2010

7

Optical Flow

Alternate minimization:

$\begin{align} E_O(\mathbf{u},\mathbf{v})=&E_D(x,y,u_{x,y},v_{x,y})+\lambda E_S(x,y,u_{x,y},v_{x,y})\nonumber\\&+\lambda_2 E_C(\mathbf{u},\mathbf{v},\mathbf{\hat{u}},\mathbf{\hat{v}})\end{align}$

$\begin{align} E_M(\hat{\mathbf{u}},\hat{\mathbf{v}})=&\lambda_2 E_C(\mathbf{u},\mathbf{v},\mathbf{\hat{u}},\mathbf{\hat{v}})+\lambda_3 E_{NL}(\mathbf{u},\mathbf{v},\mathbf{\hat{u}},\mathbf{\hat{v}}) \end{align}$

D. Sun, S. Roth and M.J. Black. Secrets of optical flow estimation and their principles, CVPR 2010

6

Optical Flow

5

Transformation

Affine transformation:

$w\left[\begin{array}{c}x_2\\y_2\\1\end{array}\right]=\left[\begin{array}{ccc}p_{11}&p_{12}&p_{13}\\p_{21}&p_{22}&p_{23}\\p_{31}&p_{32}&1\end{array}\right]\left[\begin{array}{c}x_1\\y_1\\1\end{array}\right]$

4

Transformation

$\left[ \begin{array}{cccccccc} x_{11} & y_{11} & 1 & 0 & 0 & 0 & -x_{12}x_{11} & -x_{12}y_{11}\\ 0 & 0 & 0 & x_{11} & y_{11} & 1 & -y_{12}x_{11} & -y_{12}y_{11}\\ x_{21} & y_{21} & 1 & 0 & 0 & 0 & -x_{22}x_{21} & -x_{22}y_{21}\\ 0 & 0 & 0 & x_{21} & y_{21} & 1 & -y_{22}x_{21} & -y_{22}y_{21}\\ \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots & \vdots\\ x_{n1} & y_{n1} & 1 & 0 & 0 & 0 & -x_{n2}x_{n1} & -x_{n2}y_{n1}\\ 0 & 0 & 0 & x_{n1} & y_{n1} & 1 & -y_{n2}x_{n1} & -y_{n2}y_{n1} \end{array} \right]\left[\begin{array}{c}p_{11}\\p_{12}\\p_{13}\\p_{21}\\p_{22}\\p_{23}\\p_{31}\\p_{32}\end{array}\right]= \left[\begin{array}{c}x_{12}\\y_{12}\\x_{22}\\y_{22}\\\vdots\\x_{n2}\\y_{n2}\end{array}\right]$

$\mathbf{A}\mathbf{p}=\mathbf{B}\\\Rightarrow \mathbf{p}=(\mathbf{A}^{\top}\mathbf{A})^{-1}\mathbf{A}^{\top}\mathbf{B}$

3

Stabilization

$K$-frame piecewise smoothing:

$I_1\xrightarrow{P_{1,2}}I_2\xrightarrow{P_{2,3}}I_3\xrightarrow{}\cdots$

$T_m=\sqrt[k]{P_{1,2}\cdot P_{2,3}\cdot\cdots}$

$I_2^\prime=T_m\cdot P_{1,2}^{-1}(I_2)$

$I_3^\prime=T_m^2\cdot (P_{1,2}^{-1}P_{2,3}^{-1})(I_3)$

2

Results

Easy

2

Results

Medium

2

Results

Hard

2

Results

Experimental

1

Thank You!

Powered by impress.js^*and mathjax