# What is Optical Flow?

Optical flow, or optic flow, is the apparent motion of objects as perceived by an observer. As it can be understood, this motion can be due to actual movement of the observed objects, movement of the observer itself (egomotion), or a combination of both. By observer here we refer to either an eye or a camera. The term optical flow was introduced by an American psychologist James J. Gibson in 1940s. In computer vision optical flow typically refers to movement observed in the image plane defined by a vector $$[u,v]$$ where the first component describes horizontal movement while the second component describes vertical movement. Optical flow is close related to a process known as image registration. One might be left wondering where one would actually even use this kind of information. Following are just a few examples for use of optical flow:

• Video compression from motion estimation
• Tracking of objects in videos
• Tracking of objects in robotics
• ADAS (advanced driving assistance systems), see the example at the end
• ...and so on...

## Visualization Of Optical Flow Field

Optical flow can be visualized using vectors, but it the vectors are to be shown for each pixel then this kind of visualization quickly becomes unreadable. Another possibility is to codify the movement direction using colours while intensity codifies magnitude of the movement. Typically a border showing the colour codification is drawn around the optical flow field. The image below shows one possible colour codification.

## Optical Flow Constraint

As it was mentioned already, by optical flow we refer apparent movement of the pixels on the image plane. Based on the Lambertian surface model, we expect that any change in the illumination level of a pixel is due to movement, as seen in the following figure:

The figure above shows two different images taken at times $$t$$ and $$t+1$$. Movement of the pixel is observed as a change in the illumination level of the pixel. As it can be understood, there exists several different ways of finding the corresponding pixels, not necessarily based on the 'illumination' (gray) level, but other more robust image features, such as change in the phase (phase based optical flow) and so on. The one that I explain below is a so called 'variational' model (based on the calculus of variations), based on the original model of Horn&Schunck. If order to get a better insight of the model and how it is resolved, have a look at my thesis. In order to see how optical flow (or disparity) estimations can be constrained, have a look at this article.

## Constancy constraint

From the above we come up with the following constancy assumption:

$I(x,\,y\,t) = I(x+u,\,y+v,\,t+1)$

## Basic Variational Model for Optical Flow

In the variational optical-flow calculation the energy functional describing the system is as follows:

$E(u,v) = \min_{u,v} \int_{\Omega} \Psi \Big( Edata(u,v)^2 \Big) dx + \alpha \int_{\Omega} \Psi \Big( Esmooth(u,v)^2 \Big) dx$

where Edata and Esmooth are the data and the smoothness (energy) terms and (u,v) is the optical-flow. The data term measures how well the 'model' fits the data, while the smoothness term regularizes the solution to be smooth. Edata and Esmooth are defined as follows:

\begin{align} Edata(u,v) &= I(x,y,t) - I_R(x+u,y+v,t+1) \\Esmooth(u,v)^2 &= | \nabla u|^2 +| \nabla v|^2 \end{align}

where $$I(x,y,t)$$ refers to the image taken at time t, while $$\nabla = [\frac{\partial}{\partial x}, \frac{\partial}{\partial y}]$$ is the gradient operator. In a sense, we are looking for a transformation, defined by $$[u,v]$$, that moves/morphs the image at time t=1 into the image at t=0, while imposing smoothness on the solution (i.e. $$[u,v]$$) simultaneously.

One possible robust error function is as follows:

$\Psi( s^2) = \sqrt{ s^2 + \epsilon^2 }$

Purpose of the robust error function is to admit 'outliers' in the model. In the Edata case such outliers are, for example, occluded zones (i.e. image structures that are present/visible only in one of the images). In the Esmooth case purpose of the robust error function is to make the solution piece-wise smooth: we do not want to propagate values across object boundaries or, in other words, values residing on different 'velocity' (optical-flow) levels.

## Resolving The Equation(s)

A necessary (but not sufficient) condition for the minima is for the corresponding Euler-Lagrange equation(s) to be zero. Because of the late linearization of the data term the model copes with large displacements. However, this comes at a price: the energy functional may not be convex. Therefore searching for a suitable minimizer becomes more difficult. Following are the corresponding Euler-Lagrange equations in elliptic form.

\begin{align} \Psi{\prime}\left( Edata(u,v)^2\right) Edata(u,v) \frac{\partial I(x+u,y+v,t+1)}{\partial x} + \alpha DIV \left( \Psi{\prime} \left(Esmooth(u,v)^2 \right) \nabla u \right) = 0 \\ \Psi{\prime}\left( Edata(u,v)^2\right) Edata(u,v) \frac{\partial I(x+u,y+v,t+1)}{\partial y} + \alpha DIV \left( \Psi{\prime} \left(Esmooth(u,v)^2 \right) \nabla v \right) = 0 \end{align}

where DIV is the divergence operator. A suitable minimizer is searched for using a coarse-to-fine strategy, while non-linearities are dealt with using a fixed point scheme.

## Example For Use of Optical Flow

Here is an example for use of optical flow in vehicular technology. The idea is to detect overtaking vehicles based on optical flow field calculated, in real-time, from on-board cameras facing towards reaf ot the vehicle. The system, designed and programmed by Pablo Gúzman in collaboration with Anafocus, was implemented on Anafocus's Q-EYE system. See the related publication.

Below you will find a video of the overtaking detection results. Red box indicates a false detection (the system knows it is a false detection), while green box means a positive detection.

Author: Jarno Ralli
Jarno Ralli is a computer vision scientist and a programmer.