Foreground detection
≈ 12 min read
Names in other languages
No names in other languages are held for this entry yet.
Article bodies are English and are not machine-translated. A name in another language is not a translation of the article.
Foreground detection is one of the major tasks in the field of computer vision and image processing whose aim is to detect changes in image sequences. Background subtraction is any technique which allows an image's foreground to be extracted for further processing (object recognition etc.). Many applications do not need to know everything about the evolution of movement in a video sequence, but only require the information of changes in the scene, because an image's regions of interest are objects (humans, cars, text etc.) in its foreground. After the stage of image preprocessing (which may include image denoising, post processing like morphology etc.) object localisation is required which may make use of this technique. Foreground detection separates foreground from background based on these changes taking place in the foreground. It is a set of techniques that typically analyze video sequences recorded in real time with a stationary camera.
Foreground detection is one of the major tasks in the field of computer vision and image processing whose aim is to detect changes in image sequences. Background subtraction is any technique which allows an image's foreground to be extracted for further processing (object recognition etc.). Many applications do not need to know everything about the evolution of movement in a video sequence, but only require the information of changes in the scene, because an image's regions of interest are objects (humans, cars, text etc.) in its foreground. After the stage of image preprocessing (which may include image denoising, post processing like morphology etc.) object localisation is required which may make use of this technique. Foreground detection separates foreground from background based on these changes taking place in the foreground. It is a set of techniques that typically analyze video sequences recorded in real time with a stationary camera.
Description
All detection techniques are based on modelling the background of the image, i.e., setting the background and detecting which changes occur. Defining the background can be difficult when it contains shapes, shadows, and moving objects. In defining the background, it is assumed that stationary objects may vary in color and intensity over time. Scenarios in which these techniques apply tend to be very diverse. There can be highly variable sequences, such as images with different lighting, interiors, exteriors, quality, and noise. In addition to real-time processing, systems need to adapt to these changes. A foreground detection system should be able to: Develop a background model (estimate). Be robust to lighting changes, repetitive movements (leaves, waves, shadows), and long-term changes.
Background subtraction
Background subtraction is a widely used approach for detecting moving objects in videos from static cameras. The rationale in the approach is that of detecting the moving objects from the difference between the current frame and a reference frame, often called "background image", or "background model". Background subtraction is mostly done if the image in question is a part of a video stream. Background subtraction provides important cues for numerous applications in computer vision, for example surveillance tracking or human pose estimation. Background subtraction is generally based on a static background hypothesis which is often not applicable in real environments. With indoor scenes, reflections or animated images on screens lead to background changes. Similarly, due to wind, rain or illumination changes brought by weather, static backgrounds methods have difficulties with outdoor scenes.
Temporal average filter
The temporal average filter is a method that was proposed at the Velastin. This system estimates the background model from the median of all pixels of a number of previous images. The system uses a buffer with the pixel values of the last frames to update the median for each image. To model the background, the system examines all images in a given time period called training time. At this time, we only display images and will find the median, pixel by pixel, of all the plots in the background this time. After the training period for each new frame, each pixel value is compared with the input value of funds previously calculated. If the input pixel is within a threshold, the pixel is considered to match the background model and its value is included in the pixbuf. Otherwise, if the value is outside this threshold pixel is classified as foreground, and not included in the buffer. This method cannot be considered very efficient because they do not present a rigorous statistical basis and requires a buffer that has a high computational cost.
Conventional approaches
A robust background subtraction algorithm should be able to handle lighting changes, repetitive motions from clutter and long-term scene changes. The following analyses make use of the function of V(x,y,t) as a video sequence where t is the time dimension, x and y are the pixel location variables. e.g. V(1,2,3) is the pixel intensity at (1,2) pixel location of the image at t = 3 in the video sequence. P[F(t)]=P[I(t)]-P[B] The background is assumed to be the frame at time t. This difference image would only show some intensity for the pixel locations which have changed in the two frames. Though we have seemingly removed the background, this approach will only work for cases where all foreground pixels are moving, and all background pixels are static. A threshold "Threshold" is put on this difference image to improve the subtraction (see Image thresholding): |P[F(t)]-P[F(t+1)] |>\mathrm{Threshold} This means that the difference image's pixels' intensities are 'thresholded' or filtered on the basis of value of Threshold. The accuracy of this approach is dependent on speed of movement in the scene. Faster movements may require higher thresholds. B(x,y,t)={1 \over N} \sum_{i=1}^N V(x,y,t-i) where N is the number of preceding images taken for averaging. This averaging refers to averaging corresponding pixels in the given images. N would depend on the video speed (number of images per second in the video) and the amount of movement in the video. After calculating the background B(x,y,t) we can then subtract it from the image V(x,y,t) at time t = t and threshold it. Thus the foreground is: |V(x,y,t)-B(x,y,t) |> \mathrm{Th} where Th is a threshold value. Similarly, we can also use median instead of mean in the above calculation of B(x,y,t). Usage of global and time-independent thresholds (same Th value for all pixels in the image) may limit the accuracy of the above two approaches. \mu_0 = I_0 \sigma^2_0 = \langle \text{some default value}\rangle where I_t is the value of the pixel's intensity at time t . In order to initialize variance, we can, for example, use the variance in x and y from a small window around each pixel. Note that background may change over time (e.g. due to illumination changes or non-static background objects). To accommodate for that change, at every frame t , every pixel's mean and variance must be updated, as follows: \mu_t = \rho I_t + (1 - \rho) \mu_ {t-1} \sigma^2_t = d ^ 2 \rho + (1 - \rho) \sigma ^ 2_ {t-1} d = |(I_t-\mu_t) | Where \rho determines the size of the temporal window that is used to fit the pdf (usually \rho = 0.01 ) and d is the Euclidean distance between the mean and the value of the pixel. We can now classify a pixel as background if its current intensity lies within some confidence interval of its distribution's mean: \frac{ |(I_t-\mu_t) |}{\sigma_t} > k \longrightarrow \text{foreground} \frac{ |(I_t-\mu_t) |}{\sigma_t} \le k \longrightarrow \text{background} where the parameter k is a free threshold (usually k = 2.5 ). A larger value for k allows for more dynamic background, while a smaller k increases the probability of a transition from background to foreground due to more subtle changes. In a variant of the method, a pixel's distribution is only updated if it is classified as background. This is to prevent newly introduced foreground objects from fading into the background. The update formula for the mean is changed accordingly: \mu_t =M\mu_{t-1} +(1-M)(I_t\rho +(1-\rho)\mu_{t-1}) where M = 1 when I_t is considered foreground and M = 0 otherwise. So when M = 1 , that is, when the pixel is detected as foreground, the mean will stay the same. As a result, a pixel, once it has become foreground, can only become background again when the intensity value gets close to what it was before turning foreground. This method, however, has several issues: It only works if all pixels are initially background pixels (or foreground pixels are annotated as such). Also, it cannot cope with gradual background changes: If a pixel is categorized as foreground for a too long period of time, the background intensity in that location might have changed (because illumination has changed etc.). As a result, once the foreground object is gone, the new background intensity might not be recognized as such anymore. X_1 , \ldots , X_t = \{V (x_0 , y_0 , i) : 1 \leqslant i \leqslant t\} This history is modeled by a mixture of K Gaussian distributions: P(X_t) = \sum_{i=1}^K \omega_{i,t} N \left ( X_t \mid \mu_{i,t} , \sigma_{i,t} \right ) where: N \left (X_t \mid \mu_{i,t}, \sigma_{i,t} \right ) = \dfrac{1}{(2\pi)^{D/2}} { 1 \over |\sigma_{i,t} |^{1/2}} \exp\left(-{ 1 \over 2 }( X_t -\mu_{i,t} )^T \sigma_{i,t}^{-1} \left (X_t -\mu_{i,t} \right )\right) First, each pixel is characterized by its intensity in RGB color space. Then probability of observing the current pixel is given by the following formula in the multidimensional case: P(X_t) = \sum_{i=1}^K \omega_{i,t} \eta\left ( X_t, \,\mu_{i,t} , \sigma_{i,t} \right ) Where K is the number of distributions, ω is a weight associated to the ith Gaussian at time t and μ, Σ are the mean and standard deviation of said Gaussian respectively. \eta\left ( X_t, \,\mu_{i,t} , \sigma_{i,t} \right )=\dfrac{1}{(2\pi)^{D/2}} { 1 \over |\sigma_{i,t} |^{1/2}} \exp\left(-{1\over 2}(X_t-\mu_{i,t})^T \sigma_{i,t}^{-1} \left(X_t-\mu_{i,t}\right)\right) Once the parameters initialization is made, a first foreground detection can be made then the parameters are updated. The first B Gaussian distribution which exceeds the threshold T is retained for a background distribution: B=\operatorname{argmin}\left(\Sigma_{i=1}^B\omega_{i,t} >T\right) The other distributions are considered to represent a foreground distribution. Then, when the new frame incomes at times t+1 , a match test is made of each pixel. A pixel matches a Gaussian distribution if the Mahalanobis distance: \left(\left(X_{t+1}-\mu_{i,t}\right)^T \sigma_{i,t}^{-1} \left(X_{t+1}-\mu_{i,t}\right)\right)^{0.5} where k is a constant threshold equal to 2.5 . Then, two cases can occur: Case 1: A match is found with one of the k Gaussians. For the matched component, the update is done as follows: \sigma_{i,t+1}^2=\left(1-\rho\right)\sigma_{i,t}^2+\rho\left(X_{x+1}-\mu_{x+1} \right) \left(X_{x+1}-\mu_{x+1}\right)^T Power and Schoonees [3] used the same algorithm to segment the foreground of the image: \sigma_{i,t+1}=\left(1-\alpha\right)\omega_{i,t}+\alpha P\left(k\mid X_t, \varphi \right) The essential approximation to P\left(k\mid\ X_t,\varphi\right) is given by M_{k,t} : M_{k,t}= \begin{cases} 1 & \text{match}, \\ 0 & \text{otherwise}. \end{cases} Case 2: No match is found with any of the K Gaussians. In this case, the least probable distribution K is replaced with a new one with parameters: k_{i.t}= \text{low prior weight} \mu_{i,t+1}=X_{t+1} \sigma^2_{i.t+1}= \text{large initial variance} Once the parameter maintenance is made, foreground detection can be made and so on. An on-line K-means approximation is used to update the Gaussians. Numerous improvements of this original method developed by Stauffer and Grimson have been proposed and a complete survey can be found in Bouwmans et al. A standard method of adaptive backgrounding is averaging the images over time, creating a background approximation which is similar to the current static scene except where motion occur.
Comparisons
Several comparison/evaluation papers can be found in the literature: A. Sobral, A. Vacavant. "A comprehensive review of background subtraction algorithms evaluated with synthetic and real videos ". Computer Vision and Image Understanding, CVIU 2014, 2014. A. Shahbaz, J. Hariyono, K. Jo, "Evaluation of Background Subtraction Algorithms for Video Surveillance", FCV 2015, 2015. Y. Xu, J. Dong, B. Zhang, D. Xu, "Background modeling methods in video analysis: A review and comparative evaluation', CAAI Transactions on Intelligence Technology, pages 43–60, Volume 1, Issue 1, January 2016. A. Petrosino, L. Maddalena, T. Bouwmans, Special Issue on "Scene Background Modeling and Initialization", Pattern Recognition Letters, September 2017. T. Bouwmans, Special Issue on "Detection of Moving Objects", MDPI Journal of Imaging, 2018. Stuttgart Artificial Background Subtraction Dataset (For more information: http://www.vis.uni-stuttgart.de/index.php?id=sabs ) SBMI dataset (For more information: http://sbmi2015.na.icar.cnr.it/) SBMnet dataset (For more information: http://pione.dinf.usherbrooke.ca/dataset/ )
Text derived from https://en.wikipedia.org/w/index.php?oldid=1368856673 under CC BY-SA 4.0; modifications have been made.