3. Proposed Modification to Xiaofeng et al Approach

Xiaofeng et al[10] suggested the pre-processing stage in their algorithm, where the color image has to be converted to gray level image. This step converts the RGB color image to single plane gray level image and there by minimizes the computational complexity. Although this produced very good results but further improvement can be achieved by the help of color information. Hence a modified algorithm based on the existing Xiaofeng algorithm has been proposed to increase the efficacy of the algorithm in terms of the misclassification error.

Background Updating

Let f(x, y, z)_R, f(x, y, z)_G and f(x, y, z)_B are the RGB components of pixel (x, y) respectively. The Xiaofeng method is applied to each of these three plane of each frame and then the results are segregated to get the required output. The modifications are as follows:

1. Read three consecutive frames, say f_k-1(x,y,z), f_k(x,y,z) and f_k+1(x,y,z).

2. Get the gray scale difference between the two consecutive frames,

IZ_k,k-1(x,y,z)=|I_k-1(x,y,z) – I_k(x,y,z)|

IZ_k+1,k(x,y,z)=|I_k(x,y,z ) – I_k+1(x,y,z)|

3. If

then B(x,y,z)=I_k+1(x,y,z)

else M(x,y,z)=I_k+1(x,y,z)

Where Th is the gray threshold, the term (1/N)∑|I_k(x,y,z) – I_k+1(x,y,z)| denotes image changes in overall light, so that the threshold Th can adapt to the slight changes of environmental lighting conditions. N is the number of pixels in the image, B(x,y,z) and M(x,y,z) represents separately the background and object regions.

Moving Object extraction

The object regions in video sequences can be obtained by thresholding the absolute differences between the pixels of background image B(x, y, z) and those of video image Ik(x, y, z). In this thesis work we adopt the discrimination method based on the relative difference which can enhance the detection effect of moving object in shadow regions.

(4)

(5)

The detection results can be further enhanced by selecting the threshold Th’ properly, as this is one of the key component for the binarization of the CDM of DBk(x, y, z). Since the three frame differencing method has strong adaptability to variation in lighting condition while background subtraction can makeup the disadvantage of interior cavity in the moving object extracted by the traditional temporal differencing, hence we can get the moving object segment from the background images through merging the motion regions obtained by the help of equations (4), (5) and the following equation

(6)

where Re_k (x, y, z) represents the final extracted moving object based on the above stated method.

Post Processing

The image obtained in the stated method, Re_k(x, y, z) contains a lot of residual noises except those moving objects in the image. The moving object(s) thus obtained may also contain cavities inside. Thus the median filtering and morphological filtering like erosion and dilation of the Re_k(x, y, z), will minimize such noises as well as it will further improve the small cavity regions present inside the object region, which were misclassified as background class earlier.

Experimental Results

Three video sequences are considered to verify the Color Based Three Frame Differencing (CBTFD) algorithm. In Fig.2, the aswini video sequence is considered and it is assumed that the reference frame is not available. The three frame differencing algorithm based Xiaofeng method results in a good result which is shown in Fig.2 (b). It can be observed here that some portion near the chest area has been missing in the change detection mask (CDM). Thus a large number of pixels get misclassified as background. By increasing the mask size in the morphological filtering operation this can be overcome but by that the size of the object will be larger than the actual size of the object. Fig.2(c) shows the CDM obtained by the modified Xiaofeng method. By introducing the color information to segment the moving object area the misclassification error get reduced to a greater extent. Fig.3(a) shows the original walk video sequence. Fig.3(b) and Fig.3(c) show the CDM obtained by the Xiaofeng method and color based three frame differencing method respectively. In Fig.3(b), the size of the object obtained by Xiaofeng method is quite larger than the original size which is overcome by the color based three frame differencing method. The similar kind of observations can be made from Fig.4(b) and Fig.4(c) respectively. The values of the Th and Th’ used in the experiments are given in the table-1.

Figure 2(a). Original frames of aswini video sequence

Figure 2(b). CDM obtained by Xiaofeng et al method

Figure 2(c). CDM obtained by the proposed CB3FD method

Figure 3(a). Original frames of walk video sequence

Figure 3(b). CDM obtained by Xiaofeng et al method

Figure 3(c). CDM obtained by the proposed CB3FD method

Figure 4(a). Original frames of hand video sequence

Figure 4(b). CDM obtained by Xiaofeng et al method

Figure 4(c). CDM obtained by the proposed CB3FD method

Figure 5. Overview of the proposed algorithm

In this part of the paper it is assumed that the reference frame as well as any a priori knowledge regarding the object of concern is not available. In order to cope with such problem, the three frame differencing followed by the morphological or median filtering technique proposed by Xiofeng et al is used, which yields very good results. It also takes care of slight illumination variations. The Xiaofeng method is a very faster algorithm as it works only on gray level images. But; due to this gray level conversion the video sequences loose the color information present in each frame. Thus the modification to the Xiaofeng algorithm of three frame differencing has been proposed which not only gives very good detection but also retain the color information of the object of concern, in terms of the misclassification error.

4. Proposed Inter-Frame Similarity Approach

Detection of a moving object in a video can be done by temporal segmentation methods. Many existing work explain about the methodologies. Although it works up to some extent but, fails to detect exact moving area in a scene. This is because of the fact that, the temporal difference can detect the relatively changed information in successive frames. If the object is moving very fast then, up to some extent it can yield good results but, if the object is moving with a slow speed, then, the temporal segmentation methods result in cavities inside the object body. Many such cases have been solved by the use of morphological operations like erosion and dilation. Such operations are basically supervised operations. Thus, each time the new object enters into the scene, the morphological filter parameters need new assignments. To make this more effective and robust, we have proposed a spatio-temporal framework based method, which takes care of the object shape, size as well as the cavities inside the object body. Morphological filtering some times result in larger or smaller object size than the actual size of the object, which unnecessarily add to misclassification error. Temporal difference can detect the relatively changed information in successive frames. Therefore it is difficult to detect the overlaps in the moving object and always cause some cavities inside the moving entity. In addition the gray changing region detected by the temporal difference method contains the real moving object and the background changing area which occluded by moving object in the previous frame, thus causing the detected object larger than it’s real and even overlaps. In order to cope with such issue, we have proposed a three frames temporal similarity method which computes the inter-frame similarity and then determines the moving edge image and then fuses them in a deterministic framework. Comparing with the traditional method, the improved method has a significant adaptability to any change in speed of moving object and illumination variation.

The three color planes are extracted from the color video sequences. Then the inter frame similarity between the three consecutive Red, Green and Blue planes such as P_red, P_green and P_blue is computed. The similarity criterion considered here is the intensity value of each color plane.

The similarity is discarded and the dissimilarity in intensity between the three color planes is used as the reference foreground region. The algorithm for the computation of the crude reference foreground is:

Algorithm-1

1. Extract the color planes such as P_red, P_green and P_blue

2. Determine the inter-frame similarity between frames f ^t, f ^t-1and f ^t+1 frames using a window ‘w’

(7)

where ‘m’ is the mean and var is the variance computed for the respective ‘w’ window. ‘x’ can be the ‘Red – plane’, Green – Plain or Blue – plane.

3. The above step is repeated for the consecutive Red, Green and Blue planes of the f ^t, f ^t-1 and f ^t+1 video sequences and the inter-frame similarity is computed.

4. The three inter-frame similarities obtained are then fused by logical ‘OR’ operator to get the final similarity matrix.

(8)

5. Compare the elements of the final similarity matrix with a manually chosen threshold value. T_sim.

6. The pixels whose similarity is less than the defined threshold are marked as foreground and rest all are treated as background. i.e,

(9)

The result obtained here is a crude result which contains the expected foreground region and not the exact region. In order to get the exact region the moving edge is determined. Canny’s edge detector[13] is used to get the edge information in each of the consecutive video sequences. The values of the elements of the final similarity matrix generally vary within 0 to 1. Less is the value less is the correlation between consecutive planes and hence, more is the probability that the pixel belongs to object class. More is the value of the element more is the temporal similarity and hence, less is the probability that the pixel belongs to object class. Thus, a threshold of similarity is chosen in such a way that below which it corresponds to the object class and above which it corresponds to the background.

The selection of such a threshold basically depends upon the amount of overall illumination variation and the change in overall intensity between the three conjugative frames. In this paper, we have selected this threshold manually by trial and error basis. The edge information is very vital in case of the object detection and recognition. In our method, we have computed the moving edge map by determining the differences between the edge maps of the three consecutive video sequences. The moving edge estimate is computed by

(10)

where, is the edge map obtained by the Canny’s Edge detector, which is obtained by applying gradient operation on the Gaussian convoluted image G*f. Then the thinning of the edges is done by applying non-maximum suppression to the gradient magnitude and linking of the non-joined edges is achieved by the help of thresholding operation with hysteresis. The results obtained are then further improvised by the fusion of the obtained object area F_CDM, along with the moving edge image F_ME, which gives a better view of the object boundary, to give a far better result of foreground region under illumination variation condition.

The Fusion algorithm which takes the F_ME and F_CDM in to account to get the final result is given below.

Algorithm-2

where C_obj is the object class and C_back is the background class. Pixel_(i,j) is the corresponding pixel under consideration. The final foreground obtained is then combined with the original image sequence to get the video object plane (VOP).

EXPERIMENTAL RESULTS

The theoretical formulation of the inter-frame similarity based model and the algorithms proposed sounds good; but without practical analysis and simulation the actual efficacy of the algorithms can not be claimed. In order to test the robustness and efficacy of the algorithms and the framework proposed, we have considered many video sequences in indoor and outdoor environments. Special care has been taken to capture the illumination variations caused due to lighting variation and cloud movements. Among many videos three videos are considered here to prove the strength of the proposed method. These three videos are navy, aswini and hand videos, which are filmed in the lawn facing to our research lab, in the corridor and inside the lab respectively. In case of the navy video the illumination variation is slight and this variation is due to cloud movement. In the aswini video the illumination variation is due to external lighting and in case of the hand video lighting variation is achieved by varying the lighting condition in side the closed lab. In these test videos it has been assumed that the reference frames of the scene under consideration are not available. In such cases the background subtraction algorithms fail to yield accurate results.

The experiments were performed on a dual core system with 3GHz processor speed and with a DRAM of capacity 2GB. The processing time is strictly dependent on the quality of moving points and on the video frame dimension which was 640x480. Although the window size depends upon the object shape and extent of illumination variation, in our experiment and simulations we have taken the window size to be 5x5, as it gives very good results, in accordance to the time complexity and accuracy. The thresholds used in all the algorithms are choosen manually depending upon the object body color and dimension of the object. The threshold T_sim varies from 0 to 1 depending upon the scene content and lighting variation. In case of the three videos shown here, the different threshold selected by trial and error basis are given in the following table.

Table 1. Thresholds used in the considered video sequences

Table 2. Computation of percentage of misclassification error (PME)

The Fig-6(a) shows the original sequences of aswini video, which is a outdoor video. Fig 6(b) and Fig 6(c) show the CDM obtained by the modified Xiaofeng method (CB3FDM) and our inter-frame similarity based method. Fig 6(d) shows the VOP resulted by our proposed method. It can be seen here that some silhouette present in the result of Xiaofeng method where as our proposed method gives far better result. Similar kind of observation is being made from Fig 7, that shows the navy video sequence which is an outdoor video. The resultant CDM depicted in Fig 7(c) quite better than the images obtained by the modified Xiaofeng method which is shown in Fig 7(b).Fig 8(a) shows the original sequences of hand video, which is an indoor video. Fig 8(b) and Fig 8(c) show the CDM obtained by the modified Xiaofeng method (CB3FDM) and our inter-frame similarity based method. Fig 8(d) shows the VOP obtained by our proposed method, which is far better than the Xiaofeng method. The table-2 shows the percentage of misclassification error in case of the modified Xiaofeng method (CB3FDM) and our proposed inter-frame similarity based method. Here, it is clearly observed that our proposed method decreases the PME to a good extent. Our proposed method yields such a better result as compared to Xiofeng method because of the spatio-temporal framework which computes the inter-plane similarity, use of color information and use of moving edge information between the three consecutive video sequences. The percentage of misclassification error was calculated by comparing the segmented video frames obtained with their corresponding manually constructed ground truth sequences with the help of GIMP software. This was done only for a comparison and analysis purpose. The algorithm was tested with many video sequences. Here we are providing three video results.

Figure 6(a). Original frames of aswini video sequence

Figure 6(b). CDM obtained by modified Xiaofeng et al method

Figure 6(c). CDM obtained by proposed inter- frame similarity based method

Figure 6(d). Final VOP obtained by proposed method

Figure 7(a). Original frames of navy video sequence

Figure 7(b). CDM obtained by modified Xiaofeng et al method

Figure 7(c). CDM obtained by proposed inter- frame similarity based method

Figure 7(d). Final VOP obtained by proposed method

Figure 8(a). Original frames of hand video sequence

Figure 8(b). CDM obtained by modified Xiaofeng et al method

Figure 8(c). CDM obtained by proposed inter- frame similarity based method

Figure 8(d). Final VOP obtained by proposed method

5. Conclusions

In this paper, the problem of moving object detection under illumination variation has been addressed. The color based three frame difference method is implemented which is just a modification of the Xiaofeng method. Although it yields good result but the silhouettes present around the object. Thus the proposed method is used to detect the object perfectly. The inter-frame similarity is used to tackle the problem of illumination variation. Efficacy of the algorithms proposed is compared on the basis of misclassification error. The algorithm can further be modified to tackle the stop and go motion detection, shadow elimination problems. The thresholds used are all chosen manually which can be made adaptive to the video sequence to make the algorithm more efficient in different environment and situations.