Image Deringing with Adaptive Bilateral Filter1Zhai Guangtao, Xu Yi*, Yang Xiaokang, Zhang Wenjun, Yu Songyu Institute of Image Communication and Information Processing, Shanghai Jiao Tong University,Shanghai, China (200240)AbstractIn this paper, we tailor the bilateral filter towards the task of suppressing the ringing artifact commonly occurred on JPEG2000 images under low bitrates. The proposed adaptive bilateral filter varies from its original form as the pixel moves from a monotone area towards an edge one. Also the local spreads of the domain and range filters are tuned with the extent of texture activity. The edge detection and distance transform are used to indicate the local edge and texture activity indexes. Experimental results show that the adaptive bilateral filter can effectively smooth out the annoying ringing artifact and ameliorate the visual quality.Keywords：Bilateral Filter, Image Postprocessing, Deringing1.IntroductionRinging is a kind of Gibbs phenomenon, which is caused by heavy truncation on transform coefficients and manifests itself as spurious oscillations around strong edges. Also, ringing can come from improper image restoration operations [1]. The ringing artifact encountered in the new image coding standard JPEG2000 is much more difficult to model and/or suppress than the blockiness artifact in the last generation block-based coding standard (JPEG). Among the deringing algorithms, the postfiltering schemes are the most attractive due to their compatibility with existing standards and codecs. Based on O’Rourke and Stevenson’s work on blockiness reduction [2], Shen and Kuo [3] formulated deringing into a classical maximum a posteriori (MAP) estimation problem using a Markov random field (MRF) model, and further proposed a non-iterative nonlinear filter to approximate the global optimum solution. Oguz et al. [4] proposed to use combined binary and grayscale morphological operations to filter out ringing artifact. And they also suggested a new perceptual ringing artifact measure named visible ringing measure (VRM) [5]. Fan and Cham [6] designed an edge model under the framework of multiscale edge analysis and used it to reconstruct the corrupted edges in low bitrate wavelet coded image. Nosratinia [7] re-applied JPEG2000 compression on a redundant representation of pixel-by-pixel shifted images and finally integrated the shift-backs to generate the postfiltered image. Essentially, this approach is thought to be deeply related to translation-invariant denoising algorithms introduced in [8]. Yang et al. [9] employed a maximum likelihood estimation approach together with a k-means algorithm and a cluster-segmentation processing to suppress ringing artifact. Recently, Tan and Wu [10] designed a vision model for postfiltering JPEG2000 coded color images. Their model considers both inter and intra band visual masking effects to guarantee a HVS plausible processing result. This postfiltering algorithm, however, is designed for a specific codec designed by the authors themselves [11], and this somewhat restricted its usage. Chen et al. [12] applied grayscale morphological operation together with a voting stage to choose an optimal postfiltering for deringing JPEG2000 images on the encoder side. Consequently this algorithm needs extra bitrate overhead of the morphological filter details to be transmitted to the decoder, and thus is not compatible with the existing standards. And more1 This work was supported by National Natural Science Foundation of China (60332030, 60502034, 60625103,60703044), Shanghai Rising-Star Program (05QMX1435), Hi-Tech Research and Development Program of China 863 (2006AA01Z124), NCET-06-0409, the 111 Project and the specialized Research Fund for the Doctoral Program of Higher Education under grant No. 20040248047.recently, Li [13] introduced a POCS based decoding process exploiting both the quantization and geometric constraints to suppress the ringing artifact.Bilateral filter (BF), which consists of a geometrical domain filter and photometrical range filter, is a kind of non-iterative edge-preserving nonlinear filter proposed by Tomasi and Manduchi [14]. It is known that the BF has a fundamental connection with anisotropic diffusion and adaptive smoothing [15]. In essence, BF can be interpreted as a single iteration of some iterative algorithms emerged from the Bayesian framework [16]. Due to its low computational complexity and high effectiveness in noise suppression/edge preservation, BF is widely used in various image filtering schemes, such as color TV signal crawling dot pattern reduction [17], image detail removal for enhanced compression ratio [18], color demosaicking [19], denoising [20], contrast reducing [20] and picture resizing [21]. In this paper, we adapt the BF towards deringing for JPEG2000 images. The proposed adaptive BF varies its forms from a pure edge preserving range filter for edge areas to a pure noise reduction domain filter for monotone areas. The local spreads of the domain and range filters are also tuned by the local image detail activities. Comparison with the state-of-the art deringing postfilters justifies the effectiveness of the proposed filter.The rest of the paper is organized as follows: Section II reviews the BF, Section III defines the edge and texture activity indexes, Section IV introduces the adaptive bilateral filter, Section V shows some experimental results, and finally Section VI concludes the paper.2. Bilateral filteringThe 2D bilateral filter gives a weighted sum of the neighboring pixels in a local window, and the pixels with nearer geometrical or photometrical distances are assigned with higher weights. This process is defined as()()()()()()[][]()()()()[][],,,,,,,,,,,,,,,,,,i w w j w w i w w j w w f x y S x y x i y j C f x y f x i y j f x y S x y x i y j C f x y f x i y j ∈−∈−∈−∈−++++⎡⎤⎡⎤⎣⎦⎣⎦′=++++⎡⎤⎡⎤⎣⎦⎣⎦∑∑∑∑ (1)where w controls the span of the filter, (),x y is the pixel index,(),f x y is the original image and (),f x y ′is the filtered image. In a local window of size ()()2121w w +×+, for the two neighboring pixels (),f x y and (),f x i y j ++, ()(),,,S x y x i y j ++⎡⎤⎣⎦ and ()(),,,C f x y f x i y j ++⎡⎤⎣⎦ measure the geometric and photometric similarity respectively, and they are referred to as the domain filter and range filter. With Gaussian kernel, these filters can be defined as:()()()222,,,exp 2d i j S x y x i y j σ⎡⎤+⎢⎥++=−⎡⎤⎣⎦⎢⎥⎣⎦ (2)()()()()22,,,,,exp 2r f x y f x i y j C f x y f x i y j σ⎧⎫−++⎡⎤⎪⎪⎣⎦++=−⎡⎤⎨⎬⎣⎦⎪⎪⎩⎭(3) where d σand r σare the standard deviations (SD) for the filters. Though other kernels can also be employed, Gaussian kernel remains as the most popular choice [22] [23].I. Edge and texture activity indexAs abovementioned, ringing artifact is a kind of edge related distortion, and in order to smooth out which while preserving image details, edge detection is often employed in deringing algorithms to differentiate major edges [4]. In this paper, we choose Canny edge detector due to its robustness and efficiency. We first detect the major edges in the image with a threshold 1T to get the binary edge map ()1,E x y 2. A morphological operation is then applied to eliminate the small edge regions to make the following computation more stable. The 2D Euclidian distance transform is employed to produce the distance map, the process of which is()(),1,,0e if E x y D x y otherwise ≠=⎪⎩ (4)where ()','x y is the nearest nonzero neighbor point of (),x y , and is determined as:()()()}','','arg min ','1x y x y E x y == (5)This distance map is then normalized within the range of []0,1 and further processed to give the edge activity index as()()(){}1,1,/max ,e e e A x y D x y D x y γ=−⎡⎤⎣⎦. (6)It can be observed from (4) that the edge points correspond to 1 in the edge activity map to indicate the highest local edge activity. And the value drops as the pixel moves away from the edges.The other two edge maps ()2,E x y and ()3,E x y are then computed with detection thresholds 2T and 3T (123T T T >>) to yield texture information. Note that since 23T T >, the edge points in ()2,E x y is included in ()3,E x y . By taking a binary “XOR”, the overlap between ()2,E x y and ()3,E x y , which corresponds to the heavier edges in ()2,E x y , is cancelled out. And we then get the texture map, which consists of slight edges as()()()23,,,T x y E x y xor E x y =⎡⎤⎡⎤⎣⎦⎣⎦ (7) We employ morphological operations to eliminate the remaining edge regions which are too large or too small. Since we choose 1T ,2T and 3T empirically, there may be some major edges left in the texture map (),T x y , and these major edges usual occupy a large span. To if a connected edge cluster covers a width more than half the image dimension, they are removed in (),T x y . The clusters made up of less than 16 pixels are also eliminated to make the following distance transform stable.Similar as the process in(4), the distance map for (),T x y is computed and denoted as (),t D x y . This distance map is further manipulated to generate the texture activity map as2 Canny edge detector in fact needs a higher and a lower thresholds, the thresholds given hereinafter are all the higher ones, while the lowers thresholds are defined as a quarter of the higher ones in this work.()()(){}2,1,/max ,t t t A x y D x y D x y γ=−⎡⎤⎣⎦ (8)Fig.1 shows the edge and texture activity indexes computed on “Lena” image compressed at 0.1 bpp with Kakadu software [24].II. Adaptive bilateral filter (ABF)The geometric and photometric SD d σand r σdetermine the spread the BF, while w defines the size of the filter support. It is easy to understand that bigger local d σ,r σ and/or widow size w cause heavier truncation to the high frequency component in a local window, and therefore brings smoother resultant image. However, unlike blockiness, ringing is a kind of long range distortion, which means that it can occupy a broad spatial range. To effectively suppress the artifact, a wide window size that fully covers the ringing extension is preferable. So in this paper, we adjust d σand r σwhile keeping a large w to guarantee a sufficient filtering. Before applying BF to ringing reduction, we have the following observations:1. The edge-preserving property of BF comes from the range filter.2. The ringing artifact, being local gray level oscillations, can be smoothed out by a wide domain filter.3. The domain filter with a large spread tends to smear the image details.Based on the above observations, the idea here is to tune the BF towards a pure range filter on edge areas and towards a pure domain filter on monotone areas. Also the filter spreads areattenuated in texture areas to protect image details. This adaptation process is as followsda dσσ=(9) ra σσ= (10)substituting (9) and (10) into (2) and (3), we have()()()2221,,,exp 21e a t i j A S x y x i y j A σξ⎡⎤+−⎢⎥++=−⎡⎤⎣⎦+−⎢⎥⎣⎦ (11)()()()()22,,,,,exp 21e r t f x y f x i y j A C f x y f x i y j A σξ⎧⎫−++⎡⎤⎪⎪⎣⎦++=−⎡⎤⎨⎬⎣⎦+−⎪⎪⎩⎭.(12) We add a small constant ξ to prevent division by zero. Note that as the edge activity index e A goes to 1, the domain filter tends to be uniformly distributed. And as e A approaches 0, the range filter tends to be uniformly distributed. The evolution process of the proposed adaptive BF (ABF) is shown in Fig.2. By substituting (11) and (12) into (1), we can get the final form of the ABF.3. Experimental resultsThe processing results of the BF and the proposed ABF are shown in Fig. 3. The parameters used are 5w =, 3d σ=, 30r σ=, 10.6T =, 10.4T =, 10.3T =, 116γ=, 24γ= andξ=. It can be observed that for the ringing artifacts, such as areas around the shoulder0.001and hat, the BF and the ABF achieve the comparable processing result. However, the adaptive processing algorithm preserves more image details from being smeared out, e.g. texture on the hat and hair.We compare the processing result of the proposed filter with BF and some state of the art deringing postfilters, namely, Shen et. al’s [3] nonlinear filter, Oguz et al.’s morphological filter [4] and Nosratinia’s [7] shift filter. The visual processing results are demonstrated in Fig.4. Shen et al.’s filter protects image details but leaves some ringing untouched. This can be explained as that the filter span used in Shen et al.’s algorithm is very small, so the ringing artifact cannot be sufficiently smoothed. Oguz et al.’s filter also keeps image details and the ringing reduction result is somewhat better than that of Shen et al.’s. Nosratinia’s filter suppresses the ringing artifact, however, it also smears the image details. The BF gives better ringing reduction result, but severely blurs image texture. The proposed ABF has similar performance in ringing suppression, yet with better texture protection.We also compare the numerical performance of the deringing filters with PSNR and VRM [5] as quality measures, and list the result in Table I. VRM averages the local variance in visual saliency edge areas as a perceptual ringing measure (a smaller value corresponding to better deranging). It can be found that though sometimes slightly lower than BF or Shen et al.’s method, the proposed ABF has the most competitive PSNR performance in the test. The VRM performance confirms an effectively ringing reduction of the proposed algorithm. We noted that the BF has a little better VRM performance, due to its smoother result. However, as we analyzed, BF tends to blur image details as well.4.ConclusionIn this paper, we have extended the bilateral filter into an adaptive form for deringing JPEG2000 images. The adaptive bilateral filter evolves from an edge preserving range filter into a Gaussian averaging mask as the pixel moves form edge areas to monotone areas. The filter spans are also attenuated in texture regions to prevent oversmoothing. The edge and texture activity indexes are computed with efficient edge detection and distance transform. Comparison with both the non-adaptive bilateral filter and the existing deringing postfilters confirms the effectiveness of the proposed algorithm, in terms of overall signal fidelity and ringing removal.References[1] R. L. Lagendijk, J. Biemond, and D. E. Boekee, "Regularized iterative image restoration with ringing reduction," IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 36, no. 12, pp. 1874-1888, 1988.[2] T. P. O'Rourke and R. L. Stevenson, "Improved image decompression for reduced transform coding artifacts," Ieee Transactions on Circuits and Systems for Video Technology, vol. 5, no. 6, pp. 490-499, 1995. [3] S. Mei-Yin and C. C. J. Kuo, "Artifact reduction in low bit rate wavelet coding with robust nonlinear filtering," in 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No.98EX175) Redondo Beach, CA, USA: IEEE, 1998, pp. 480-485.[4] S. H. Oguz, Y. H. Hu, and T. Q. Nguyen, "Image coding ringing artifact reduction using morphological post-filtering," in 1998 IEEE Second Workshop on Multimedia Signal Processing (Cat. No.98EX175) Redondo Beach, CA, USA: IEEE, 1998, pp. 628-633.[5] S.H.Oguz, "Morphological post-filtering of ringing and lost data concealment in generalized lapped orthogonal transform based image and video coding," Ph. D dissertation, University of Wisconsin, Madison, 1999.[6] G. Fan and W. K. Cham, "Model-based edge reconstruction for low bit-rate wavelet-compressed images," Ieee Transactions on Circuits and Systems for Video Technology, vol. 10, no. 1, pp. 120-132, 2000.[7] A. Nosratinia, "Postprocessing of JPEG-2000 images to remove compression artifacts," IEEE Signal Processing Letters, vol. 10, no. 10, pp. 296-299, 2003.[8] R.R.Coifman and D.L.Donoho, "Translation-invariant denoising," in Wavelet and statistics. A.Antoniadis and G.Oppenheim, Eds. Berlin: Springer Verlag, 1995, pp. 125-150.[9] S. Yang, Y. H. Hu, T. Q. Nguyen, and D. L. Tull, "Maximum-likelihood parameter estimation for image ringing-artifact removal," Ieee Transactions on Circuits and Systems for Video Technology, vol. 11, no. 8, pp. 963-973, 2001.[10] C. S. Tan and H. R. Wu, "Vision model based perceptual post filtering of JPEG2000 coded colour images," in Proceedings of SPIE - The International Society for Optical Engineering, 5960 ed Beijing, China: International Society for Optical Engineering, Bellingham WA, WA 98227-0010, United States, 2005, pp. 1235-1245.[11] C. S. Tan, D. M. Tan, and H. R. Wu, "Perceptual coding of digital colour images based on a vision model," in Proceedings - IEEE International Symposium on Circuits and Systems, 5 ed Vancouver, BC, Canada: Institute of Electrical and Electronics Engineers Inc., Piscataway, NJ 08855-1331, United States, 2004, p. 441. [12] Y. Y. Chen, S. C. Tai, C. X. Wang, and K. W. Lin, "Design of a filter against artifacts for JPEG2000," Journal of Electronic Imaging, vol. 14, no. 4 2005.[13] X.Li, "Improved wavelet decoding via set theoretic estimation," IEEE Transaction on Circuits and System for Video Technology, vol. 15, no. 1, pp. 108-112, 2005.[14] C. Tomasi and R. Manduchi, "Bilateral filtering for gray and color images," in Journal of Engineering and Applied Science Bombay, India: IEEE, Piscataway, NJ, USA, 1998, pp. 839-846.[15] D. Barash, "A fundamental relationship between bilateral filtering, adaptive smoothing, and the nonlinear diffusion equation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 24, no. 6, pp. 844-847, 2002.[16] M. Elad, "On the origin of the bilateral filter and ways to improve it," IEEE Transactions on Image Processing, vol. 11, no. 10, pp. 1141-1151, 2002.[17] S. Kim and K. Hong, "Composite video artifact removal by nonlinear bilateral filtering," in Proceedings of SPIE - The International Society for Optical Engineering, 5960 ed Beijing, China: International Society for Optical Engineering, Bellingham WA, WA 98227-0010, United States, 2005, pp. 306-315.[18] T. Q. Pham and L. J. van Vliet, "Separable bilateral filtering for fast video preprocessing," in 2005 IEEE International Conference on Multimedia and Expo Amsterdam, Netherlands: IEEE, 2005, p. 4.[19] R. Ramanath and W. E. Snyder, "Demosaicking as a bilateral filtering process," in Proc. SPIE - Int. Soc. Opt. Eng. (USA), 4667 ed San Jose, CA, USA: SPIE-Int. Soc. Opt. Eng, 2002, pp. 236-244.[20] W. C. Kao and Y. J. Chen, "Multistage bilateral noise filtering and edge detection for color image enhancement," IEEE Transactions on Consumer Electronics, vol. 51, no. 4, pp. 1346-1351, 2005.[21] S. Yang and K. Hong, "Bilateral interpolation filters for image size conversion," in Proceedings - International Conference on Image Processing, ICIP, 2 ed Genova, Italy: Institute of Electrical and Electronics Engineers Computer Society, Piscataway, NJ 08855-1331, United States, 2005, pp. 986-989.[22] F. Durand and J. Dorsey, "Fast bilateral filtering for the display of high-dynamic-range images," in ACM Transactions on Graphics, 21 ed United States: Association for Computing Machinery, 2002, pp. 257-266. [23] J. J. Francis and G. De Jager, "The bilateral median filter," Transactions of the South African Institute of Electrical Engineers, vol. 96, no. 2, pp. 106-111, 2005.[24] D.Taubman and M.Marcellin, JPEG 2000: image compression fundamentals, standards and practices. Kluwer Academic Publisher, 2002.List of figures(a)(b)Figure 1. Edge and texture activity indexes (Darker pixel indicating higher activity value).(a) Edge activity map. (b) Texture activity map.Domain Ffilter Range Filter Bilateral FilterFigure 2. Evolution of the adaptive bilateral filter(a)(b)(c)(d)Figure 3. Performance comparison of BF and ABF for Lena. (a) Original “Lena”, 512*512. (b) JPEG2000 compressed at 0.1 bpp. PSNR=29.6 dB. (c) Processing result with BF, PSNR=29.2 dB. (d) Processing result withABF, PSNR=29.9 dB.-11-(a) JPEG2000 compressed (b) Shen’s nonlinear filter [3] (c) Oguz’s morphological filter [4](d) Nosratinia’s shift filter [7] (e) Bilateral filter (f) Adaptive bilateral filterFigure 4. Visual comparison of the deringing postfilters, with hat part of Lena-12-List of tableTable I Objective performance of the deringing postfiltersImage Lena Barbara Peppers Baboon MetricsBitrate 0.05 0.1 0.15 0.2 0.05 0.1 0.15 0.2 0.05 0.1 0.15 0.2 0.05 0.1 0.15 0.2 JPEG2000 26.88 29.63 31.2832.5422.7324.5625.8627.01 26.2129.4231.0232.0020.3521.1621.9022.60 Shen [3] 26.87 29.59 31.2232.4522.7324.5425.8426.97 26.1929.3730.9531.9120.3521.1521.8922.58 Oguz [4] 26.80 29.11 30.2231.0422.6624.2625.1926.08 26.1829.1730.4931.2820.3221.0521.6422.19 Nosratinia[7]26.89 29.41 30.6831.3822.6623.2523.6524.04 25.9628.3729.3229.7220.2420.9521.4921.74 BF 27.18 29.53 30.6531.4622.8024.4625.5626.51 26.6129.8531.1031.7920.3421.0621.5822.22 PSNRABF 27.18 29.94 31.4532.6422.7924.4625.4926.44 26.4629.9031.3532.1920.3921.1821.8722.55JPEG2000 13.14 13.27 12.6511.7111.7911.5611.11 10.47 10.1510.2310.3110.1117.0215.2615.2111.98Shen [3] 9.67 13.25 12.7911.5711.5911.3110.8310.58 10.1810.3010.4110.1516.9715.2115.3012.03 Oguz [4] 11.00 9.71 9.84 9.31 8.70 8.16 8.36 8.25 8.44 8.61 8.37 8.77 12.509.65 10.548.43 Nosratinia[7]10.22 10.80 10.0010.229.07 8.68 9.07 8.97 9.14 9.13 9.21 9.27 10.7710.279.81 8.80 BF 11.39 9.91 8.92 9.08 9.03 8.46 8.40 7.60 8.23 8.04 7.98 8.02 10.7610.1110.057.58 VRMABF 13.14 11.09 9.91 9.48 10.509.65 9.46 8.72 8.84 8.65 8.51 8.63 12.7411.8510.879.86。