Skip to main content
EURASIP Journal on Advances in Signal Processing
Download PDF
  • Research Article
  • Open access
  • Published:

Video-to-Video Dynamic Super-Resolution for Grayscale and Color Sequences

EURASIP Journal on Advances in Signal Processing volume 2006, Article number: 061859 (2006) Cite this article

Abstract

We address the dynamic super-resolution (SR) problem of reconstructing a high-quality set of monochromatic or color super-resolved images from low-quality monochromatic, color, or mosaiced frames. Our approach includes a joint method for simultaneous SR, deblurring, and demosaicing, this way taking into account practical color measurements encountered in video sequences. For the case of translational motion and common space-invariant blur, the proposed method is based on a very fast and memory efficient approximation of the Kalman filter (KF). Experimental results on both simulated and real data are supplied, demonstrating the presented algorithms, and their strength.

References

  1. Huang TS, Tsai RY: Multi-frame image restoration and registration. In Advances in Computer Vision and Image Processing. Volume 1. JAI Press, Greenwich, Conn, USA; 1984:317–339. chapter 7

    Google Scholar 

  2. Nguyen N, Milanfar P, Golub GH: A computationally efficient super-resolution image reconstruction algorithm. Transactions on Image Processing 2001, 10(4):573–583. 10.1109/83.913592

    Article  Google Scholar 

  3. Irani M, Peleg S: Improving resolution by image registration. CVGIP: Graphical Models and Image Processing 1991, 53(3):231–239. 10.1016/1049-9652(91)90045-L

    Google Scholar 

  4. Elad M, Feuer A: Restoration of a single super-resolution image from several blurred, noisy, and undersampled measured images. Transactions on Image Processing 1997, 6(12):1646–1658. 10.1109/83.650118

    Article  Google Scholar 

  5. Zomet A, Peleg S: Efficient super-resolution and applications to mosaics. Proceedings of IEEE 15th International Conference on Pattern Recognition (ICPR '00), September 2000, Barcelona, Spain 1: 579–583.

    Article  Google Scholar 

  6. Ng MK, Bose NK: Mathematical analysis of super-resolution methodology. IEEE Signal Processing Magazine 2003, 20(3):62–74. 10.1109/MSP.2003.1203210

    Article  Google Scholar 

  7. Altunbasak Y, Patti AJ, Mersereau RM: Super-resolution still and video reconstruction from MPEG-coded video. IEEE Transactions on Circuits and Systems for Video Technology 2002, 12(4):217–226. 10.1109/76.999200

    Article  Google Scholar 

  8. Farsiu S, Robinson MD, Elad M, Milanfar P: Fast and robust multi-frame super-resolution. IEEE Transactions on Image Processing 2004, 13(10):1327–1344. 10.1109/TIP.2004.834669

    Article  Google Scholar 

  9. Segall CA, Katsaggelos AK, Molina R, Mateos J: Bayesian resolution enhancement of compressed video. IEEE Transactions on Image Processing 2004, 13(7):898–911. 10.1109/TIP.2004.827230

    Article  Google Scholar 

  10. Borman S, Stevenson RL: Super-resolution from image sequences—a review. Proceedings of Midwest Symposium on Circuits and Systems (MWSCAS '98), August 1998, Notre Dame, Ind, USA 374–378.

    Google Scholar 

  11. Park SC, Park MK, Kang MG: Super-resolution image reconstruction: a technical overview. IEEE Signal Processing Magazine 2003, 20(3):21–36. 10.1109/MSP.2003.1203207

    Article  Google Scholar 

  12. Farsiu S, Robinson MD, Elad M, Milanfar P: Advances and challenges in super-resolution. International Journal of Imaging Systems and Technology 2004, 14(2):47–57. 10.1002/ima.20007

    Article  Google Scholar 

  13. Elad M, Feuer A: Super-resolution restoration of an image sequence: adaptive filtering approach. IEEE Transactions on Image Processing 1999, 8(3):387–395. 10.1109/83.748893

    Article  Google Scholar 

  14. Elad M, Feuer A: Super-resolution reconstruction of image sequences. IEEE Transactions on Pattern Analysis and Machine Intelligence 1999, 21(9):817–834. 10.1109/34.790425

    Article  Google Scholar 

  15. Elad M, Hel-Or Y: A fast super-resolution reconstruction algorithm for pure translational motion and common space-invariant blur. IEEE Transactions on Image Processing 2001, 10(8):1187–1193. 10.1109/83.935034

    Article  Google Scholar 

  16. Farsiu S, Elad M, Milanfar P: Multiframe demosaicing and super-resolution from undersampled color images. Computational Imaging II, January 2004, San Jose, Calif, USA, Proceedings of SPIE 5299: 222–233.

    Article  Google Scholar 

  17. Farsiu S, Elad M, Milanfar P: Multiframe demosaicing and super-resolution of color images. IEEE Transactions on Image Processing 2006, 15(1):141–159.

    Article  Google Scholar 

  18. Kay SM: Fundamentals of Statistical Signal Processing, Volume I: Estimation Theory. Prentice-Hall, Englewood Cliffs, NJ, USA; 1993.

    MATH  Google Scholar 

  19. Jazwinski AH: Stochastic Processes and Filtering Theory. Academic Press, New York, NY, USA; 1970.

    MATH  Google Scholar 

  20. Elad M: Super-resolution reconstruction of continuous image sequence, Ph.D. dissertation. Technion-Israel Institute of Technology, Haifa, Israel; 1997.

    Google Scholar 

  21. Rauch HE, Striebel CT, Tung F: Maximum likelihood estimates of dynamic linear systems. American Institute of Aeronautics and Astronautics 1965, 3(8):1445–1450.

    Article  Google Scholar 

  22. Harvey AC: Forecasting, Structural Time Series Models and the Kalman Filter. Cambridge University Press, Cambridge, UK; 1990.

    Book  Google Scholar 

  23. Bovik A: Handbook of Image and Video Processing. Academic Press, New York, NY, USA; 2000.

    MATH  Google Scholar 

  24. Rudin LI, Osher S, Fatemi E: Nonlinear total variation based noise removal algorithms. Physica D 1992, 60(1–4):259–268.

    Article  MathSciNet  Google Scholar 

  25. Li Y, Santosa F: A computational algorithm for minimizing total variation in image restoration. IEEE Transactions on Image Processing 1996, 5(6):987–995. 10.1109/83.503914

    Article  Google Scholar 

  26. Chan TF, Osher S, Shen J: The digital TV filter and nonlinear denoising. IEEE Transactions on Image Processing 2001, 10(2):231–241. 10.1109/83.902288

    Article  Google Scholar 

  27. Tomasi C, Manduchi R: Bilateral filtering for gray and color images. Proceedings of IEEE 6th International Conference on Computer Vision (ICCV '98), January 1998, Bombay, India 839–846.

    Google Scholar 

  28. Elad M: On the origin of the bilateral filter and ways to improve it. IEEE Transactions on Image Processing 2002, 11(10):1141–1151. 10.1109/TIP.2002.801126

    Article  MathSciNet  Google Scholar 

  29. Laroche C, Prescott M: Apparatus and method for adaptive for adaptively interpolating a full color image utilizing chrominance gradients. United States Patent 5,373,322, 1994

  30. Kimmel R: Demosaicing: image reconstruction from color CCD samples. IEEE Transactions on Image Processing 1999, 8(9):1221–1228. 10.1109/83.784434

    Article  Google Scholar 

  31. Keren D, Osadchy M: Restoring subsampled color images. Machine Vision and Applications 1999, 11(4):197–202. 10.1007/s001380050102

    Article  Google Scholar 

  32. Hel-Or Y, Keren D: Demosaicing of color images using steerable wavelets. In Tech. Rep. HPL-2002-206R1 20020830. HP Laboratories Israel, Haifa, Israel; 2002. Online, available: https://doi.org/citeseer.nj.nec.com/

    Google Scholar 

  33. Muresan DD, Parks TW: Optimal recovery demosaicing. Proceedings of IASTED International Conference on Signal and Image Processing (SIP '02), August 2002, Kauai, Hawaii, USA

    Google Scholar 

  34. Gunturk BK, Altunbasak Y, Mersereau RM: Color plane interpolation using alternating projections. IEEE Transactions on Image Processing 2002, 11(9):997–1013. 10.1109/TIP.2002.801121

    Article  Google Scholar 

  35. Alleysson D, Süsstrunk S, Hérault J: Color demosaicing by estimating luminance and opponent chromatic signals in the Fourier domain. Proceedings of IS&T/SID 10th Color Imaging Conference, November 2002, Scottsdale, Ariz, USA 331–336.

    Google Scholar 

  36. Ramanath R, Snyder WE, Bilbro GL, Sander WA: Demosaicking methods for the Bayer color arrays. Journal of Electronic Imaging 2002, 11(3):306–315. 10.1117/1.1484495

    Article  Google Scholar 

  37. Pei S-C, Tam I-K: Effective color interpolation in CCD color filter arrays using signal correlation. IEEE Transactions on Circuits and Systems for Video Technology 2003, 13(6):503–513. 10.1109/TCSVT.2003.813422

    Article  Google Scholar 

  38. Zomet A, Peleg S: Multi-sensor super-resolution. Proceedings of IEEE 6th Workshop on Applications of Computer Vision (WACV '02), December 2002, Orlando, Fla, USA 27–31.

    Google Scholar 

  39. Gotoh T, Okutomi M: Direct super-resolution and registration using raw CFA images. Proceedings of IEEE Computer Society Conference on Computer Vision and Patern Recognition (CVPR '04), June–July 2004, Washington, DC, USA 2: 600–607.

    Google Scholar 

  40. Shah NR, Zakhor A: Resolution enhancement of color video sequences. IEEE Transactions on Image Processing 1999, 8(6):879–885. 10.1109/83.766865

    Article  Google Scholar 

  41. Tom BC, Katsaggelos AK: Resolution enhancement of monochrome and color video using motion compensation. IEEE Transactions on Image Processing 2001, 10(2):278–287. 10.1109/83.902292

    Article  Google Scholar 

  42. Pratt WK: Digital Image Processing. 3rd edition. John Wiley & Sons, New York, NY, USA; 2001.

    Book  Google Scholar 

  43. Golland P, Bruckstein AM: Motion from color. Computer Vision and Image Understanding 1997, 68(3):346–362. 10.1006/cviu.1997.0553

    Article  Google Scholar 

  44. Bergen JR, Anandan P, Hanna KJ, Hingorani R: Hierarchical model-based motion estimation. Proceedings of European Conference on Computer Vision (ECCV '92), May 1992, Santa Margherita Ligure, Italy 237–252.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Electrical Engineering Department, University of California Santa Cruz, Santa Cruz, CA, 95064, USA

    Sina Farsiu & Peyman Milanfar

  2. Computer Science Department, Technion – Israel Institute of Technology, Haifa, 32000, Israel

    Michael Elad

Authors
  1. Sina Farsiu
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Michael Elad
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. Peyman Milanfar
    View author publications

    You can also search for this author inPubMed Google Scholar

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 2.0 International License ( https://creativecommons.org/licenses/by/2.0 ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Farsiu, S., Elad, M. & Milanfar, P. Video-to-Video Dynamic Super-Resolution for Grayscale and Color Sequences. EURASIP J. Adv. Signal Process. 2006, 061859 (2006). https://doi.org/10.1155/ASP/2006/61859

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1155/ASP/2006/61859

Keywords