SACS | An Enhanced Fluid Registration for Image Multi-Frame Super Resolution

Published in Volume XXXII, Issue 2, 2022, pages 255-283, doi: 10.7561/SACS.2022.2.255

Authors: A. Laghrib, A. Hadri , M. Hakim

Abstract

The main idea of multi-frame super resolution (SR) algorithms is to recover a single high-resolution image from a sequence of low resolution ones of the same object. The success of the SR approaches is often related to a well registration and restoration steps. Therefore, we propose a new approach based on fluid image registration and we use a second order partial differential equation (PDE) to treat both the registration and restoration steps that guarantees the success of SR algorithms. Since the registration step is usually a variational ill-posed model, a mathematical study is needed to check the existence of the solution to the regularized problem. Thus, we prove the existence and uniqueness of the well posed fluid image registration and assure also the existence of the used second order PDE in the restoration step. The results show that the proposed method is competitive with the existing methods.

Full Text (PDF)

References

[1] Lekbir Afraites, Aissam Hadri, and Amine Laghrib. A denoising model adapted for impulse and Gaussian noises using a constrained-PDE. Inverse Problems, 36(2):025006, 2020. doi:10.1088/1361-6420/ab5178.

[2] Lekbir Afraites, Aissam Hadri, Amine Laghrib, and Mourad Nachaoui. A non-convex denoising model for impulse and Gaussian noise mixture removing using bi-level parameter identification. Inverse Problems and Imaging, 16(4):827-870, 2022. doi:10.3934/ipi.2022001.

[3] Mohamed Alahyane, Abdelilah Hakim, Amine Laghrib, and Said Raghay. A fast approach of nonparametric elastic image registration problem. Mathematical Methods in the Applied Sciences, 42(18):7059-7075, 2019.
doi:10.1002/mma.5810.

[4] Gilles Aubert and Pierre Kornprobst. Mathematical Problems in Image Processing – Partial Differential Equations and the Calculus of Variations, 2nd Edition, volume 147 of Applied Mathematical Sciences. Springer, 2006.

[5] Simon Baker and Takeo Kanade. Limits on super-resolution and how to break them. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(9):1167-1183, 2002. doi:10.1109/TPAMI.2002.1033210.

[6] Honggang Chen, Xiaohai He, Chao Ren, Linbo Qing, and Qizhi Teng. CISRDCNN: super-resolution of compressed images using deep convolutional neural networks. Neurocomputing, 285:204-219, 2018. doi:10.1016/j.neucom.2018.01.043.

[7] Thaı́s Pedruzzi do Nascimento and Evandro Ottoni Teatini Salles. Multi-frame super-resolution combining demons registration and regularized Bayesian reconstruction. IEEE Signal Processing Letters, 27:2009-2013, 2020. doi:10.1109/LSP.2020.3033422.

[8] Chao Dong, Chen Change Loy, and Xiaoou Tang. Accelerating the super-resolution convolutional neural network. In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, 14th European Conference on Computer Vision, ECCV 2016, volume 9906 of Lecture Notes in Computer Science, pages 391-407. Springer, 2016. doi:10.1007/978-3-319-46475-6_25.

[9] Weisheng Dong, Lei Zhang, Guangming Shi, and Xiaolin Wu. Image deblurring and super-resolution by adaptive sparse domain selection and adaptive regularization. IEEE Transactions on Image Processing, 20(7):1838-1857, 2011. doi:10.1109/TIP.2011.2108306.

[10] Abdelmajid El Hakoume, Lekbir Afraites, and Amine Laghrib. An improved coupled PDE system applied to the inverse image denoising problem. Electronic Research Archive, 30(7):2618-2642, 2022. doi:10.3934/era.2022134.

[11] Idriss El Mourabit, Mohammed El Rhabi, Abdelilah Hakim, Amine Laghrib, and Eric Moreau. A new denoising model for multi-frame super-resolution image reconstruction. Signal Processing, 132:51-65, 2017. doi:10.1016/j.sigpro.2016.09.014.

[12] Idriss El Mourabit, Abdelilah Hakim, and Amine Laghrib. An anisotropic PDE for multi-frame super-resolution image reconstruction. In Abdeljalil Nachaoui, Abdelilah Hakim, and Amine Laghrib, editors, Mathematical Control and Numerical Applications, pages 29-41. Springer International Publishing, 2021. doi:10.1007/978-3-030-83442-5_3.

[13] Sina Farsiu, M Dirk Robinson, Michael Elad, and Peyman Milanfar. Fast and robust multiframe super resolution. IEEE Transactions on Image Processing, 13(10):1327-1344, 2004. doi:10.1109/TIP.2004.834669.

[14] Rik Fransens, Christoph Strecha, and Luc Van Gool. Optical flow based super-resolution: A probabilistic approach. Computer Vision and Image Understanding, 106(1):106-115, 2007. doi:10.1016/j.cviu.2005.09.011.

[15] Debabrata Ghosh, Naima Kaabouch, and Wen-Chen Hu. A robust iterative super-resolution mosaicking algorithm using an adaptive and directional Huber-Markov regularization. Journal of Visual Communication and Image Representation, 40:98-110, 2016. doi:10.1016/j.jvcir.2016.06.008.

[16] Hayit Greenspan, G Oz, N Kiryati, and SLBG Peled. MRI inter-slice reconstruction using super-resolution. Magnetic resonance imaging, 20(5):437-446, 2002. doi:10.1016/s0730-725x(02)00511-8.

[17] Aissam Hadri, Hamza Khalfi, Amine Laghrib, and Mourad Nachaoui. An improved spatially controlled reaction–diffusion equation with a non-linear second order operator for image super-resolution. Nonlinear Analysis: Real World Applications, 62:103352, 2021. doi:10.1016/j.nonrwa.2021.103352.

[18] Khizar Hayat. Multimedia super-resolution via deep learning: A survey. Digital Signal Processing, 81:198-217, 2018. doi:10.1016/j.dsp.2018.07.005.

[19] Neeraj Kumar, Ruchika Verma, and Amit Sethi. Convolutional neural networks for wavelet domain super resolution. Pattern Recognition Letters, 90:65-71, 2017. doi:10.1016/j.patrec.2017.03.014.

[20] Amine Laghrib, Lekbir Afraites, Aissam Hadri, and Mourad Nachaoui. A non-convex PDE-constrained denoising model for impulse and Gaussian noise mixture reduction. Inverse Problems and Imaging, 2022. doi:10.3934/ipi.2022031.

[21] Amine Laghrib, Fatimzehrae Aitbella, and Abdelilah Hakim. A second-order nonlocal regularized variational model for multiframe image super-resolution. International Journal of Nonlinear Sciences and Numerical Simulation, 23(3-4):385-399, 2021. doi:10.1515/ijnsns-2019-0109.

[22] Amine Laghrib, A Ben-Loghfyry, Aissam Hadri, and Abdelilah Hakim. A nonconvex fractional order variational model for multi-frame image super-resolution. Signal Processing: Image Communication, 67:1-11, 2018. doi:10.1016/j.image.2018.05.011.

[23] Amine Laghrib, Abdelkrim Chakib, Aissam Hadri, and Abdelilah Hakim. A nonlinear fourth-order PDE for multi-frame image super-resolution enhancement. Discrete & Continuous Dynamical Systems-B, 25(1):415-442, 2020. doi:10.3934/dcdsb.2019188.

[24] Amine Laghrib, Mahmoud Ezzaki, Mohammed El Rhabi, Abdelilah Hakim, Pascal Monasse, and Said Raghay. Simultaneous deconvolution and denoising using a second order variational approach applied to image super resolution. Computer Vision and Image Understanding, 168:50-63, 2018. doi:10.1016/j.cviu.2017.08.007.

[25] Amine Laghrib, Abdelghani Ghazdali, Abdelilah Hakim, and Said Raghay. A multi-frame super-resolution using diffusion registration and a nonlocal variational image restoration. Computers & Mathematics with Applications, 72(9):2535-2548, 2016. doi:10.1016/j.camwa.2016.09.013.

[26] Amine Laghrib, Aissam Hadri, Abdelilah Hakim, and Said Raghay. A new multiframe super-resolution based on nonlinear registration and a spatially weighted regularization. Information Sciences, 493:34-56, 2019. doi:10.1016/j.ins.2019.04.029.

[27] Amine Laghrib, Aissam Hadri, Moad Hakim, and Hssaine Oummi. An improved PDE-constrained optimization fluid registration for image multi-frame super resolution. RAIRO-Operations Research, 56(4):3047-3069, 2022. doi:10.1051/ro/2022137.

[28] Baraka Jacob Maiseli, Nassor Ally, and Huijun Gao. A noise-suppressing and edge-preserving multiframe super-resolution image reconstruction method. Signal Processing: Image Communication, 34:1-13, 2015. doi:10.1016/j.image.2015.03.001.

[29] Mourad Nachaoui, L Afraites, and Amine Laghrib. A regularization by denoising super-resolution method based on genetic algorithms. Signal Processing: Image Communication, 99:116505, 2021. doi:10.1016/j.image.2021.116505.

[30] Mourad Nachaoui and Amine Laghrib. An improved bilevel optimization approach for image super-resolution based on a fractional diffusion tensor. Journal of the Franklin Institute, 359(13):7165-7195, 2022. doi:10.1016/j.jfranklin.2022.07.016.

[31] Stanley J. Osher, Martin Burger, Donald Goldfarb, Jinjun Xu, and Wotao Yin. An iterative regularization method for total variation-based image restoration. Multiscale Modeling & Simulation, 4(2):460-489, 2005. doi:10.1137/040605412.

[32] Konstantinos Papafitsoros and Carola-Bibiane Schönlieb. A combined first and second order variational approach for image reconstruction. Journal of Mathematical Imaging and Vision, 48(2):308-338, 2014. doi:10.1007/s10851-013-0445-4.

[33] Matan Protter, Michael Elad, Hiroyuki Takeda, and Peyman Milanfar. Generalizing the nonlocal-means to super-resolution reconstruction. IEEE Transactions on Image Processing, 18(1):36-51, 2008. doi:10.1109/TIP.2008.2008067.

[34] Diego A. Sorrentino and Andreas Antoniou. Storage-efficient quasi-Newton algorithms for image super-resolution. In 16th International Conference on Digital Signal Processing, DSP 2009, pages 1-6. IEEE, 2009. doi:10.1109/ICDSP.2009.5201145.

[35] Heng Su, Nan Jiang, Ying Wu, and Jie Zhou. Single image super-resolution based on space structure learning. Pattern Recognition Letters, 34(16):2094-2101, 2013. doi:10.1016/j.patrec.2013.07.012.

[36] Kaicong Sun and Sven Simon. Bilateral spectrum weighted total variation for noisy-image super-resolution and image denoising. IEEE Transactions on Signal Processing, 69:6329-6341, 2021. doi:10.1109/TSP.2021.3127679.

[37] Tuomo Valkonen, Kristian Bredies, and Florian Knoll. Total generalized variation in diffusion tensor imaging. SIAM Journal on Imaging Sciences, 6(1):487-525, 2013. doi:10.1137/120867172.

[38] Shuyuan Yang, Min Wang, Yaxin Sun, Fenghua Sun, and Licheng Jiao. Compressive sampling based single-image super-resolution reconstruction by dual-sparsity and non-local similarity regularizer. Pattern Recognition Letters, 33(9):1049-1059, 2012. doi:10.1016/j.patrec.2012.02.006.

[39] Xianfeng Yang and Jian Yang. Efficient diffeomorphic metric image registration via stationary velocity. Journal of Computational Science, 30:90-97, 2019. doi:10.1016/j.jocs.2018.11.011.

[40] Y-L You and Mostafa Kaveh. Fourth-order partial differential equations for noise removal. IEEE Transactions on Image Processing, 9(10):1723-1730, 2000. doi:10.1109/83.869184.

[41] Qiangqiang Yuan, Liangpei Zhang, and Huanfeng Shen. Multiframe super-resolution employing a spatially weighted total variation model. IEEE Transactions on Circuits and Systems for Video Technology, 22(3):379-392, 2012. doi:10.1109/TCSVT.2011.2163447.

[42] Linwei Yue, Huanfeng Shen, Jie Li, Qiangqiang Yuan, Hongyan Zhang, and Liangpei Zhang. Image super-resolution: The techniques, applications, and future. Signal Processing, 128:389-408, 2016. doi:10.1016/j.sigpro.2016.05.002.

[43] Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a Gaussian denoiser: Residual learning of deep cnn for image denoising. IEEE Transactions on Image Processing, 26(7):3142-3155, 2017. doi:10.1109/TIP.2017.2662206.

[44] Wen-Yi Zhao, Harpreet S. Sawhney, Michael W. Hansen, and Supun Samarasekera. Super-fusion: A super-resolution method based on fusion. In 16th International Conference on Pattern Recognition, ICPR 2002, pages 269-272. IEEE Computer Society, 2002. doi:10.1109/ICPR.2002.1048290.

Bibtex

@article{sacscuza:laghrib22aefrimfsr,
  title={An Enhanced Fluid Registration for Image Multi-Frame Super Resolution},
  author={A. Laghrib , A. Hadri , M. Hakim},
  journal={Scientific Annals of Computer Science},
  volume={32},
  number={2},
  organization={Alexandru Ioan Cuza University, Ia\c si, Rom\^ania},
  year={2022},
  pages={255-283},
  publisher={Alexandru Ioan Cuza University Press, Ia\c si},
  doi={10.7561/SACS.2022.2.255}
}