Open Source Framework for Enabling HPC and Cloud Geoprocessing Services

José Miguel MontañanaPaolo MarangioAntonio Hervás

DOI 10.7160/aol.2020.120405
No 4/2020, December
pp. 61-76

Montañana, J. M., Marangio, P. and Hervás, A. (2020) “Open Source Framework for Enabling HPC and Cloud Geoprocessing Services", AGRIS on-line Papers in Economics and Informatics, Vol. 12, No. 4, pp. 61-76. ISSN 1804-1930. DOI 10.7160/aol.2020.120405.

Abstract

Geoprocessing is a set of tools that can be used to efficiently address several pressing chal-lenges for the global economy ranging from agricultural productivity, the design of transport networks, to the prediction of climate change and natural disasters. This paper describes an Open Source Framework developed, within three European projects, for Ena-bling High-Performance Computing (HPC) and Cloud geoprocessing services applied to agricultural challenges. The main goals of the European Union projects EUXDAT (EUro-pean e-infrastructure for eXtreme Data Analytics in sustainable developmenT), CYBELE (fostering precision agriculture and livestock farming through secure access to large-scale HPC-enabled virtual industrial experimentation environment empowering scalable big data analytics), and EOPEN (opEn interOperable Platform for unified access and analysis of Earth observatioN data) are to enable the use of large HPC systems, as well as big data management, user-friendly access and visualization of results. In addition, these projects focus on the development of software frameworks, and fuse Earth-observation data, such as Copernicus data, with non-Earth-observation data, such as weather, environmental and social media information. In this paper, we describe the agroclimatic-zones pilot used to validate the framework. Finally, performance metrics collected during the execution (up to 182 times speedup with 256 MPI processes) of the pilot are presented.

Keywords

High performance computing, cloud computing, big data; agriculture, land monitoring, geoprocessing.

References

  1. Amdahl, G. M. (1967) "Validity of the single processor approach to achieving large scale computing capabilities", In Proceedings of the Spring Joint Computer Conference (NY, USA), AFIPS ’67 (Spring), Association for Computing Machinery, p. 483-485. DOI 10.1145/1465482.1465560.
  2. Carnero, J. and Nieto, F. J. (2018) "Running simulations in HPC and cloud resources by implementing enhanced Tosca workflows", In 2018 International Conference on High Performance Computing & Simulation (HPCS), pp. 431-438. DOI 10.1109/HPCS.2018.00075.
  3. Dalcin, L. (2019) "MPI for Python". [Online]. Available: https://mpi4py.readthedocs.io/en/stable/index.html [Accessed: 15 Sept. 2020].
  4. Davy, S. (2020) "CYBELE Fostering Precision Agriculture And Livestock Farming Through Secure Access To Large-Scale Hpc-Enabled Virtual Industrial Experimentation Environment Empowering Scalable Big Data Analytic". [Online]. Available: https://www.cybele-project.eu [Accessed: 15 Sept. 2020].
  5. Esposito, R., Mastroserio, P., Tortone, G. and Taurino, F. M. (2003) "Standard FTP and GridFTP protocols for international data transfer in Pamela Satellite Space Experiment. In Proceedings from the 13th International Conference on Computing in High-Enery and Nuclear Physics (CHEP 2003).
  6. Li, Z. (2020) "Geospatial Big Data Handling with High Performance Computing: Current Approaches and Future Directions", In High Performance Computing for Geospatial Applications, pp. 53-76. E-ISBN 978-3-030-47998-5, ISBN 978-3-030-47997-8. DOI 10.1007/978-3-030-47998-5_4.
  7. Mineter, M. J., Dowers, S. and Gittings, B. M. (2000) "Towards a HPC Framework for Integrated Processing of Geographical Data: Encapsulating the Complexity of Parallel Algorithms", Transactions in GIS, Vol. 4, No. 3, pp. 245-262. E-ISSN 1467-9671. DOI 10.1111/1467-9671.00052.
  8. Montañana, J. M. (2010) "Providing Fault Tolerance in Interconnection Networks for PC Clusters: Efficient Mechanisms", Lap Lambert Academic Publishing. ISBN-13 978-3838318905, ISBN-10 9783838318905.
  9. Montañana, J. M. and Gorroñogoitia, J. (2020a) "Data mover plugin provides support for GridFTP data transfers to croupier cloudify orchestrator". [Online]. Available: https://github.com/ari-apc-lab/croupier/tree/euxdat/croupier_plugin/data_mover [Accessed: 20 Sept. 2020].
  10. Montañana, J. M., Hervás, A. and Hoppe, D. (2020b) "HPC-Enabled Geoprocessing Services Cases: EUXDAT, EOPEN, and CYBELE European Frameworks", In Proccedings of the 12th International Conference on Advanced Geographic Information Systems, Applications, and Services (GEOProcessing), pp 31-35.
  11. Muhollem, J. (2017) "Warm winter has put state’s apple crop at risk, expert warns", Pennsylva-nia State University. [Online]. Available: https://phys.org/news/2017-03-winter-state-apple-crop-expert.html [Accessed: 20 Sept. 2020].
  12. NCSA (2020) "uberftp - GridFTP-enabled client". Linux man page. [Online]. Available: https://linux.die.net/man/1/uberftp [Accessed: 20 Sept. 2020].
  13. EUXDAT (2020) "EUXDAT European e-Infrastructure for Extreme Data Analytics in Sustainable Development". [Online]. Available: https://www.euxdat.eu [Accessed: 20 Sept. 2020].
  14. Perakis, K., Lampathaki, F., Nikas, K., Georgiou, Y., Marko, O. and Maselyne, J. (2020) "CYBELE – Fostering precision agriculture & livestock farming through secure access to large-scale HPC enabled virtual industrial experimentation environments fostering scalable big data analytics", Computer Networks, Vol. 168, ISSN 1389-1286. DOI 10.1016/j.comnet.2019.107035.
  15. PESSL INSTRUMENTS GMBH (2020) "Stations and datalogger". [Online]. Available: http://metos.at/micrometos-clima [Accessed: 15 Aug. 2020].
  16. Serfon, C., Barisits, M., Beermann, T., Garonne, V., Goossens, L., Lassnig, M., Nairz, A. and Vigne, R., ATLAS Collaboration (2019) "Rucio, the next-generation Data Management system in ATLAS", Nuclear and Particle Physics Proceedings, Vol. 273-275, pp. 969-975. ISSN 2405-6014. DOI 10.1016/j.nuclphysbps.2015.09.151.
  17. SURFsara (2015) "Globus client. Grid Documentation v1.0." [Online]. Available: http://doc.grid.surfsara.nl/en/latest/Pages/Advanced/storage_clients/globus.html [Accessed: 15 Aug. 2020].
  18. EOPEN (2020) "EOPEN Open interoperable platform for unified access and analysis of earth observation data". [Online]. Available: https://eopen-project.eu [Accessed: 20 Sept. 2020].
  19. Vitasse, Y. and Rebetez, M. (2018) "Unprecedented risk of spring frost damage in Switzerland and Germany in 2017", Climatic Change, Vol. 149, pp. 233-246. E-ISSN 1573-1480, ISSN 0165-0009. DOI 10.1007/s10584-018-2234-y.
  20. Zhang, J. (2010) "Towards personal high-performance geospatial computing (hpc-g): perspectives and a case study", In Proceedings of the ACM SIGSPATIAL International Workshop on High Performance and Distributed Geographic Information Systems (HPDGIS), pp. 3-10. DOI 10.1145/1869692.1869694.

Full paper

  Full paper (.pdf, 3.19 MB).