Architecture of Processing and Analysis Systems for Big Astronomical Data

被引：0
|
作者：
Kolosov, Ivan [1 ]
Gerasimov, Sergey [1 ]
Meshcheryakov, Alexander [2 ,3 ]
机构：
[1] Lomonosov Moscow State Univ, Fac Computat Math & Cybernet, Moscow, Russia
[2] Russian Acad Sci, Space Res Inst, Profsoyuznaya 84-3, Moscow 117997, Russia
[3] Kazan Fed Univ, Kremlevskaya Str 18, Kazan 420008, Russia
来源：
ASTRONOMICAL DATA ANALYSIS SOFTWARE AND SYSTEMS XXVI | 2019年 / 521卷
基金：
俄罗斯基础研究基金会;
关键词：
D O I：
暂无
中图分类号：
P1 [天文学];
学科分类号：
0704 ;
摘要：
This work explores the use of big data technologies deployed in the cloud for processing of astronomical data. We have applied Hadoop and Spark to the task of co-adding astronomical images. We compared the overhead and execution time of these frameworks. We conclude that the performance of both frameworks is generally on par. The Spark API is more flexible, which allows one to easily construct astronomical data processing pipelines.
引用
收藏
页码：428 / 430
页数：3
相关论文
共 50 条
  • [31] End-Edge-Cloud Collaborative System: A Video Big Data Processing and Analysis Architecture
    Xing, Peiyin
    Wang, Yaowei
    Peng, Peixi
    Tian, Yonghong
    Huang, Tiejun
    THIRD INTERNATIONAL CONFERENCE ON MULTIMEDIA INFORMATION PROCESSING AND RETRIEVAL (MIPR 2020), 2020, : 233 - 236
  • [32] Analysis and processing aspects of data in big data applications
    Rahul, Kumar
    Banyal, Rohitash Kumar
    Goswami, Puneet
    JOURNAL OF DISCRETE MATHEMATICAL SCIENCES & CRYPTOGRAPHY, 2020, 23 (02): : 385 - 393
  • [33] An Architecture for Near-Data Processing Systems
    Vermij, Erik
    Hagleitner, Christoph
    Fiorin, Leandro
    Jongerius, Rik
    van Lunteren, Jan
    Bertels, Koen
    PROCEEDINGS OF THE ACM INTERNATIONAL CONFERENCE ON COMPUTING FRONTIERS (CF'16), 2016, : 357 - 360
  • [34] DECH: A Software Package for Astronomical Spectral Data Processing and Analysis
    Galazutdinov, G. A.
    ASTROPHYSICAL BULLETIN, 2022, 77 (04) : 519 - 529
  • [35] Processing of the Astronomical Image Data obtained from UWFC Optical Systems
    Rerabek, Martin
    Pata, Petr
    Koten, Pavel
    IMAGE RECONSTRUCTION FROM INCOMPLETE DATA V, 2008, 7076
  • [36] A Proposed Big Data Architecture Using Data Lakes for Education Systems
    Oukhouya, Lamya
    El Haddadi, Anass
    Er-Raha, Brahim
    Asri, Hiba
    Laaz, Naziha
    EMERGING TRENDS IN INTELLIGENT SYSTEMS & NETWORK SECURITY, 2023, 147 : 53 - 62
  • [37] Distribution, Data, Deployment Software Architecture Convergence in Big Data Systems
    Gorton, Ian
    Klein, John
    IEEE SOFTWARE, 2015, 32 (03) : 78 - 85
  • [38] DECH: A Software Package for Astronomical Spectral Data Processing and Analysis
    G. A. Galazutdinov
    Astrophysical Bulletin, 2022, 77 : 519 - 529
  • [39] Big Data Analysis with Signal Processing on Graphs
    Sandryhaila, Aliaksei
    Moura, Jose M. F.
    IEEE SIGNAL PROCESSING MAGAZINE, 2014, 31 (05) : 80 - 90
  • [40] Analysis and mathematical modeling of big data processing
    Kairat Imanbayev
    Bakhtgerey Sinchev
    Saulet Sibanbayeva
    Axulu Mukhanova
    Assel Nurgulzhanovа
    Nurgali Zaurbekov
    Nurbike Zaurbekova
    Natalya V. Korolyova
    Lyazzat Baibolova
    Peer-to-Peer Networking and Applications, 2021, 14 : 2626 - 2634
12345