Architecture of Processing and Analysis Systems for Big Astronomical Data

被引：0
|
作者：
Kolosov, Ivan [1 ]
Gerasimov, Sergey [1 ]
Meshcheryakov, Alexander [2 ,3 ]
机构：
[1] Lomonosov Moscow State Univ, Fac Computat Math & Cybernet, Moscow, Russia
[2] Russian Acad Sci, Space Res Inst, Profsoyuznaya 84-3, Moscow 117997, Russia
[3] Kazan Fed Univ, Kremlevskaya Str 18, Kazan 420008, Russia
来源：
ASTRONOMICAL DATA ANALYSIS SOFTWARE AND SYSTEMS XXVI | 2019年 / 521卷
基金：
俄罗斯基础研究基金会;
关键词：
D O I：
暂无
中图分类号：
P1 [天文学];
学科分类号：
0704 ;
摘要：
This work explores the use of big data technologies deployed in the cloud for processing of astronomical data. We have applied Hadoop and Spark to the task of co-adding astronomical images. We compared the overhead and execution time of these frameworks. We conclude that the performance of both frameworks is generally on par. The Spark API is more flexible, which allows one to easily construct astronomical data processing pipelines.
引用
收藏
页码：428 / 430
页数：3
相关论文
共 50 条
  • [1] Machine Learning for Astronomical Big Data Processing
    Xu, Long
    Yan, Yihua
    2017 IEEE VISUAL COMMUNICATIONS AND IMAGE PROCESSING (VCIP), 2017,
  • [2] Hadoop and Spark for Data Management, Processing and Analysis of Astronomical Big Data: Applicability and Performance
    Harischandra, Lloyd
    ASTRONOMICAL DATA ANALYSIS SOFTWARE AND SYSTEMS XXV, 2017, 512 : 41 - 44
  • [3] Research on big data processing and analysis architecture based on MongoDB
    1600, Academy of Sciences of the Czech Republic, Dolejskova 5, Praha 8, 182 00, Czech Republic (61):
  • [4] A reference architecture for serverless big data processing
    Werner, Sebastian
    Tai, Stefan
    FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE, 2024, 155 : 179 - 192
  • [5] A Reference Architecture for Big Data Systems
    Sang, Go Muan
    Xu, Lai
    de Vrieze, Paul
    PROCEEDINGS OF 2016 10TH INTERNATIONAL CONFERENCE ON SOFTWARE, KNOWLEDGE, INFORMATION MANAGEMENT & APPLICATIONS (SKIMA), 2016, : 370 - 375
  • [6] ASTROIDE: A Unified Astronomical Big Data Processing Engine over Spark
    Brahem, Mariem
    Zeitouni, Karine
    Yeh, Laurent
    IEEE TRANSACTIONS ON BIG DATA, 2020, 6 (03) : 477 - 491
  • [7] Software Systems for Processing and Analysis of Big Data and Event Streams
    Stojnev, Aleksandra I.
    Stojanovic, Dragan H.
    2017 13TH INTERNATIONAL CONFERENCE ON ADVANCED TECHNOLOGIES, SYSTEMS AND SERVICES IN TELECOMMUNICATIONS (TELSIKS), 2017, : 128 - 131
  • [8] Astronomical big data processing using machine learning: A comprehensive review
    Sen, Snigdha
    Agarwal, Sonali
    Chakraborty, Pavan
    Singh, Krishna Pratap
    EXPERIMENTAL ASTRONOMY, 2022, 53 (01) : 1 - 43
  • [9] Astronomical big data processing using machine learning: A comprehensive review
    Snigdha Sen
    Sonali Agarwal
    Pavan Chakraborty
    Krishna Pratap Singh
    Experimental Astronomy, 2022, 53 : 1 - 43
  • [10] Data Modifications in Blockchain Architecture for Big-Data Processing
    Tulkinbekov, Khikmatullo
    Kim, Deok-Hwan
    SENSORS, 2023, 23 (21)
12345