Architecture of Processing and Analysis Systems for Big Astronomical Data

被引：0
|
作者：
Kolosov, Ivan [1 ]
Gerasimov, Sergey [1 ]
Meshcheryakov, Alexander [2 ,3 ]
机构：
[1] Lomonosov Moscow State Univ, Fac Computat Math & Cybernet, Moscow, Russia
[2] Russian Acad Sci, Space Res Inst, Profsoyuznaya 84-3, Moscow 117997, Russia
[3] Kazan Fed Univ, Kremlevskaya Str 18, Kazan 420008, Russia
来源：
ASTRONOMICAL DATA ANALYSIS SOFTWARE AND SYSTEMS XXVI | 2019年 / 521卷
基金：
俄罗斯基础研究基金会;
关键词：
D O I：
暂无
中图分类号：
P1 [天文学];
学科分类号：
0704 ;
摘要：
This work explores the use of big data technologies deployed in the cloud for processing of astronomical data. We have applied Hadoop and Spark to the task of co-adding astronomical images. We compared the overhead and execution time of these frameworks. We conclude that the performance of both frameworks is generally on par. The Spark API is more flexible, which allows one to easily construct astronomical data processing pipelines.
引用
收藏
页码：428 / 430
页数：3
相关论文
共 50 条
  • [41] Rhipe Platform for Big Data Processing and Analysis
    Jung, Byung Ho
    Shin, Ji Eun
    Lim, Dong Hoon
    KOREAN JOURNAL OF APPLIED STATISTICS, 2014, 27 (07) : 1171 - 1185
  • [42] A Review on Complex Event Processing Systems for Big Data
    Tawsif, K.
    Hossen, J.
    Raja, J. Emerson
    Jesmeen, M. Z. H.
    Arif, E. M. H.
    2018 FOURTH INTERNATIONAL CONFERENCE ON INFORMATION RETRIEVAL AND KNOWLEDGE MANAGEMENT (CAMP), 2018, : 2 - 7
  • [43] A survey of state management in big data processing systems
    Quoc-Cuong To
    Soto, Juan
    Markl, Volker
    VLDB JOURNAL, 2018, 27 (06): : 847 - 872
  • [44] A survey of state management in big data processing systems
    Quoc-Cuong To
    Juan Soto
    Volker Markl
    The VLDB Journal, 2018, 27 : 847 - 872
  • [45] Multilevel Ontologies for Big Data Analysis and Processing
    Popova, Maryna
    Globa, Larysa
    Noyogrudska, Rina
    PROCEEDINGS OF THE 9TH INTERNATIONAL CONFERENCE ON APPLIED INNOVATIONS IN IT, 2021, 9 (01): : 41 - 53
  • [46] Analysis and mathematical modeling of big data processing
    Imanbayev, Kairat
    Sinchev, Bakhtgerey
    Sibanbayeva, Saulet
    Mukhanova, Axulu
    Nurgulzhanova, Assel
    Zaurbekov, Nurgali
    Zaurbekova, Nurbike
    Korolyova, Natalya, V
    Baibolova, Lyazzat
    PEER-TO-PEER NETWORKING AND APPLICATIONS, 2021, 14 (05) : 2626 - 2634
  • [47] Big Data Processing Technologies in Distributed Information Systems
    Shakhovska, Nataliya
    Boyko, Nataliya
    Zasoba, Yevgen
    Benova, Eleonora
    10TH INT CONF ON EMERGING UBIQUITOUS SYST AND PERVAS NETWORKS (EUSPN-2019) / THE 9TH INT CONF ON CURRENT AND FUTURE TRENDS OF INFORMAT AND COMMUN TECHNOLOGIES IN HEALTHCARE (ICTH-2019) / AFFILIATED WORKOPS, 2019, 160 : 561 - 566
  • [48] Using Microservices and Event Driven Architecture for Big Data Stream Processing
    Zhelev, Svetoslav
    Rozeva, Anna
    PROCEEDINGS OF THE 45TH INTERNATIONAL CONFERENCE ON APPLICATION OF MATHEMATICS IN ENGINEERING AND ECONOMICS (AMEE'19), 2019, 2172
  • [49] Fog computing: from architecture to edge computing and big data processing
    Simar Preet Singh
    Anand Nayyar
    Rajesh Kumar
    Anju Sharma
    The Journal of Supercomputing, 2019, 75 : 2070 - 2105
  • [50] A Design of Pipelined Architecture for on-the-Fly Processing of Big Data Streams
    Algemili, Usamah
    Berkovich, Simon
    INTERNATIONAL JOURNAL OF ADVANCED COMPUTER SCIENCE AND APPLICATIONS, 2015, 6 (01) : 23 - 30
12345