An efficient storage and service method for multi-source merging meteorological big data in cloud environment

被引:6
|
作者
Yang, Ming [1 ,3 ]
He, Wenchun [2 ]
Zhang, Zhiqiang [2 ]
Xu, Yongjun [2 ]
Yang, Heping [2 ]
Chen, Yufeng [1 ]
Xu, Xiaolong [3 ,4 ,5 ,6 ]
机构
[1] Zhejiang Meteorol Informat Network Ctr, Hangzhou, Zhejiang, Peoples R China
[2] Natl Meteorol Informat Ctr, Beijing, Peoples R China
[3] Nanjing Univ Informat Sci & Technol, Sch Comp & Software, Nanjing, Jiangsu, Peoples R China
[4] Nanjing Univ Informat Sci & Technol, Jiangsu Engn Ctr Network Monitoring, Nanjing, Jiangsu, Peoples R China
[5] Nanjing Univ, State Key Lab Novel Software Technol, Nanjing, Jiangsu, Peoples R China
[6] Michigan State Univ, Dept Comp Sci & Engn, E Lansing, MI 48824 USA
关键词
Multi-source merging sensors data; Meteorological data storage; Meteorological data service; Distributed NoSQL; Semi/unstructured data; COMPUTATION OFFLOADING METHOD; PRIVACY PRESERVATION; NETWORKS; INTERNET;
D O I
10.1186/s13638-019-1576-0
中图分类号
TM [电工技术]; TN [电子技术、通信技术];
学科分类号
0808 ; 0809 ;
摘要
With the development of the meteorological IoT (Internet of Things) and meteorological sensing network, the collected multi-source meteorological data have the characteristics of large amount of information, multidimensional and high accuracy. Cloud computing technology has been applied to the storage and service of meteorological big data. Although the constant evolution of big data storage technology is improving the storage and access of meteorological data, storage and service efficiency is still far from meeting multi-source big data requirements. Traditional methods have been used for the storage and service of meteorological data, and a number of problems still persist, such as a lack of unified storage structure, poor scalability, and poor service performance. In this study, an efficient storage and service method for multidimensional meteorological data is designed based on NoSQL big data storage technology and the multidimensional characteristics of meteorological data. In the process of data storage, multidimensional block compression technology and data structures are applied to store and transmit meteorological data. In service, heterogeneous NoSQL common components are designed to improve the heterogeneity of the NoSQL database. The results show that the proposed method has good storage transmission efficiency and versatility, and can effectively improve the efficiency of meteorological data storage and service in meteorological applications.
引用
收藏
页数:12
相关论文
共 50 条
  • [41] Multi-source heterogeneous data storage methods for omnimedia data space
    Zhuo, Wenbo
    INTERNATIONAL JOURNAL OF GRID AND UTILITY COMPUTING, 2024, 15 (3-4) : 314 - 322
  • [42] GEOCUBE: TOWARDS THE MULTI-SOURCE GEOSPATIAL DATA CUBE IN BIG DATA ERA
    Yue, Peng
    Shangguan, Boyi
    Zhang, Mingda
    Gao, Fan
    Cao, Zhipeng
    Jiang, Liangcun
    Fang, Zhe
    IGARSS 2020 - 2020 IEEE INTERNATIONAL GEOSCIENCE AND REMOTE SENSING SYMPOSIUM, 2020, : 3127 - 3130
  • [43] A DYNAMIC CLOUD BAYES NETWORK-BASED CLEANING METHOD OF MULTI-SOURCE UNSTRUCTURED DATA
    Yin Chao
    Liao Xinian
    Li Xiaobin
    PROCEEDINGS OF ASME 2022 17TH INTERNATIONAL MANUFACTURING SCIENCE AND ENGINEERING CONFERENCE, MSEC2022, VOL 2, 2022,
  • [44] A Deep Neural Collaborative Filtering Based Service Recommendation Method with Multi-Source Data for Smart Cloud-Edge Collaboration Applications
    Lin, Wenmin
    Zhu, Min
    Zhou, Xinyi
    Zhang, Ruowei
    Zhao, Xiaoran
    Shen, Shigen
    Sun, Lu
    TSINGHUA SCIENCE AND TECHNOLOGY, 2024, 29 (03): : 897 - 910
  • [45] Data Storage Security Algorithms for Multi Cloud Environment
    Ashalatha, R.
    Agarkhed, Jayashree
    Patil, Siddarama
    PROCEEDINGS OF THE 2016 IEEE 2ND INTERNATIONAL CONFERENCE ON ADVANCES IN ELECTRICAL & ELECTRONICS, INFORMATION, COMMUNICATION & BIO INFORMATICS (IEEE AEEICB-2016), 2016, : 686 - 690
  • [46] Location Recommendation for Enterprises by Multi-Source Urban Big Data Analysis
    Zhao, Guoshuai
    Liu, Tianlei
    Qian, Xueming
    Hou, Tao
    Wang, Huan
    Hou, Xingsong
    Li, Zhetao
    IEEE TRANSACTIONS ON SERVICES COMPUTING, 2020, 13 (06) : 1115 - 1127
  • [47] A Multi-Source Big Data Framework for Capturing and Analyzing Customer Feedback
    Ali, No'aman M.
    Novikov, Boris
    PROCEEDINGS OF THE 2021 IEEE CONFERENCE OF RUSSIAN YOUNG RESEARCHERS IN ELECTRICAL AND ELECTRONIC ENGINEERING (ELCONRUS), 2021, : 185 - 190
  • [48] Data Protection as a Service in the Multi-cloud Environment
    Colombo, Maurizio
    Asal, Rasool
    Hieu, Quang
    El-Moussa, Fadi Ali
    Sajjad, Ali
    Dimitrakos, Theo
    2019 IEEE 12TH INTERNATIONAL CONFERENCE ON CLOUD COMPUTING (IEEE CLOUD 2019), 2019, : 81 - 85
  • [49] Meteorological data layout and task scheduling in a multi-cloud environment
    Hao, Yongsheng
    Cao, Jie
    Wang, Qi
    Ma, Tinghuai
    Wang, Qin
    Zhang, Xin
    ENGINEERING APPLICATIONS OF ARTIFICIAL INTELLIGENCE, 2023, 126
  • [50] Multi-source heterogeneous cultural big data integration platforms design
    Liu P.
    Wang H.
    Zheng D.
    Liu F.
    Huazhong Keji Daxue Xuebao (Ziran Kexue Ban)/Journal of Huazhong University of Science and Technology (Natural Science Edition), 2021, 49 (02): : 95 - 101