Exactly-Once Semantics with Real-Time Data Pipelines

被引:0
|
作者
Rastogi, Avnish Kumar [1 ]
Malik, Naveen [2 ]
Hooda, Sakshi [3 ]
机构
[1] HCL Technol, Noida, India
[2] Royal Bank Scotland, Noida, India
[3] Surajmal Inst Technol, Delhi, India
来源
AMBIENT COMMUNICATIONS AND COMPUTER SYSTEMS, RACCCS 2017 | 2018年 / 696卷
关键词
Exactly once processing; Spark; Streaming; Distributed; Kafka; Redis; Vertica; Nosql;
D O I
10.1007/978-981-10-7386-1_26
中图分类号
TP [自动化技术、计算机技术];
学科分类号
0812 ;
摘要
Real-time systems like IoT, recommendation systems, fraud detection systems often have a need of ensuring that the application processes the data only once. In real-time streaming applications there is often a possibility that a batch of data might be handed over to the application multiple times resulting in duplicate data being processed by the application. Any stream processing product cannot unilaterally guarantee exactly once processing semantics. It is true under certain assumptions or when the application and the stream processing framework collaborate in certain ways. In this paper, we present a design to address the problem of real-time streaming applications by achieving an end-to-end exactly once delivery. The main contribution of our work is to provide solution to the complex task of recovering the application state from application restarts, network crashes, etc., and detecting and filtering out of order duplicate data while maintaining a high throughput.
引用
收藏
页码:293 / 303
页数:11
相关论文
共 50 条
  • [21] An operational semantics and compiler for Real-Time specifications
    Puchol, C
    Stuart, DA
    Mok, AK
    INTEGRATED COMPUTER-AIDED ENGINEERING, 1998, 5 (03) : 187 - 206
  • [22] A compositional real-time semantics of STATEMATE designs
    Damm, W
    Josko, B
    Hungar, H
    Pnueli, A
    COMPOSITIONALITY: THE SIGNIFICANT DIFFERENCE, 1998, 1536 : 186 - 238
  • [23] COMPOSITIONAL SEMANTICS OF A REAL-TIME PROTOTYPING LANGUAGE
    KRAMER, B
    LUQI
    BERZINS, V
    IEEE TRANSACTIONS ON SOFTWARE ENGINEERING, 1993, 19 (05) : 453 - 477
  • [24] A real-time semantics of temporal logic of actions
    Kaminski, M
    Yariv, Y
    JOURNAL OF LOGIC AND COMPUTATION, 2003, 13 (06) : 921 - 937
  • [25] Real-Time Monitoring of Data Pipelines: Exploring and Experimentally Proving that the Continuous Monitoring in Data Pipelines Reduces Cost and Elevates Quality
    Narayanan, Shammy
    Maheswari, S.
    Zephan, Prisha
    EAI ENDORSED TRANSACTIONS ON SCALABLE INFORMATION SYSTEMS, 2024, 11 (04):
  • [26] A Real-Time Semantics for the IEC 61499 standard
    Lindgren, Per
    Lindner, Marcus
    Lindner, Andreas
    Vyatkin, Valeriy
    Pereira, David
    Pinho, Luis Miguel
    PROCEEDINGS OF 2015 IEEE 20TH CONFERENCE ON EMERGING TECHNOLOGIES & FACTORY AUTOMATION (ETFA), 2015,
  • [27] Formal semantics of UML with real-time constructs
    Shankar, S
    Asa, S
    UML 2003 - THE UNIFIED MODELING LANGUAGE, PROCEEDINGS: MODELING LANGUAGES AND APPLICATIONS, 2003, 2863 : 60 - 75
  • [28] Real-time squared: A real-time data set for real-time GDP forecasting
    Golinelli, Roberto
    Parigi, Giuseppe
    INTERNATIONAL JOURNAL OF FORECASTING, 2008, 24 (03) : 368 - 385
  • [29] A semantics-based approach to sensor data segmentation in real-time Activity Recognition
    Triboan, Darpan
    Chen, Liming
    Chen, Feng
    Wang, Zumin
    FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE, 2019, 93 : 224 - 236
  • [30] Real-Time Data ETL Framework for Big Real-Time Data Analysis
    Li, Xiaofang
    Mao, Yingchi
    2015 IEEE INTERNATIONAL CONFERENCE ON INFORMATION AND AUTOMATION, 2015, : 1289 - 1294