MMM and MMMSynth: Clustering of heterogeneous tabular data, and synthetic data generation

被引:0
|
作者
Kumari, Chandrani [1 ,2 ]
Siddharthan, Rahul [1 ,2 ]
机构
[1] Inst Math Sci, Chennai, India
[2] Homi Bhabha Natl Inst, Mumbai, India
来源
PLOS ONE | 2024年 / 19卷 / 04期
关键词
D O I
10.1371/journal.pone.0302271
中图分类号
O [数理科学和化学]; P [天文学、地球科学]; Q [生物科学]; N [自然科学总论];
学科分类号
07 ; 0710 ; 09 ;
摘要
We provide new algorithms for two tasks relating to heterogeneous tabular datasets: clustering, and synthetic data generation. Tabular datasets typically consist of heterogeneous data types (numerical, ordinal, categorical) in columns, but may also have hidden cluster structure in their rows: for example, they may be drawn from heterogeneous (geographical, socioeconomic, methodological) sources, such that the outcome variable they describe (such as the presence of a disease) may depend not only on the other variables but on the cluster context. Moreover, sharing of biomedical data is often hindered by patient confidentiality laws, and there is current interest in algorithms to generate synthetic tabular data from real data, for example via deep learning. We demonstrate a novel EM-based clustering algorithm, MMM ("Madras Mixture Model"), that outperforms standard algorithms in determining clusters in synthetic heterogeneous data, and recovers structure in real data. Based on this, we demonstrate a synthetic tabular data generation algorithm, MMMsynth, that pre-clusters the input data, and generates cluster-wise synthetic data assuming cluster-specific data distributions for the input columns. We benchmark this algorithm by testing the performance of standard ML algorithms when they are trained on synthetic data and tested on real published datasets. Our synthetic data generation algorithm outperforms other literature tabular-data generators, and approaches the performance of training purely with real data.
引用
收藏
页数:17
相关论文
共 50 条
  • [41] DeepTLF: robust deep neural networks for heterogeneous tabular data
    Borisov, Vadim
    Broelemann, Klaus
    Kasneci, Enkelejda
    Kasneci, Gjergji
    INTERNATIONAL JOURNAL OF DATA SCIENCE AND ANALYTICS, 2023, 16 (01) : 85 - 100
  • [42] DeepTLF: robust deep neural networks for heterogeneous tabular data
    Vadim Borisov
    Klaus Broelemann
    Enkelejda Kasneci
    Gjergji Kasneci
    International Journal of Data Science and Analytics, 2023, 16 : 85 - 100
  • [43] FinDiff: Diffusion Models for Financial Tabular Data Generation
    Sattarov, Timur
    Schreyer, Marco
    Borth, Damian
    PROCEEDINGS OF THE 4TH ACM INTERNATIONAL CONFERENCE ON AI IN FINANCE, ICAIF 2023, 2023, : 64 - 72
  • [44] Automatic Generation of Conversational Interfaces for Tabular Data Analysis
    Gomez-Vazquez, Marcos
    Cabot, Jordi
    Clariso, Robert
    PROCEEDINGS OF THE 6TH CONFERENCE ON ACM CONVERSATIONAL USER INTERFACES, CUI 2024, 2024,
  • [45] Synthetic Tabular Data Validation: A Divergence-Based Approach
    Apellaniz, Patricia A.
    Jimenez, Ana
    Arroyo Galende, Borja
    Parras, Juan
    Zazo, Santiago
    IEEE ACCESS, 2024, 12 : 103895 - 103907
  • [46] Synthetic Data Generation for Data Envelopment Analysis
    Lychev, Andrey V.
    DATA, 2023, 8 (10)
  • [47] LONGITUDINAL CLUSTERING FOR HETEROGENEOUS BINARY DATA
    Zhu, Xiaolu
    Tang, Xiwei
    Qu, Annie
    STATISTICA SINICA, 2021, 31 (02) : 603 - 624
  • [48] Fusing heterogeneous fuzzy data for clustering
    Hathaway, RJ
    Rogers, GW
    Bezdek, JC
    Pedrycz, W
    SIGNAL PROCESSING, SENSOR FUSION, AND TARGET RECOGNITION VI, 1997, 3068 : 559 - 568
  • [49] Tabular Data Anomaly Detection Based on Density Peak Clustering Algorithm
    Liang, Dong
    Wang, Jun
    Zhang, Wenping
    Liu, Yuqi
    Wang, Lei
    Zhao, Xiaoyong
    2022 INTERNATIONAL CONFERENCE ON BIG DATA, INFORMATION AND COMPUTER NETWORK (BDICN 2022), 2022, : 16 - 21
  • [50] SYNTHETIC PRECIPITATION DATA GENERATION
    ABTEW, W
    MORAS, RG
    CAMPBELL, KL
    COMPUTERS & INDUSTRIAL ENGINEERING, 1990, 19 (1-4) : 582 - 586