Deriving an Optimal Noise Adding Mechanism for Privacy-Preserving Machine Learning

Differential privacy is a standard mathematical framework to quantify the degree to which individual privacy in a statistical dataset is preserved. We derive an optimal (epsilon, delta)-differentially private noise adding mechanism for real-valued data matrices meant for the training of models by machine learning algorithms. The aim is to protect a machine learning algorithm from an adversary who seeks to gain an information about the data from algorithm's output by perturbing the value in a sample of the training data. The fundamental issue of trade-off between privacy and utility is addressed by presenting a novel approach consisting of three steps: (1) the sufficient conditions on the probability density function of noise for (epsilon, delta)-differential privacy of a machine learning algorithm are derived; (2) the noise distribution that, for a given level of entropy, minimizes the expected noise magnitude is derived; (3) using entropy level as the design parameter, the optimal entropy level and the corresponding probability density function of the noise are derived.