This paper studies the impact of fiscal decentralization on public sector efficiency (PSE). We first use a theoretical framework that illustrates the two opposing forces that shape a non-monotonic effect of fiscal decentralization on PSE. Subsequently, we carry out an empirical analysis for 21 OECD countries, between 1970 and 2000. A country-level dataset is used to measure PSE in delivering education and health services and the new indices are regressed on well-established decentralization measures. Irrespective of whether PSE concerns education or health services, an inverted U-shaped relationship has been identified between government efficiency in providing these services and fiscal decentralization. This relationship is robust across several different specifications and estimation methods.