The mature Wittgenstein's groundbreaking analyses of sense and the logical must-and the powerful new method that made them possible-were the result of a multi-year process of writing, re-arranging, re-writing and one large-scale revision that eventually produced the Philosophical Investigations and RFM I. In contrast, his struggles during the same period with questions of arithmetic and higher mathematics remained largely in first-draft form, and he drops the topic entirely after 1945. In this paper, I argue that Wittgenstein's new method can be applied to the cases of arithmetic and set theory and that the result is innovative, recognizably Wittgensteinian, and independently appealing. I conclude by acknowledging the reasons Wittgenstein himself might have had to resist applying his own proven method to the case of mathematics-particularly to set theory-and by indicating why I think those reasons are ultimately unsound.