Planet or brown dwarf? Inferring the companion mass in HD 100546 from the wall shape using mid-infrared interferometry

Context. Giant planets form in protoplanetary disks while these disks are still gas-rich, and can reveal their presence through the annular gaps they carve out. HD 100546 is a gas-rich disk with a wide gap between a radius of similar to 1 and 13 AU, possibly cleared out by a planetary companion or planetary system. Aims. We aim to identify the nature of the unseen companion near the far end of the disk gap. Methods. We used mid-infrared interferometry at multiple baselines to constrain the curvature of the disk wall at the far end of the gap. We used 2D hydrodynamical simulations of embedded planets and brown dwarfs to estimate the viscosity of the disk and the mass of a companion close to the disk wall. Results. We find that the disk wall at the far end of the gap is not vertical, but rounded-off by a gradient in the surface density. This gradient can be reproduced in hydrodynamical simulations with a single, heavy companion (greater than or similar to 30 ... 80 M-Jup) while the disk has a viscosity of at least alpha greater than or similar to 5 x 10(-3). Taking into account the changes in the temperature structure after gap opening reduces the lower limit on the planet mass and disk viscosity to 20 M-Jup and alpha = 2 x 10(-3). Conclusions. The object in the disk gap of HD 100546 that shapes the disk wall is most likely a 60(-40)(+20) M-Jup brown dwarf, while the disk viscosity is estimated to be at least alpha = 2 x 10(-3). The disk viscosity is an important factor in estimating planetary masses from disk morphologies: more viscous disks need heavier planets to open an equally deep gap.