Lay abstract People with autism spectrum disorders sometimes report difficulties with following observed conversations in social situations, especially those where several people are interacting with each other. But this has rarely been investigated directly. This study determines whether people with autism spectrum disorders do indeed have problems following observed conversations even when they perform well on IQ tests and investigates two possible reasons for any difficulty found: (1) some people may have a problem integrating stimuli from multiple speakers; (2) following a conversation between many people might make particularly high demands on mentalising abilities. We used a variety of video clips of people conversing together to investigate these two possibilities in 30 adults diagnosed with autism spectrum disorder and 43 age- and IQ-matched typical-developing adults. We found that it was the amount of mentalising required, rather than the number of people involved, which caused problems for people with autism spectrum disorder in following conversations. Furthermore, when the autism spectrum disorder participants made a mistake, the error they made was frequently not the same error that typically developed participants made, and the autism spectrum disorder population made a more varied set of errors than the typically developed participants. Together, these results suggest that people with autism spectrum disorders observe significant problems with following conversations between many people when they contain a lot of mentalising material, but where they do make a mistake, the conclusions they draw from the conversation they are observing may have a more complex cause than an impairment in mentalising alone.