Walliams has since apologised for his remarks, saying they were part of a 'private conversation' that was not intended to be shared