"A lot of Walliams's comments need editing out. It certainly can't go on for the rest of filming," a source told The Sun. "If we don't get clean comments, that act won't get a big showing.
"It's a learning curve. He was very funny but we need him to be more family-friendly. But we're still confident we made the right choice. Simon Cowell loves the banter and thinks the new panel has great chemistry."
Another insider told The Mirror that bosses are expected to "have a word" with Walliams before the next set of auditions.
"Simon picked David because he likes his sense of humour and knew he'd be a success," an audition onlooker claimed. "But in doing so he may just have put himself in the shade - and that's somewhere he doesn't enjoy being."