Walsh was booed by the Manchester crowd after he had a spat with Barlow about a male singer, who the Irish judge didn't rate.
Barlow was shocked at Walsh's verdict, claiming that the singer had the potential to "shake the show up" and that he "loved it".
A frustrated Walsh retaliated when the crowds mocked him and shouted "shut up" at the hecklers.
Mel B finished her stint as a guest judge yesterday and the Spice Girls star is now believed to be one of the frontrunners to become the fourth judge full-time.
X Factor's filmed auditions continue in Glasgow, Newcastle and Cardiff later this month. 'I'm Outta Love' singer Anastacia is rumoured to be the guest judge at the Glasgow tryouts.
Meanwhile, Walsh has been given a surprise call-up to fill in for a sick Simon Cowell during the X Factor USA auditions in Kansas City.