The Irish music manager also called the 'Young' singer "rough around the edges", accusing her of thinking she is "a bigger star than she actually is".
Contostavlos has since questioned Walsh's relevance, saying he has not had a hit in years.
"I'm not any less educated in music just because I'm a newbie on the scene. I still deserve my spot on the panel," she told Bliss.
"When was Westlife's last number one? Because mine was a couple of weeks ago. Ha! Don't give me all that 'selling millions of records' rubbish. Because how long ago was that?"
Speaking of a recent night out with One Direction, she added: "We got pretty hammered, it was quite funny.
"Everyone knows I'm a party animal. But they convinced me to go out then they all ended up absolutely hammered and I was there, still standing, going, 'I told you not to bring me out'."
Watch a trailer for The X Factor, which starts this Saturday (August 18) at 8pm on ITV1, below: