The Only Way Is Essex star, who struck up a close bond with Frankie Cocozza in the house, left the show last Friday.
"I wasn't surprised about Kirk at all because I figured that, while we were seeing him running around doing seven tasks per task and endlessly buffooning with Frankie, he may have been perceived as being too eager and too aggressive and overdoing it," Madsen told Romeo Dunn, who disagreed.
The Reservoir Dogs actor went on: "Maybe he'd latched onto Frankie as his court jester, completely dependent upon Frankie."
So Solid Crew rapper Dunn commented: "I did have that thought as well - if they came in separately, they would've been two completely different people. If Kirk came on his own and Frankie wasn't here, he would've had a different mindset."
On being head-to-head with Natalie Cassidy, who also exited the house last week, Madsen added: "The one I was definitely bewildered by was Natalie. Everyone else is gone, I get that, but not her.
"She was staring at me like, 'You're leaving Michael. I'm sorry, I know it's you'. When they said her name man, her face went pale-white. She was completely shocked... as was everyone."
> Video: Michael Madsen judges poetry task