The Loose Women host starred alongside Whiteley on the Channel 4 gameshow from its 1982 premiere until his death in 2005.
"It still hurts very much. We worked together for 23 years," she said during filming of Piers Morgan's Life Stories. "It was a total and utter joy to be with him. It was special, we loved each other - though not in that way."
Declaring that Countdown "wasn't the same after Richard died", Vorderman admitted that she could somewhat understand why producers "wanted [her] out" in time for its 2009 revamp.
"It was six months before my contract was due to end and I knew there were budget cuts," she recalled. "I said I'd take a third, or a half, but the new bosses had decided they wanted a fresh face. They gave me 48 hours to make the decision - why?"
On Whiteley's successors Des Lynam and Des O'Connor, she said: "[They] were wonderful but maybe I couldn't make the same chemistry. Maybe the bosses were right.
"It upsets me they've not invited me back for guest appearances. They had their 5,000th episode and I wasn't asked.
"I wish it well. I'd go back if they invited me. It's the 30th year of Channel 4 next year. Countdown was the first show and I was the first woman."
Vorderman added that her Countdown role has in the past brought with it a fair amount of unwanted attention.
"I have had stalkers over the years," she said. "The police deal with it but it is very scary."
"One man kept turning up where we filmed Countdown in Leeds, which was scary. It was sad as he'd been sectioned and thought I was talking to him through the TV."
Vorderman's episode of Piers Morgan's Life Stories will air later this year.
> Carol Vorderman rules out Countdown hosting role