However, the new Britain's Got Talent judge said that his wife Lara Stone was passionately against the plan, which would have taken place in New York.
In an interview with the Radio Times, the Little Britain comic revealed that the challenge was scheduled shortly after his Sport Relief swim across the Thames.
"Just after the Thames swim we went to New York," he explained. "Lara had to go for fashion week, and I wanted to get away. I'd had so much attention.
"I'm friends with Bear Grylls and he was there too. It was only scaling a rock face, but Lara got really upset and said no."
The idea was put on ice after Walliams was struck by another bout of the illness he contracted in the Thames.
Recalling the 140-mile swim, Walliams added that his wife Stone was completing her driving test on the day Walliams caught his stomach bug.
"She actually had her driving test on the day I got ill, so she heard it on the radio and was doubly upset she hadn't been there to look after me," Walliams went on. "I didn't like to see her upset, but it was great having her support."
Watch an unwell David Walliams speak about his Sport Relief 2011 challenge below: