Duncan Smith, 64, was in high spirits while on ITV's Good Morning Britain and he couldn't stop smiling even as he refused to defend his former boss.
Dyer blasted David Cameron for calling the "whole Brexit thing".
He said: "So what's happened to that t**t David Cameron who brought it on? How comes he can scuttle off. He called all this on."
Insisting he had never called the former PM a "t**t", Duncan-Smith said he would "never ever go against" Dyer's opinion.
Duncan-Smith told Piers Morgan: "Danny Dyer is a member of royalty I know, so I would never ever go against him.
"All I would say is Danny Dyer has a short but rich vocabulary."
Dyer launched his rant on a one-off edition of Good Morning Britain.
The programme was hosted by Piers Morgan and Susanna Reid and featured a panel with Danny Dyer, Labour leader Jeremy Corbyn and Pamela Anderson.
DANNY DYER'S BREXIT RANT IN FULL
"Let's go back to this whole Brexit thing, when you're judging them because they don't know what Brexit is.
"Who knows about Brexit? No one has got a f****g clue. You watch Question Time, it's comedy.
"No one knows what it is. It's like this mad riddle no one knows what it is.
"So what's happened to that t**t David Cameron who brought it on? Let's be fair. How comes he can scuttle off. He called all this on.
"Where is he? He's in Europe. He's in Nice with his trotters up. Where is the geezer? I think he should be held accountable for it."
Duncan Smith's GMB appearance came after he attended a party for former cabinet minister Jonathan Aitken.
Aitken, who served time in jail, was made a deacon last week.
Duncan Smith, Jeffrey Archer and Diane Abbott were among a crowd of 200 expected to join his party at the nearby Old Bailey.
Source: Read Full Article