The Spurs coach is accusing the official of showing "no common sense" over a key incident in the first half.
Tottenham, who went into the tie unbeaten away from home in all competitions this season, fell behind when Fabian Schar headed in from a corner after a moment of confusion that left Frank furious.
The controversy came in the 24th minute when Djed Spence lost his boot, briefly delaying the corner. The defender took time to put it back on, and as soon as he had it secured, though still scrambling to get back into position, Sandro Tonali swung the ball into the box and Schar, the player Spence was meant to be marking, powered his header past Antonin Kinsky.
Frank was left fuming on the touchline, confronting the fourth official as Tottenham players surrounded Kavanagh to protest.
"I think we conceded two goals because of two mistakes," Frank said. "Unfortunately, the referee was part of the mistake in the first one. You can clearly see on the footage that Djed's boot is completely off.
"It's common sense to allow him a few seconds to get ready. Instead, they took the corner as soon as he had it back on."
Frank admitted Tottenham were to blame for Newcastle's second goal, when Kinsky misjudged a cross that allowed Nick Woltemade to head into an empty net. But the Dane insisted the opener changed the course of the game.
"To make it perfect, Djed's man to mark was Schar, and he's the one who scored," Frank added. "That's why we were so animated. It was a big mistake, and those small margins decided the game today."
The defeat ended Tottenham's hopes of a first domestic cup triumph since 2008 and continued Newcastle's strong cup form under Eddie Howe, with the Magpies booking their place in the Carabao Cup quarter-finals.