The Crystal Palace manager was dismissed by Spurs in February, just eight months after taking charge, following a disappointing spell in north London.
Tottenham went on to finish 17th in the Premier League for a second consecutive season, while the club also underwent significant changes behind the scenes, including the departure of chairman Daniel Levy.
Frank has now returned to the Premier League with Palace but insists he has no regrets about his Tottenham experience and is not concerned about what his critics think.
"The interesting thing is that they only know a tenth of the truth. That's a tenth of the truth at Brentford, and a tenth of the truth at Tottenham," Frank said.
"That's just the way it is. And if you can't handle that as a coach or a player, then you shouldn't be in this business."
The Dane explained that his focus is on learning from his experiences and continuing to improve rather than worrying about outside opinions.
"I don't know if for me it's about the critics. It's more about what I can bring myself, what I can learn, what I can improve, how I can get better," he added.
Frank also acknowledged that everyone wants to be appreciated, but accepted that criticism comes with working in football.
"We all have got an ego. Some are better at controlling it than others," he said.
"You can say that you want everyone to love you and think you're the best. There's not one person that doesn't want that. But hey, it's football."
The 52-year-old revealed that he avoids social media and does not follow what is being written about him, helping him maintain his focus amid the scrutiny surrounding his Tottenham departure.
"The big thing is that I don't read anything. I'm not on social media. That's probably pretty good in this situation," Frank said.
Despite the disappointment of his short spell at Tottenham, Frank insists he has no bitterness towards his former club.
"No regrets for my part. I wish Tottenham and Spurs all the best for the future," he added.
Frank's immediate focus is now on his new challenge at Crystal Palace, who face Coventry City away from home on Saturday.