Spurs have collected just two points from their opening five matches, with De Zerbi frustrated by another match in which his side struggled to maintain its level after a promising start.
"It's difficult to explain because, in my career, 14 years as a coach, I've never seen one team like us," De Zerbi said. "So fragile, so inconsistent."
Tottenham had created several chances in the first half but fell behind while Micky van de Ven was temporarily off the pitch receiving treatment following a clash of heads. De Zerbi felt his side should have coped better despite being reduced to 10 men.
Villa then doubled their lead through Nicolas Jackson, who scored from a long clearance by goalkeeper Zion Suzuki. De Zerbi was particularly unhappy with the goal after Tottenham had specifically prepared for Villa's increased use of long balls.
"We knew this season Aston Villa are playing 86% more long balls than last season and we spoke in many meetings about it," he said.
The Spurs boss also highlighted a lack of communication and leadership when games turn against his side.
"Maybe the communication between the players [is the problem], maybe the leaders have to be stronger," De Zerbi said. "We need leaders to drive and lead the team."
De Zerbi also demanded more from Mateus Fernandes and Dominic Solanke, while acknowledging that Solanke's physical condition was not yet where it needed to be.
"I like Dom as a player, and I stay with him until the end," he said. "He needs to stay better, to feel better. The physical condition is not good enough."
Tottenham eventually scored twice late on after Villa had moved 3-0 ahead, but De Zerbi remained unhappy with the manner of the defeat.
"We can't lose the game in this way," he said. "We can't lose balance just because we are losing."
Villa manager Unai Emery praised his players for showing belief and committing to the defensive side of their game as they secured their first Premier League win of the season.
"The players did well because they believed," Emery said.
Tottenham now have the international break to regroup before returning to Premier League action away to Manchester United.
Michael Carrick has urged Manchester United to ensure the refereeing controversy from their derby defeat to Manchester City does not define their season, while admitting the officials' apology meant little.
Brighton were "horrible to play against" as they produced the intensity and togetherness needed to beat Arsenal 3-0 at the Amex, according to Fabian Hurzeler.
Mikel Arteta accused Arsenal of "not respecting the game" as their unbeaten Premier League start came to a crashing end with a 3-0 defeat at Brighton.
Tottenham Hotspur's difficult start to the Premier League season continued as Aston Villa secured a dramatic 3-2 victory to leave Roberto De Zerbi under increasing pressure.
