Bulelani Nikani scored the only goal at the Princess Magogo Stadium after just 15 minutes.
The three points lifted his side up to fifth place on the standings with 19 points from 12 games - and left Usuthu in sixth with 18 points - although from one game less.
It was a disappointing result after a run of three games without defeat and following a 3-0 hammering of Chippa United last weekend.
Speaking after the game, Zwane admitted that they wasted an opportunity to move up in the standings.
He said on IOL: "Last season, we had five opportunities to move into fifth position, and we didn't take any of them.
"It was easy to get into position five. Whenever there was a game for us to move higher up, we dropped points. That's the mentality I need to fix.
"We don't need that. We need to take our chances and ensure that, by hook or by crook, we see ourselves in a top position this time around. But it's a work in progress. I'm just disappointed that we lost the game."
Zwane also revealed his concern at a growing injury list after defender Keagan Allan was forced off against Polokwane.
"I'm still going to meet with the medical team, and they'll give their feedback to the technical team," explained the coach.
"However, it's a worrying factor because we've lost a few important players from the starting lineup. That also breaks the rhythm of the team. It's worrying the way we're losing players, especially the core of the team."
Kaizer Chiefs will attempt to steady a faltering campaign when they travel to the uMhlathuze Sports Complex to face Richards Bay on Tuesday evening in a Betway Premiership clash.
Sekhukhune United defender Ryan Baartman has reportedly been forced to retire from football at 28-years-old due to a complicated back injury.
We take a look at five talking points from the Betway Premiership action last week where valuable points and goals led to surprising shifts on the table.
Marumo Gallants have confirmed that the club have suspended head coach Alexandre Lafitte for "issues relating to a failure to comply with contractual duties".
