Amakhosi were due to compete in the annual one-day friendly competition alongside AmaZulu, Njampela FC and Eswatini champions Nsingizini Hotspurs this Saturday.
However, the Zululand District Municipality issued a statement announcing the withdrawal of the Soweto giants, citing the CAF commitments.
In the statement, the Municipality acknowledged receipt of an official letter from Kaizer Chiefs' club Chairman explaining the reason for the withdrawal.
"This difficult decision has been necessitated primarily by unforeseen changes to our international fixture calendar following the official draw for the CAF Confederation Cup group stages, which was conducted earlier this week," stated Dr Kaizer Motaung in the letter.
"The draw has imposed specific travel logistics and preparatory requirements that directly conflict with our participation in the tournament.
"As a club with a proud history on the continent, our commitments to CAF competitions must take precedence and the revised schedule leaves us unable to fulfil our obligation to your event without compromising our performance and obligations."
Richards Bay will replace the Glamour Boys at the tournament on Saturday, where they will face Hotspurs FC in the second semi-final.
Meanwhile, Kaizer Chiefs will travel to Egypt next week to face El Masry for their first Group D encounter next Sunday.