It took until the 58th minute for Bruno Guimaraes to break the deadlock, and Bukayo Saka finally sealed the result with his 98th minute penalty after Reinildo Mandava was dismissed.
The result saw Mikel Arteta's team move onto 12 points after four games at the top of the standings
Regis Le Bris's team have recorded just four points and occupy 14th place.
Saka wasted an early effort with a mistimed shot from just eight yards that was comfortably saved by Robin Roefs as Arsenal started on the front foot.
Daniel Ballard was close to connecting to a superb Granit Xhaka delivery as the action swung to the other end.
Gabriel Magalhaes was wide with a speculative effort but Kai Havertz forced a fine save from Roefs.
The Gunners were largely dominant, but Sunderland continued to ask questions, although Enzo Le Fee directed his next effort well wide.
There was a scare for the Gunners as Ballard was fouled by Ezri Konsa in the box in the 54th minute but David Raya saved Le Fee's powerful spot kick.
Half-time substitute Guimaraes then curled a spectacular long-range strike into the top corner of the goal for a long-awaited opener.
Saka missed another close-range effort before slotting home from the penalty mark late on after being brought down by Reinildo in the area.
After the game, Arteta hit out at the award of a penalty against his side, saying on SkySports: "This is not acceptable at this level.
"It was a beautiful game of football with two teams, with an atmosphere in the stadium that was unbelievable.
"Sunderland are so well organised, they are so good in so many phases, they have great players and they create a lot of issues. We have to cope with all of that in a really difficult week.
"And on top of that, we have to go through what we have done today, that, in my opinion, is not acceptable.
"I watched it 20 times and I cannot find a way to understand how they think to give a penalty in that situation, or the way the VAR has to intervene."
Le Bris told BBC: "Frustrating result. Good game, especially in the second half. I think we went toe to toe with them and they are flying at the moment.
"We had the opportunity to score, I think it was deserved, [we] didn't score and thirty second after we conceded the goal. This is the level. I think they are relentless. When you don't seize your opportunities it's hard.
"We have to believe and then it's a question of fine margins. Those fine margins are so hard to improve because it is the last percent and you need to work hard to get this one. We are in a good place. It's important to digest quickly because we have an impressive week ahead."
Arsenal next play Ipswich Town in a EFL Cup third round clash on Tuesday as Sunderland take on AZ Alkmaar in the Europa League on Wednesday.
Alisson Becker has warned Liverpool they must find the "right mindset" to cope with the demands of a season across multiple competitions after a disappointing 0-0 draw with Fulham at Anfield.
Tottenham's frustrating start to the Premier League season continued as they were held to a goalless draw by Everton at home, leaving Roberto de Zerbi's side with just two points from four matches.
Chelsea were held to a 2-2 draw at home by newly-promoted Hull City, while Liverpool failed to score in a goalless stalemate against Fulham at Anfield in the Premier League.
Joao Gomes made an immediate impression on his Champions League debut as Aston Villa claimed a 3-2 victory over Club Brugge in Belgium on Tuesday.
