The Gunners, who rotated their squad with six changes, took an early lead when Gabriel Martinelli tapped in a rebound after a Viktor Gyokeres shot hit the post in the 13th minute.
Despite the early goal, the visitors proved resilient, and Arsenal struggled to find a decisive second, with a nervous second half seeing Olympiakos threaten an equaliser.
Goalkeeper David Raya had to make several key saves to preserve the clean sheet. However, substitute Bukayo Saka finally sealed the points in stoppage time, drilling a low shot past the Olympiakos keeper after an excellent pass from Man of the Match Martin Odegaard. The victory gives Arsenal six points from two games.
"I'm very happy, winning in the Champions League is always very complicated," said boss Mikel Arteta.
"We started really well, got the goal, a lot of threats and two or three big chances, but when you don't put them away in this competition you have to be careful.
"We had to make an incredible save to deny them a goal which you need in this competition as well.
"After that we had moments of overall dominance but we got a little bit deeper and every time they put balls in the box it was dangerous.
"We missed two big chances but found a way with Bukayo and breathed a little bit."
Tottenham snatch late point in Arctic draw
Tottenham had to show considerable character to salvage a 2-2 draw against Norwegian champions Bodo/Glimt in the Arctic Circle.
Thomas Frank's side were well below par, especially in a first half where they were arguably fortunate not to concede, as Kasper Hogh missed a 35th-minute penalty for the hosts.
The Norwegian side took a deserved lead with a fine double from Jens Petter Hauge in the 53rd and 66th minutes, leaving Spurs facing a significant upset.
However, Micky van de Ven quickly pulled one back two minutes later with a powerful header from a Pedro Porro cross.
Tottenham's persistence was rewarded in the 89th minute when an Archie Gray shot ricocheted off Bodo/Glimt defender Jostein Gundersen for a late own goal. The draw ensures Spurs stay unbeaten, moving to four points from two matches.
Manchester City secured a second successive Premier League victory at home against Sunderland, Chelsea were held scoreless away to Bournemouth while Tottenham managed to beat Brentford.
Aston Villa delivered a major jolt to the Premier League title race as Emiliano Buendía struck deep into stoppage time to seal a 2-1 win over Arsenal on Saturday.
Mikel Arteta has said that Aston Villa's blistering run of form has made them genuine contenders in the Premier League title race, as Arsenal prepare for a high-stakes trip to Villa Park.
Gabriel Jesus is expected to reassess his Arsenal future in January after falling down the pecking order during a year dominated by injury setbacks.
