The Hammers twice led during normal time but were pegged back on both occasions before holding their nerve to win the shootout 5-3.
Jarrod Bowen was West Ham's hero in regulation time. The forward struck twice in the first half to put the hosts in control, continuing his excellent recent form in front of goal. However, Brentford responded through Igor Thiago, who scored two goals of his own to force extra-time after an intense second period.
West Ham pushed hard for a winner in the additional 30 minutes but could not find the decisive breakthrough. The tie was eventually settled from the spot.
Bowen stepped up confidently in the shootout, alongside Taty Castellanos, Callum Wilson, Tomas Soucek and Dinos Mavropanos, with all five converting clinical penalties. Goalkeeper Alphonse Areola then proved decisive when he saved Dango Ouattara's chipped effort, a risky attempt that ultimately cost Brentford.
Nuno admitted he could not watch the shootout unfold.
"I've been told the penalties were very good, but I didn't see them," he said. "What I saw in the days before was the players practising again and again. When the moment came, they were ready and they did their job amazingly."
The Portuguese coach was full of pride for his squad's resilience.
"The boys worked very hard and we deserved to go through. We played with purpose and kept pushing in normal time and extra-time. Overall, we were consistent and dominant."
Nuno also reserved special praise for the supporters.
"The atmosphere was amazing. Through difficult moments they were always there. We cannot thank them enough."
The victory secured West Ham's first FA Cup quarter-final in a decade and set up a home tie against Leeds United.
With five wins, four draws and just two defeats in their last eleven matches in all competitions, Nuno believes momentum and togetherness could prove vital in the closing weeks of the season.
"Every player knows they are important," he said. "That spirit inside the group is crucial for what lies ahead."
Mauricio Pochettino has admitted he misses English football and remains open to a Premier League return in the future, as speculation continues to link him with a comeback to former club Tottenham.
West Ham United midfielder Mateus Fernandes has found a silver lining in a turbulent season, with his form earning him a maiden Portugal call-up.
Liverpool are entering their FA Cup quarter-final against Manchester City on Saturday, 4 April with several first-team players possibly unavailable due to injury.
Former Tim Sherwood insists he has the solution to keep Tottenham in the Premier League, as pressure mounts on current boss Igor Tudor following a damaging run of results.
