Ole Gunnar Solskjaer’s side, showing nine changes from the one that lost at West Ham United in the Premier League on Sunday, laboured against their League One opponents before taking the lead through Mason Greenwood in the 68th minute.
However, 16-year-old Luke Matheson equalised eight minutes later, to send the tie into a shootout.
United prevailed, although, with keeper Sergio Romero making one save to spare their blushes as the hosts successfully converted all their spot-kicks and Daniel James slotted home the winner.
Although United struggled, they at least avoided the complete humiliation suffered by West Ham United who were thrashed 4-0 at League One Oxford United.
Top flight Bournemouth were also humbled by third-tier opposition, losing 2-0 at Burton Albion.
Sheffield United joined the Premier League clubs to depart the competition as they lost 1-0 at home to Sunderland.
There were no alarms for Premier League leaders Liverpool or Chelsea.
Liverpool won 2-0 at MK Dons to set up a clash with Arsenal while Chelsea crushed Grimsby Town 7-1 to give manager Frank Lampard a first home win since taking over in the summer.
Chelsea will face Manchester United in round four.
Aston Villa won an all Premier League match against Brighton and Hove Albion, prevailing 3-1 to set up a fourth-round meeting with Wolverhampton Wanderers who also needed penalties to get past Championship side Reading.