The budget's plan to replace Medicare with a less generous voucher system has proved a tough sell for the party, and judging by the reaction on the right, Gingrich's words touched on a particularly sore spot for the party.
Cantor went so far as to hint Gingrich may have ended his nascent campaign entirely.
"I think that many have said now he's finished," Cantor said. "I haven't had a chance to really dissect what in the world he's thinking ... so I probably would reserve judgment on that."
Paul Ryan, the architect of the House GOP budget, lit into Gingrich as well. "With allies like that, who needs the left?" he told radio host Laura Ingraham on Monday.
South Carolina Governor Nikki Haley, considered one of the party's most important endorsements because of her state's early primary, held nothing back in an interview with CNN on Tuesday.
"What he said was absolutely unfortunate," Haley said. "Here you've got Representative Ryan trying to bring common sense to this world of insanity, and Newt absolutely cut him off at the knees."
Dick Armey, who had a legendarily tempestuous relationship with Gingrich when they were in the House leadership together and is now a Tea Party organizer, told Politico that Newt was "confused and conflicted" on policy.
"We always say: Newt always has so many great ideas," Armey said. "Well yeah, but then he shifts between them at such a rate it's pretty hard to track it let alone keep up with it."
The conservative press wasn't any kinder, as contributors to the National Review and the editorial board of the Wall Street Journal unloaded on the former Speaker in a piece entitled "Gingrich to House GOP: Drop Dead."
"The episode reveals the Georgian's weakness as a candidate, and especially as a potential President -- to wit, his odd combination of partisan, divisive rhetoric and poll-driven policy timidity," they wrote.
Conservative columnist Charles Krauthammer told FOX News Gingrich's remarks were a "capital offense" that ruled out any chance of winning the nomination. "This is a big deal," he said. "He's done."
Gingrich, for his part, has sought to walk back his comments a bit. But on the basic point -- that the Ryan budget is "too big a jump" -- he has stuck to his guns.