Specter voted to confirm Supreme Court nominees John Roberts and Samuel Alito, two others that drew fire from the left.
Specter said that his voting record -- which he called "independent" -- made it clear that he's the best choice to defeat likely Republican nominee Pat Toomey in the fall.
"People recognize that I'm the only guy who can beat Pat Toomey," he said. "You gotta face up to the issues and you gotta be tough and I'm the guy to do it."
Sestak, meanwhile, said that he was the one with the stronger chances against Toomey.
"[Specter] is behind Toomey by 12 points," Sestak said. "I'm tied with Pat Toomey."
Sestak said that voters were upset by Specter's decision to switch parties and were eager to support a candidate like him who has been a strong Democrat for years.
Crowley asked him if one of the reasons he was getting strong support was due to the fact that he was not the candidate supported by the Democratic establishment. She asked if maybe some supporters on the left were turning to him to send a message to the White House.
"This is absolutely not about President Obama," Sestak said. He said the race came down to Democrats supporting a Democrat, plain and simple.
But Sestak wouldn't say if he would support his party's nominee to beat Toomey should it be Specter.
"[I] never deal with something that's not going to happen," Sestak told Crowley. "Because we're going to win."
Specter was more open to standing with Sestak should things not go his way May 18.
"I'm going to support anybody against Pat Toomey," Specter said.