Carson, the retired neurosurgeon and Fox News regular, joined Hayes, Demos President Heather McGhee and Telemundo's Jose Diaz-Balart for an amicable chat about race in America.
Hayes questioned Carson about the notion that racial identities are "contrived by liberals to divide people."
Carson, whose book "One Nation: What We Can All Do to Save America's Future" was released this week, said he isn't sure that "it's just liberals who contrive them." But Carson said his experience as a neurosurgeon confirmed his belief that someone's skin color has "very little with who that person is."
"And it's something that we have just allowed to define us and it really has very little to do with who a person is," Carson said.
McGhee took Carson's point, but gently pushed back against the "myth that we can be color blind." After Carson said he's mostly interested in one's personality, Hayes reminded him that many aren't so tolerant.
"That is true of Dr. Carson, it is not true necessarily of, say, the predatory contract buyers in the south side of Chicago who spent 20 years targeting African-Americans for their home equity and ripping them off as an explicitly racialized project of exploitation and predation," Hayes said.
"It does exist. I'm not saying it doesn't exist," Carson said. "What I'm saying is we need to get to the point where we think more deeply."
Last night's civility notwithstanding, Carson has a history of divisive comments. In January, he said progressives could transform the United States into a country that resembles Nazi Germany. He made yet another Nazi reference in March, declaring that the country is currently mired in a "Gestapo age" due to political correctness.