The Brown campaign has now provided us with the audio of the exchange the report was based on. And as is often the case, there's a lot of crosstalk and incomplete sentences. But I think the clear reading of Brown's statement is not that he's "unfamiliar" with the Tea Party movement or hasn't heard of it but that he didn't accept or wasn't buying the reporter's characterization of it.
The audio is clear that Brown did not want to get into a discussion of Tea Partiers or his relationship or connection to the movement. But he wasn't saying he'd never heard of or wasn't familiar with the Tea Party movement.