I suspect the latter category is quite large: not personally bothered by it, but who feel that it should be moved to spare the feelings of others. Although polls and pundits lump this group in with all the rest who "oppose the mosque", it would be nice to see this distinction made and explored, since those who are against it for the sake of the feelings of others aren't, in an important sense, really against the "mosque" at all. They do support (voluntarily) moving it, yes, but if those who were directly offended ceased to be, these centrists would also switch sides. In that sense, they shouldn't really be counted as part of the majority who oppose the building -- and without them, I doubt it would be a majority.
I think I half agree with TB. I think they are part of the majority who oppose the building. I don't see how you get around that. But I do think it's important to note and contextualize the different sorts of opposition.
In other respects, the nature of the question (Yes! or No! in old-fashioned Crossfire or McLaughlin Group fashion) is uncomfortable and distorting. Do I support the Park51 project or oppose it? Frankly, I'd like to be able to have no opinion at all and just let it go ahead like a million others do without any input from me at all. I'd prefer to be indifferent to the project and just be among those who support the right (which one would hope would go without saying) to build it. But at some point in this madness that position simply became untenable. In a climate like this really supporting the right to build it means supporting it. So I'm for it.