It is important to note that people don't always answer polls honestly or completely when asked their motivations. Indeed, people do not always know their own motivations. This is key to understanding all polling. You find yourself watching fewer games and when asked you point to your upsetness about players not standing for the national anthem. That said, I think it would be wrong to dismiss these numbers out of hand - especially when we look at the demographic breakdown of the numbers.
YouGov points out that this 40% who point to anthem protests constitutes only 12% of fans surveyed. So it's a relatively small number of fans. But who are they? They're disproportionately white and over the age of 55. In other words, they correlate highly with Trump supporters.
I don't see why this should surprise us. As race has come more to the surface of national politics, it has come into collision with a perennial fact (going back decades) of American professional sports: massive overrepresentation of black players watched overwhelmingly by white fans.
Again, I don't think what people say about their changing viewership patterns are necessarily accurate. Others have suggested that the biggest driver is just a run of not very interesting match-ups in nationally televised games. That may be the case. But even saying it is significant. Let's look at some numbers: 70% of NFL players are black in a country where about 13.3% are black. You could even make these numbers sound a bit more stark by noting that only about 6.6% of the population is made up of black men. What's more, in the nature of things, players are very young - the great majority in their 20s, especially in football where most careers are short. But its fan base is whiter than the general population (though not dramatically so) and leans older than most other sports. According to a widely cited study, 37% of NFL fans are older than 55 compared to 25% of NBA fans, 27% of MLS fans and 50% of MLB fans.
To be clear, it's not like football is a white sport in terms of its fan base. Far from it, the study I'm referring to finds that African-Americans are slightly over-represented among NFL fans. The drop off is with Hispanics who are underrepresented. The point is that there are a lot of NFL fans who are the older non-college-educated white men who make up the core of the Trump movement. It is hardly surprising that these fans would come into conflict with the political activism of the millennial black men who make up the overwhelming number of NFL players.