Monday's vote was non-binding, but 157 residents voted in favor of keeping the seal compared to the 55 who voted against keeping it, according to the Associated Press.
One voter, Scott Hastings, told local TV station WKTV that he just wanted the matter "settled."
"Political correctness, who cares? This is our village, who cares what the world thinks? I want to see this settled today," Hastings told the NBC affiliate. "Once and for all."
The seal was developed in the early 1900s to illustrate a "friendly wrestling match between village founder Hugh White and an Oneida Indian," according to the Associated Press. White's victory won the "lasting respect" of the tribe, the AP reported.
This isn't the first time the seal has come under fire. In 1977, according to the village's website, a claim was filed against the seal saying it "demeans, disgraces and creates prejudice and distrust of Indian people." The seal was redesigned to show the village founder's hands on the Native American's shoulders instead of his neck.
The AP reported that the results, while "informal," would be discussed at a meeting on Tuesday night.
The U.S. Census Bureau data shows 99.5 percent of the residents of Whitesboro identified themselves as white, the New York Daily News reported.