"New Hampshire people want New Hampshire people," said Kim Pratt, a 52-year-old self-described independent voter, sitting at the Red Arrow Diner's breakfast counter as Brown shook hands nearby during a weekend visit. "He's not really a New Hampshire person. He's a politician from Massachusetts."
Outside after a breakfast of corned beef hash and eggs, Brown acknowledged the challenge.
"Do I have the best credentials? Probably not. 'Cause, you know, whatever. But I have long and strong ties to this state," he told The Associated Press. "People know." Brown spent the first year and a half of his life living in New Hampshire before his family moved to Massachusetts.
Brown's residency already plays prominently in his quest to defeat Democratic incumbent Sen. Jeanne Shaheen this fall. The stakes are high in New Hampshire and Washington, D.C., where Republicans are competing to gain six seats they need to win the Senate majority and transform the last two years of President Barack Obama's presidency.
Shaheen had been expected to cruise to reelection until Brown stepped into the race this month, giving the GOP a high-profile challenger with national fundraising appeal and a moderate political philosophy expected to play well among local voters.
Brown became a New Hampshire registered voter 13 weeks ago, according to the Rye, N.H., town clerk.
He and his wife moved to their 1,700-square foot seacoast New Hampshire vacation home in late December. The recent move was common knowledge inside the diners he visited over the weekend as part of a "Main Streets and Living Rooms" listening tour he launched about a week ago.
Inside the Red Arrow, Brown claimed a stool at the counter next to Pratt. As he waited for his breakfast, Pratt vowed not to vote for Shaheen. But she also pointedly questioned Brown's devotion to New Hampshire. Behind him, 71-year-old Manchester resident Connie Antoniou whispered, "I wish the Massachusetts people would stay in Massachusetts."
Brown told Pratt that "carpetbagger is a derogatory term" in New Hampshire given that roughly 60 percent of its people were born elsewhere, including the current and former Democratic governors. Gov. Maggie Hassan moved to the state in 1989. Shaheen, who was born in Missouri, has lived in New Hampshire for more than 40 years.
"Sen. Shaheen is not from here, but apparently it's a problem with me?" Brown asked during a brief interview outside the diner.
Brown's parents were stationed at New Hampshire's Pease Air Force Base before he was born. He was delivered at the Portsmouth Naval Shipyard and moved as a toddler to Massachusetts, where he spent most of his life. His mother and sister live in New Hampshire. While growing up, Brown often visited his grandparents on the New Hampshire seacoast, where he bought a modest vacation home in 1993.
New Hampshire and Massachusetts have a complicated relationship.
They share a state line, professional sports teams and major media market, but there are traces of resentment among some New Hampshire natives. Thousands of Massachusetts residents moved into southern New Hampshire in recent years, drawn by lower taxes and cheaper real estate. The migration helped give Democrats a slight voter registration advantage, although the state is considered far more balanced politically than solidly Democratic Massachusetts.
One of the original tea party favorites, Brown shocked the nation by winning the special election to replace Massachusetts Sen. Ted Kennedy in 2010. He was soundly defeated by Democratic Sen. Elizabeth Warren in 2012.
Moving to New Hampshire, Brown faces daunting historical challenges.
Just two people have served multiple states in the U.S. Senate. The most recent was elected in 1879, according to Betty Koed, an associate historian for the U.S. Senate.
Former Sen. James Shields actually served three states in the middle of the 19th century: Illinois, Minnesota and Missouri. Sen. Waitman Willey served Virginia and then West Virginia once it became a state during the Civil War.
"There was a lot of mobility in the wake of the Civil War," Koed said. "It's a very different electoral environment than it was in the 19th century."
Brown's camp privately acknowledges that residency will be a factor for some voters, but expects concerns to dissipate after he spends more time in the state. If nothing else, Brown is known as a tireless campaigner who connects well with voters. He plans to add thousands more miles campaigning across the state to the 285,000 already on his GMC Canyon.
With a film crew in tow to gather footage for campaign ads, Brown made eight stops on Saturday alone.
He made his first visit to the Tilt'n Diner, which was crowded with Massachusetts tourists passing through and recent transplants.
Brown chuckled when 39-year-old Christine Kalinowski told him she was from "Southie," or South Boston. She said she moved to New Hampshire just a few months ago, like him.
"I voted for him before," she said, "and I'd definitely vote for him again."
Copyright 2014 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.