But he accused the Democrats of turning the comment into campaign trail fodder.
"I'm not going to play the political games because I think my constituents will judge based on my actions as opposed to press releases," he told the paper.
As the Daily Herald points out, Duffy's town hall appearances are part of the freshman Rep.'s attempt to "fulfill his campaign pledge to hold annual meetings in each of the 20 counties he represents."
But "ever-present camera and microphone trained on Duffy and other elected officials has added a new pitfall" for the former reality TV star, the paper reports. Nevertheless, Duffy plans on continuing to get real with the voters in public settings.
"I think people appreciate a face-to-face conversation," he told the paper, "and I'm not going to be scared away from speaking with people who want to have access to their congressman."