He said the mayor needs to say what he plans to do himself because "when you go left, he goes right."
Ford has apologized for his bad behavior but his refusal to resign or take a leave of absence has frustrated both his opponents and allies on Toronto's City Council, which has no legal way to force him out unless he is convicted of a crime.
The mayor was mobbed by reporters at City Hall on Friday but refused to discuss what his next steps would be.
"I'm dealing with a serious personal issue right now. Please give me this time," he said before walking into an elevator.
In another sign of Ford's growing isolation, the mayor's radio show was taken off the air. NewsTalk 1010 announced Friday that Ford and his brother "have mutually determined to conclude broadcasts of The City, ending with last week's show."
Ford's influential brother, meanwhile, said he has urged the mayor "to go away for a couple of weeks," although he reinforced the family's stance that Ford has no reason to resign.
"I've mentioned to Rob, maybe go away for a week, a couple of weeks, and get your mind together," Doug Ford told a Toronto radio station on Friday.
Doug Ford, a city councilor, said his brother needs a vacation and needs to work on his weight. He also suggested that despite the mounting scandal, Rob Ford remains a potent political force ahead of next year's mayoral elections.
"If Rob goes away on a little vacation, a week, two weeks, comes back, loses 50 or 60 pounds, and stays on the straight and narrow ... it will be very tough to beat Rob Ford," Doug Ford said.
The mayor, a conservative who was elected three years ago on promises to "stop the gravy train at City Hall," has vowed to seek re-election. He still has a loyal following in Toronto's more conservative outer suburbs, who praise him for cutting the size of the city work force and taking on labor unions, including banning public transit employees from going on strike by having them classified as essential employees.
But Ford's three-year tenure has been consumed by mounting scandals ranging from the petty -- making rude gestures to Torontonians from his car -- to the more serious -- being fired from his cherished side job as a volunteer high school football coach after making disparaging remarks about parents and their kids.
The comments from Morris and the mayor's brother came a day after a video surfaced showing Ford threatening to kill an unknown person. Morris said Thursday was a defining day for the mayor of Canada's largest city.
The shaky, blurry footage, which appeared in the Toronto Star website, showed the mayor pacing around in a room, waving his arms rolling up his sleeves and he unleashes an expletive-laced tirade.
"No holds barred, brother. He dies or I die," the mayor tells someone else in the room, possibly the person taking the video.
The context of the video is unknown, and it's unclear who the target of Ford's wrath is. The Toronto Star said that it bought the video for $5,000 from "a source who filmed it from someone else's computer."
The mayor said he was "extremely, extremely inebriated" in the video and "embarrassed."
The video surfaced just two day after the mayor admitted to having smoked crack while in a "drunken stupor" about a year ago. The stunning confession came a week after police said last week they had a video of that appears to show the mayor smoking crack. News reports of the crack video's existence first surfaced in May, but it has not been released publicly.
Doug Ford insisted that the video of the mayor threatening to kill someone was an "isolated incident." His mother and sister also downplayed the mayor's troubles, insisting in a television interview Thursday that Rob Ford was not an addict of any sort.
His mother, Diane Ford, said she has advised her son during a family meeting last week work on his "huge weight problem," curb his drinking watch the company he keeps. But she insisted that her son did not need rehab.
But Toronto's city councilors are working on efforts to drive Ford from office, or at least curb his powers.
One measure, which could be voted on next week, would takes the unprecedented step of asking the province of Ontario to pass legislation to remove the mayor if he does not agree to take a leave of absence. The provincial government, however, has expressed reluctance to interfere in Toronto's municipal affairs.
Follow Rob Gillies on Twitter at -- http://twitter.com/rgilliescanada
Copyright 2013 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.