"I've known him a long time," Hagel said in an interview with CBS News.
Hagel said more facts need to be learned and that it would be premature to oust Shinseki, a retired Army general.
"There does have to be accountability, right up and down the line," said Hagel, himself a military veteran. He said the government has "no higher responsibility" than to provide top-shelf medical care to servicemen and women who have worn the country's uniform.
"We know things went wrong," the secretary said. "Somebody's got to be accountable here, like in any institution."
President Barack Obama earlier this week told a White House news conference that allegations of misconduct at VA hospitals will not be tolerated, and he left open the possibility that Shinseki could be held to account.
"I will not stand for it — not as commander in chief but also not as an American," Obama said following an Oval Office meeting with Shinseki.
The growing controversy surrounding the Department of Veterans Affairs centers on allegations of treatment delays and preventable deaths at VA hospitals. The department's inspect general's office says 26 facilities are being investigated nationwide, including a Phoenix hospital facing allegations that 40 people died while waiting for treatment and staff kept a secret list of patients in order to hide delays in care.
Copyright 2014 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.