After a three-hour investigation by police and firefighters in protective clothing, the terminal was declared safe. Police were not treating the incident as terrorist-related, and said they were investigating whether a canister of tear gas discarded by a passenger was the cause.
Police and the fire brigade said they were called just after 4 p.m. (1500GMT) Friday to reports of passengers at the airport feeling unwell.
With a fire alarm sounding, some 500 travelers and staff were evacuated to a parking lot and the tarmac near the airport runway.
David Morris, 28, said he was checking in for a flight to Edinburgh when he started coughing.
"It was getting quite bad and we saw other people starting to cough at the same time," he told Britain's Press Association news agency. "The people behind the desk were coughing the most and quite aggressively.
"Within two minutes, they shouted for everyone to get out," Morris said.
The London Ambulance Service said four ambulance crews and its hazardous area response team were sent to the scene. It said crews treated 27 patients at the airport for "minor breathing difficulties." Two of them were taken to hospitals.
Three hours after the evacuation, London Fire Brigade gave the all-clear.
"No elevated readings were found and the building was ventilated, searched and declared safe," it said in a statement.
The Metropolitan Police said a search uncovered a "CS gas spray," also known as tear gas, which causes stinging eyes and a burning throat.
Individual canisters of the spray are sometimes carried like pepper-spray for personal protection
"Whilst the cause of the incident has not yet been confirmed, officers are investigating whether it was the result of an accidental discharge of the spray" which may have been discarded by a passenger before check-in, the force said in a statement.
Departing flights were suspended for several hours, and a number of incoming flights from destinations including Frankfurt, Amsterdam, Belfast and Paris diverted to other airports.
The airport reopened Friday evening, but said disruption to flights would continue into the night.
City Airport, in east London's docklands, is the smallest of the British capital's international airports. It offers mostly short-haul flights to British and European destinations.
Copyright 2016 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.