Photos show the swastika and words were spray-painted on playground equipment in the Brooklyn Heights neighborhood.
The city named the park for the late rapper, who was Jewish, in 2013. He died of cancer in 2012 at the age of 47.
New York City Councilman Brad Lander said police were investigating and the Parks Department planned to remove the messages.
The graffiti was one of several recent instances of the Nazi symbol being found around New York and in other parts of the country.
The playground includes basketball courts, a community garden, a greenhouse, a small fitness area, an open play space, drinking fountains and a dog run.
Copyright 2016 The Associated Press. All rights reserved. This material may not be published, broadcast, rewritten or redistributed.