Two tombstones were defaced in the military section of the Mount of Olives cemetery, the Israel Police said Tuesday.
An investigation was opened into the incident.
Police received a report in the morning that hate slogans, including a reference in Hebrew to Nazis, were scrawled on the stones, according to a statement.
Officers who arrived at the scene found that a marker had been used to scrawl the graffiti on a sign at the site as well as on the two tombstones.
A photo showed what appeared to be the outline of a Palestinian flag scrawled on the sign.
Police notified the IDF of the incident so that the graffiti could be cleaned off.