It was a Roman historian and writer, St. Augustine, who coined the phrase "City of God" in a book of the very same title. But the city of God that Augustine was referring to was not in Jerusalem. When Augustine wrote his book, it was hundreds of years after the Jewish city of Jerusalem had been overrun and destroyed by the Roman legions; he was writing of Rome as the city of God and the center of the Roman Catholic faith.
In St. Augustine's day, some 400 years after the Romans had laid waste to Jerusalem, the very empire that had destroyed it was completely transformed by it. Just 400 years after the Romans had crushed Jerusalem their official religion was Christianity, a religion which, of course, has its roots in Jerusalem.
So, flash forward 400 years after Rome had laid waste to Jerusalem: it was now Rome that was threatened with ruin from invading Visigoth armies. It was the destruction of Rome, not Jerusalem, that inspired Augustine's City of God. The Romans had been completely demoralized by the sacking of their city, and many had started to blame their Christian faith itself for the city's downfall.
They started to believe that Rome was being punished for abandoning their traditional pagan Gods for Christianity. St. Augustine wrote City of God as a response to this widespread fear and disillusionment with the faith. In a great attempt to console the souls of the demoralized citizens of Rome, Augustine was the first theologian to argue that there was, in fact, no physical city of God.
In his way of thinking, neither Rome nor Jerusalem were cities of God. Augustine stated that the true city of God did not have a zip code or earthly address, but rather resided in heaven itself. And, no matter what happened to Rome, Christians should not be worried, because the true city of God, the "New Jerusalem", was still alive and well and could never be sacked by Visigoths or anyone else.