In the Middle Ages, there was a ford across the Morava and the crossroads of the Amber and Salt Roads (which was the actual continuation of the famous Silk Road). The first written record of Kromeriz dates back to 1110. On the Arab map, known as Tabula Rogeriana, the town is referred to as Agra.

The settlement, inhabited by Slavs since at least the 7th century, was founded in 1260 by Bruno von Schauenburg, bishop of Olomouc. Bruno chose Kromeriz to become his see and he also made his castle the centre of his dominion, which consisted of more than 60 vassals from all over Moravia. Kromeriz is referred to as a market village in a document by Ottokar II of Bohemia from 1256, but by 1266 it was already referred to as a town. Bruno established what was to become the famous Archbishop's Palace. The town was badly damaged in the Thirty Years' War, was plundered twice by Swedish troops (1643 and 1645), and after this the Black Death took its toll on the population. Bishop Karl II von Liechtenstein-Kastelkorn rebuilt the town and the palace after that war. The Constitutive Imperial Congress sat in Kromeriz in 1848. In August 1885 a meeting took place here between the Austrian and the Russian emperors.