The town's first written mention dates to 1455. The construction of Byržai Castle began in 1586, and the town was granted Magdeburg Rights in 1589. In 1575, as preparation for the castle's construction, a dam was built on the Agluona and Apašcia Rivers at their confluence, and the artificial Lake Širvena, covering about 40 km² (15 sq mi), was created. It is the oldest surviving artificial lake in Lithuania. The town's history is closely associated with the Radziwill family (Lithuanian: Radvila). Jerzy Radziwill was the first noble to settle in the town. Later, after his daughter, Barbara Radziwill married the Grand Duke of Lithuania and King of Poland Sigismund II Augustus in 1547, the power and influence of the family grew immensely. The Radvila family established a Protestant church and school, and the town became a cultural center of the Protestant Reformation in Lithuania.

The local community of Lithuanian Jews, which settled in the Duchy of Byržai at the end of the 16th century, was influential, establishing an interest-free loan society, two major flour mills, and an international linen export business. The Islamic Lipka Tatars performed military, police, and postal duties for the Radziwill family.

During the Wars with Sweden, Byržai Castle was an important point of defence. In 1625, Gustavus Adolphus, king of Sweden, attacked the castle with 8,000 soldiers and it was forced to surrender. The castle was left in ruins and was rebuilt, only to be burnt in 1655. In 1662-1669, it was rebuilt again in Renaissance style. On 9 March 1701, August II the Strong and Peter I of Russia (Peter the Great) signed a pact in the castle to unite their forces against Sweden. However, in 1704 the castle was completely destroyed and was left in ruins until its restoration in the 1990s.

The town's population suffered greatly due to wars and religious conflicts between the Protestants and Catholics. In the late 18th century, Byržai lost its town rights. The Radziwills lost their wealth and influence, and Byržai was sold to the Tyszkiewicz family to cover debts in 1811. In 1849-1862, the Tyszkiewicz family built a neoclassic Astravas Manor palace across the lake from the site of the original castle.

In 1869 the town had about 2,600 residents. Thirty years later the population had grown to 4,400.

During World War II, the entire Jewish population of Byržai was annihilated. 15 Jews were shot to death by German soldiers at the Byržai Jewish cemetery in July 1941. On 8 August 1941, Gestapo and Lithuanian collaborators murdered the entire Jewish population of the town, some 2,400 people, by shooting them to death at a mass grave in a forest grove 3 kilometres (1.9 miles) outside the town. The town was almost completely burned down during the war. Oldtown was destroyed.

In 1968 the population reached 10,000. Currently there are 15,000 people living in Byržai.