On the hills above Škofja Loka stood in the 12th century three fortifications, the Upper Tower on Krancelj, the Lower Fort (nider vest), and the Loka Castle, a residence of Freising Bishops, built at the brink of a natural terrace. The castle is first mentioned in 1202 as castrum firmissimum, although today's building dates from the 16th century. The castle was rebuilt in 1691 after earthquake and in 1716.
The castle was administered by Loka chief, among them Lambergs, Thurms, and Rasps. Since 1890 the castle was managed by Ursulines, who pulled down the Romanesque yard tower and rearranged it in a school. Today it hosts the Loka Museum.
The circular castle dominates above Škofja Loka town and presents conclusion of town walls. It can be dated after the earthquake in 1511, although some parts are older, which was confirmed during the renovation works in 2006–2008. The north-eastern tower, the oldest in the structure, is built on older predecessor from the 10th century. Together with the renovation work also the reconstruction of the primary entrance with wooden drawbridge was carried out.