The building to be seen nowadays is form the 13th to 18th century, it used to be one of the residences of the Hohenzollern dynasty as burgraves of Nuremberg. Frederick I. of Brandenburg rebuilt it entirely, and later died there in 1440. For the church dedicated to St. Cecilia he donated the altar (1420/25), on which he is depicted together with his wife Elisabeth of Bavaria-Landshut in the middle part, right under the crucifixion scene. The side parts show St. Cecilia together with their groom, St. Valerian. The altar was gifted to Frederick III. on his request and moved to Berlin. The original is today in the Jagdschloss Grunewald, but the church displays a copy. The Castle was not damaged during the Thirty Years' War.
1933-1945 a part of the castle, the so-called Neues Schloss (New Palace) was used as a Gebietsführerschule by the Hitler Youth. At the end of the war the castle burned down and was left a ruin for various decades. Since 1979 the Bavarian Administration of State-Owned Palaces, Gardens and Lakes together with the Bauamt Nuremberg-Erlangen is responsible for the reconstruction and maintenance of the castle. The reconstruction started in 1982 with the Castle itself and was finished in 2007 with the new creation of the gardens. For seven years the inner and outer parts of the castle were open to the public and events were held.
In 2013, it was decided to transform the Altes Schloss (Old Palace) into an experiential museum. In 2017 it was opened under the name "HerrschaftsZeiten. Erlebnis Cadolzburg".
South to the Main Castle exists an Antecastle (Vorburg) consisting of multiple buildings from the 17th and 18th century and a walled garden. The Wall Tower with the included clock is a gothic element from the 13th/14th century.