Originally a motte & bailey structure started in 1093. A stone shell keep was built at the end of the 12th century, and the stone bailey wall in 1233 after the town had been destroyed by the Welsh. The hall block which is visible from the river was added in 1300. In 1645 the castle was captured by Parliament after a seige and the walls were then slighted. Only a part of the shell keep remains.
Private, the hall block is now part of Brecon Hotel and the motte is in the garden of a private residence.
There are falconry exhibitions for small groups at the castle. Training and organised hunting days.