Sir Simon Pakeman, knight of the Shire at Parliament, owned this manor house in the 13th century.
The property changed hands many times and through marriage it eventually passed to William Hastings. In 1461 he was appointed Lord Chamberlain and Baron of Hastings.
A licence to crenelate was obtained c.1474 from his friend, King Edward IV. William employed the master mason John Cowper to alter the fabric of the old house to turn it into a fortified manor house. John was paid eight pence a day to oversee the stone work as opposed to the brick-maker in charge, Anthony Docheman, who was only paid ten pence per week.