The Grade II listed building has featured in shows including Downton Abbey and Peaky Blinders