Although now a very attractive village, Castle Acre itself was once a fortified town, it was protected by its own large bank, with a ditch and gateways to guard the entrances. The surviving so-called Bailey Gate was once the North gateway to the town, and the Medieval Parish Church of St James, which lies near to the Priory is situated outside of the town's defences, which shows the town had outgrown its defences and original boundaries by the 13th Century. When first established, Castle Acre was one of the finest examples of Norman town planning in the country, and much of this can still be seen.