The site consists of an approximately circular enclosure with an internal diameter of 68 metres (223 ft), surrounded by a bank 12 metres (39 ft) wide and 1.7 metres (5.6 ft) high.[2] Surrounding the bank is an outer ditch, partially filed in, which is 5.4 metres (18 ft) wide and 0.6 metres (2.0 ft) deep.[2] The interior of the site is around 0.4 hectares in area and generally level, with traces of a platform on the west side and a circular depression in the southwest corner.[3] In the south is an entrance to the enclosure with a causeway across the ditch.[2] Outside the bank to the south and west is an outwork, made up of a second bank and ditch up to 65 metres (213 ft) long, with an entrance and causeway across the ditch in line with the main entrance.[2] A modern fence runs north to south across the site just west of the entrance.[3]
Voley Castle is a type of hill fort known as a slight univallate hillfort, which are defined as enclosures between 1 and 10 hectares in size situated on or close to hilltops with a single line of relatively small earthworks.[2] These date from the late Bronze Age and early Iron Age, and are numerically and geographically rare, being mainly confined to Devon.[2] Voley Castle is unusual for this type of fort because it has an outwork.[2]
Voley Castle is located on a steep east facing slope overlooking the valley of the River Heddon, with uninterrupted views down the valley and north towards the sea.[3] Its location is unusual for a hill fort, because it is on a level shelf around 45 metres (148 ft) below the steeply rising crest of Heale Down hill, meaning that it would be very difficult to defend.[3] Rather than being a fort, it has been suggested that the site likely represents a defended farmstead.[3] The enclosure may have been used for holding cattle or possibly growing crops, and the outwork has been suggested to be a cattle enclosure.[3] There are eight similar sites in Exmoor, but Voley Castle is the only one in an indefensible position.[3] It is very similar to the site at Sweetworthy, in the eastern part of Exmoor in Somerset, in terms of its location, size and appearance.[3]