All Souls was built at the heart of the University of Oxford, on the corner of High Street and Catte Street. It stands next to the 'University Church' of St Mary the Virgin, where the University's self-governing 'Convocation' once met. The foundations were laid in 1438, and the College's High Street front today gives a good idea of how the main part of the original College looked. The four-storey gate tower and the two-storey ranges either side of it remain essentially as built in the 1440s, although the range to the east was extended in the sixteenth century, when battlements were added all the way along and the windows are also mostly nineteenth-century.

The Warden of the College had lived in a house at the east end of the High Street front from the middle of the sixteenth century. In 1703, however, Dr George Clarke, a fellow in public service with strong architectural interests, proposed that he should build a house for himself on College ground at his own expense, and that it would revert to the College on his death. The house he built, which became the Warden's lodgings, originally boasted battlements to harmonise with the rest of the High Street front. The more classical, Palladian facade to be seen now dates from the 1820s.