The site of 4 High Street was purchased by the Union Bank of Australia in 1881 and for several years the bank operated from the existing building, which had been the residence of Captain Daniel Scott. A new building was erected in 1889 with plans prepared by Melbourne architect William Edward Robertson, the construction was supervised by James Wright.

The bank built new premises further along High Street in 1930 and placed the existing building up for auction. It was advertised as having a 119-foot (36 m) frontage along High Street and a 64.5-foot (20 m) frontage along Cliff Street. The ground floor of the brick building had a large banking chamber, a manager's office, strongroom, entrance hall, dining room and kitchen. The first floor contained a drawing room, seven bedrooms, bathrooms and linen closet. There were balconies at both the front and rear of the building.^[1]

In 1931 the property was purchased by the Church of England for the Flying Angel Mission to Seamen organisation. In December 1937 plans were approved for the construction of the St Andrew's Mariners Chapel to the west (along High Street) behind the former bank building. It was constructed by Hawkins & Son at a cost of £1,731 and continued to function as a chapel until the late 1960s.

As of 2016^[update] both buildings were owned and used by the University of Notre Dame Australia. The Flying Angel Club had moved to 76 Queen Victoria Street by 1968.^[2] It is within the Fremantle West End Heritage area.