After providing facilities for Catholic worship in his neighbourhood, he built a church on his estate, which, by agreement with the Bishop of Newport and the superiors of the English Benedictine Congregation, became the pro-cathedral of the diocese. On the adjoining land given by him, a monastery was built, to serve as the novitiate and house of studies of the congregation. This became Belmont Abbey. Wegg-Prosser was also identified with several Catholic interests.
For many years he was a member of the Superior Council of the Society of St Vincent de Paul, a member of the Catholic Union, and a representative of the Diocese of Newport on the Catholic Education Council. In his secular life he was devoted to mathematical science, and particularly to astronomy. He wrote a book, Galileo and his Judges (London, 1889), on the question of Galileo, and translated, under the title Rome and her captors (London, 1875), the letters collected by Count Henri d'Ideville upon the Roman question of 1867–70. He married Lady Harriet Catherine, daughter of the second Earl Somers; she died in 1893, leaving two sons and two daughters.
He died near Hereford, England, in 1911, aged 87.