Following the restoration of the monarchy he frequently ran into difficulty with the Crown. In 1676 he was imprisoned in the Tower of London and later fled the country when King James II came to the throne. He spent time while abroad in the Court of the Prince of Orange and subsequently his family line was back in Royal favour when the latter came to the throne of England in 1688. He was unfailing in his admiration for Cromwell, and his belief that Parliament had been in the right in the Civil War: in 1689 he angrily demanded that Henry Hyde, 2nd Earl of Clarendon, withdraw a reference to the Civil War as a rebellion. He had one surviving daughter, Elizabeth, by his first wife, Elizabeth Wandesford. By his second wife, Jane, only daughter of Colonel Arthur Goodwin and heiress to the extensive Goodwin estates in Buckinghamshire, he had seven additional children: Anne, Margaret, Thomas Wharton, 1st Marquess of Wharton, Mary, Goodwin, Philadelphia, and Henry. On 21 April 1658, his second wife Jane died and on 26 August 1661, he married for the third time. Lord Wharton married Anne Popham, the widow of Colonel Edward Popham. Anne brought two children from her marriage with Edward: Letitia aged 13 and Alexander aged 12. Alexander was a deaf mute and under the guidance of Dr. John Wallis in Oxford, was one of the first deaf people in the world to learn to speak. Lord Wharton and his third wife had a son named William, born around June/July 1662. William died on 14 December 1687, killed in a duel. Lord Wharton was a prominent art collector and patron. In the 1630s he commissioned a series of portraits painted by Anthony van Dyck of several members of his family, including himself, his wife Jane, his father-in-law Arthur Goodwin, and his daughters Philadelphia and Elizabeth. Lord Wharton gave much support to church ministers, particularly those who shared his perspectives. he also gave money to establish chapels at Ravenstonedale and Smarber and to provide for the ministers at both places. The latter survives as Low RowUnited Reformed Church. In his will he left land near York to support a Bible charity, which was devoted to the distribution of bibles to children for use outside of the church or school. The terms of the will require the recipient to learn by rote the 1st, 15th, 25th, 37th, 101st, 113th, 145th psalms. The will also requires the Shorter Catechism also be included. Many thousands of Bibles have been distributed and the Trust still distributes Bibles to under eighteen year-olds. http://www.lordwhartonbibles.org.uk/