At the 1768 general election Spencer was returned as Member of Parliament for New Woodstock on the Marlborough interest. He was appointed a Lord of Trade in April 1770 but did not attend the Board regularly nor parliament itself. He resigned his seat in January 1771 and was elected MP for Oxford in a by election on 31 January 1771. He was returned for Oxford in 1774 and 1780. He was an ardent supporter of Fox throughout his career and after. In 1784 he was returned again for Oxford. Spencer was a member of a subscription committee set up to raise funds to support the Whigs in the forthcoming general election. He remained active in fund-raising appeals for the party for many years afterwards. In consequence the Duke of Marlborough replaced him as candidate for Oxford in 1790. Fox may have intended a seat for Winchelsea for him, but nothing transpired. He did not want to spent £3,000 to buy a seat at Wootton Bassett or risk an expensive contest at Evesham,and did not want to be a liability on his party. He maintained a vain hope that the family interest would be available at Oxford, but it was not. In the 1790 general election he was elected MP for Wareham probably at the party's expense. In the 1796 general election the Whigs appear to have paid £3,000 his return again at Wareham. However, he was in financial difficulties in 1799 and lost through gambling again so heavily that he had to sell his London house and pictures. He resigned his seat in February 1799. In March 1802 a vacancy arose for Spencer at Tavistock and he was elected at a by-election on 25 March 1802. In August 1802 he went to Paris with Fox, and met Napoleon with whom he discussed his great ancestor. Later when Napoleon was at St Helena Spencer sent him a biography of Marlborough. He was returned for Tavistock in the 1802 and 1806 general elections. He was Surveyor General of Woods, Forests, Parks, and Chases from February 1806 to March 1807. When Fox died his political raison d’être was gone and he gave up his seat at 1807. He never ceased to lament and praise Fox. He resumed his seat for Tavistock in 1817 as a stopgap for the Duke of Bedford when Lord John Russell vacated the seat through illness and was returned in a by-election on 12 March 1817. Similarly he stood in for his own family at Woodstock in the 1818 general election although his nephew the Duke of Marlborough withdrew him from Woodstock in 1820.
Personal life
Spencer was an inveterate gambler and having lost his official salary found himself in financial difficulties around 1781, until he was admitted to a twelfth share in Fox’s faro bank at Brooks's, with a fee of five or six guineas an hour. The circle surrounding Fox included Edward Bouverie, a fellow habitué of Brook’s, and his younger wife Harriet, the daughter of Sir Everard Fawkener, K.B. She was a society hostess and actively campaigned for the Whigs. She became Spencer’s mistress and the youngest Bouverie child, Diana, born in 1786 was acknowledged as a Bouverie but was referred to as “the tell-tale Bouverie, for there never was such a perfect indisputable Spencer, Lord Robert’s walking picture and the very prettiest creature that ever was seen”. By 1791 his profits from the faro bank were sufficient to allow him to purchase Woolbeding House in Sussex. Spencer married his mistress Harriet Bouverie on 2 October 1811, a year after the death of her husband. Spencer retired to Woolbeding which he had tastefully embellished. He died on 23 June 1831, “a venerable grave old man”. He left Wooldeding to his natural daughter Diana Bouverie whose mother had predeceased him. She married George Ponsonby.
