Foster was the son of Alfred Dwight Foster and Lydia Stiles. His father was a representative on the Massachusetts General Court and was involved with various civic organizations including the Worcester town council, Massachusetts Governor's Council, Leicester Academy, Amherst College, the State Lunatic Asylum, and the State Reform School. Foster married Henrietta Perkins Baldwin , the daughter of Connecticut Governor & U.S. Senator Roger Sherman Baldwin. They had eight children: Alfred Dwight Foster, Burnside Foster, Emily B. Foster, Mary Rebecca Foster, Henrietta Baldwin Foster, Roger Sherman Baldwin Foster, Reginald Foster and Elizabeth Skinner Foster. Their son Roger Sherman Baldwin Foster was a prominent lawyer. His grandfather, Judge Dwight Foster, was a Representative and a Senator from Massachusetts; born in Brookfield, Worcester County, Mass., December 7, 1757; completed preparatory studies and graduated from Brown University, Providence, R.I., in 1774; studied law; admitted to the bar in 1778 and commenced practice in Providence, R.I.; justice of the peace for Worcester County 1781–1823; special justice of the court of common pleas 1792; sheriff of Worcester County 1792; member, State house of representatives 1791–1792; elected to the Third and to the three succeeding Congresses and served from March 4, 1793, to June 6, 1800, when he resigned; chairman, Committee on Claims ; delegate to the State constitutional convention in 1799; elected to the United States Senate as a Federalist to fill the vacancy caused by the resignation of Samuel Dexter and served from June 6, 1800, to March 2, 1803, when he resigned; chief justice of the court of common pleas 1801–1811; member, State house of representatives 1808–1809; member of the Governor’s council and held other state and local offices chief justice of Worcester County, Massachusetts, a member of the Massachusetts Constitutional Convention, the Massachusetts State Legislature, and the Massachusetts Executive Council, and was a U.S. senator serving as a Federalist from 1800 to 1803. His great uncle Theodore Foster was a Senator from Rhode Island; born in Brookfield, Worcester County, Mass., April 29, 1752; pursued classical studies and graduated from Rhode Island College, Providence, R.I., in 1770; studied law; admitted to the bar about 1771 and commenced practice in Providence, R.I.; town clerk of Providence 1775–1787; member, State house of representatives 1776–1782; appointed judge of the court of admiralty in May 1785; appointed Naval Officer of Customs for the district of Providence, R.I., 1790; appointed to the United States Senate in 1790; elected in 1791 and again in 1797 as a Federalist and served from June 7, 1790, to March 3, 1803; was not a candidate for reelection in 1802; retired from public life and engaged in writing and historical research; member, State house of representatives 1812–1816; trustee of Brown University 1794–1822.