The son of peasants, Rappe was born in Audrehem, Pas-de-Calais, to Eloi and Marie Antoinette Rappe. He was one of ten children and labored in the fields until October 1820, when he entered the College of Boulogne. After graduating in 1826, Rappe entered the seminary of Arras and was later ordained to the priesthood by Bishop Hugues de la Tour d'Auvergne-Lauragais on March 14, 1829. He then served as pastor of Wismes until 1834, when he became chaplain to the Ursulinemonastery in Boulogne.
On April 23, 1847, Rappe was appointed the first bishop of the newly created Diocese of Cleveland by Pope Pius IX. He received his episcopal consecration on the following October 10 from Bishop Purcell, with Bishop Richard Vincent Whelan serving as a co-consecrator, at Cincinnati. Two days after his consecration, he published his first pastoral letter, in which he expressed his desire "to be regarded as your friend and father, rather than your superior." At that time, the diocese contained 42 churches and 21 priests; the first and only Catholic church inCleveland was St. Mary's on the Flats. He soon established the city's first parochial school, which doubled as a chapel. Rappe purchased an episcopal residence in 1848, and also laid the cornerstone of the new St. John's Cathedral on October 22 of that year. He founded a seminary at his residence that year as well. In 1849 he went to Europe to recruit clergy for the diocese, returning in 1850 with four priests, five seminarians, two Sisters of Charity and six Ursuline nuns. The Daughters of the Immaculate Heart of Mary opened St. Mary's Orphan Asylum for Females in 1851. The Sisters of Charity opened St. Vincent's Asylum for Boys in 1852. Rappe consecrated St. John's Cathedral on November 7, 1852, and, a hospital being badly needed due to the Civil War, established St. Vincent Charity Hospital in 1865. He also introduced into the diocese the Grey Nuns, the Good Shepherd Sisters, the Little Sisters of the Poor, the Friars Minor and the Jesuits, and organized the Sisters of Charity of St. Augustine as a new congregation. Rappe eventually met strong opposition. His eyesight began to fail, and upon returning from the First Vatican Council he resigned as bishop on August 22, 1870. He left the diocese with more than 100,000 Catholics, 107 priests, 160 churches, and 90 schools. He spent the next seven years at St. Albans, Vermont, attending to the missions in Vermont and Canada. He was later offered another diocese, but declined. He died at St. Albans on September 8, 1877, at the age of 76 and was buried in St. John's Cathedral in Cleveland.