The son of a Hampshire attorney, Mansfield's private life was kept somewhat guarded. His father John James changed the family name from Manfield to Mansfield. James married Grace in 1765 at St Margaret's Westminster, fathered six children, but also had another five children by another partner. One of these five was John Mansfield of Diggeswell, father of General William Mansfield, 1st Baron Sandhurst. Mansfield attended Eton from 1745 until 1750, and then King's College, Cambridge, of which he was elected a fellow in 1754. He graduated with a BA in 1755 and an MA in 1758. Mansfield pursued a career in law, obtaining admission to the Middle Temple on 11 February 1755 and being called to the bar on 28 November 1758. His career, both at common law and in chancery, was quite successful, and he was appointed one of the counsel for John Wilkes in 1768. Mansfield was one of James Somersett's lawyers; Somersett was a slave brought by his master from Jamaica to London in 1769, and freed on 22 June 1772 by a ruling from Lord Mansfield. Mansfield was made king's counsel on 24 July 1772, and a bencher of the Middle Temple shortly after, on 6 November 1772. He was involved in the trials, in 1776, of the Duchess of Kingston for bigamy, and of Smith, Hollis, Calthorpe, and Beckford for bribery. In 1777, he was defence counsel for John the Painter, and crown prosecutor in 1779 against the counsellors who arrested Lord Pigot and took over Fort St George. His ability was admired by the North Ministry, and he was offered a seat at Morpeth in 1776 by the Earl of Carlisle, but declined lest it interfere with his prospects for a judgeship. He was considered, but ultimately passed over, by North to fill the post of Attorney General or Solicitor General.
While in Parliament, in 1782, Mansfield had been elected Reader of the Middle Temple, and was Treasurer in 1785. He was counsel in a number of high-profile cases, including the Thellusson Will Case, in which he and Samuel Romilly represented the plaintiffs. He was appointed Chief Justice of Chester in July 1799 before becoming Chief Justice of the Common Pleas on 24 April 1804, becoming a serjeant-at-law and receiving a knighthood. As a judge, he was principally known for his easily provoked temper, but his knowledge of the law was thought considerable. He refused an offer of the Lord Chancellorship in 1806. As chief justice, he presided over the trial of John Bellingham, assassin of Spencer Perceval, in 1812. Mansfield resigned on 21 February 1814, due to ill health and died at his house in London on 23 May 1821.