Smith was elected for the first time – as Scotland's youngest MEP – in the 2004 European Parliament election. He was re-elected three times at the 2009, 2014 and 2019 European Parliament elections. He sat as a member of the currently seven-strong European Free Alliance Group in the Parliament, which retains its own identity within the joint Green-European Free Alliance Group. He is a member of the SNP's National Executive Committee. In his first two terms in the European Parliament, Smith served as a full member of the Committee on Agriculture and Rural Development winning the coveted Scottish Farmer Magazine award for "Outstanding Contribution to Scottish Agriculture" at the Highland Show in 2009. After his re-election in 2014, Smith served as a full member of the Foreign Affairs Committee, remaining a voice in agriculture as alternate member of the Agriculture and Rural Development Committee. He was also a full member of both the Delegation for relations with the Arabian Peninsula, and the Delegation for relations with Iraq. On 27 March 2019, in a brief speech to the European parliament, Smith stated that Brexit will cause the people of Scotland to be removed from the EU against their democratically expressed will, and that Scottish independence from the UK could provide a means for Scotland to rejoin the EU. Many newspaper headlines cited his closing line: "I'm asking you to leave a light on so we can find our way home." On Sky News in May 2019, Smith claimed that the Brexit Party is "a shell company that’s a money laundering front". After the party's chairman threatened legal action, Smith apologised unreservedly and admitted that he had no evidence for his allegation, made a major contribution to the party chairman's legal costs and made a donation to charity. After the election, Smith became President of the European Free Alliance group in the European Parliament, and by holding that office, First Vice-President of the Greens/EFA Group.
He contested Edinburgh West for the SNP at the 2001 general election, coming fourth. At the 2003 Scottish Parliament election, he contested the same seat, again coming fourth. On 12 December 2019 Smith was elected as the Member of Parliament for Stirling, unseating the previous Conservative Party incumbent Stephen Kerr with a majority of 9,254 votes or 17.6%. Following his election, Smith ceased to be an MEP, as an individual cannot simultaneously be a member of a member state's legislature and of the European Parliament.