Von Bergen began in the youth division of FC Hauterive, before joining Neuchâtel Xamax in 2000. In summer 2005, Lucien Favre, then coach at FC Zürich, scouted the defender and brought him to the club. In his first season in Zürich, Von Bergen earned 36 caps and contributed to the clubs first championship since 1981. The following season, FC Zürich defended their title.
Hertha BSC
When Lucien Favre transferred to Hertha BSC in summer 2007, he brought von Bergen with him to the Bundesliga. Von Bergen made his debut in the German top division on the second match day against Arminia Bielefeld. During his first season in the German capital, Hertha finished tenth, but qualified for the UEFA Cup via fair-play rating. In the 2009–10, after the departure of Josip Šimunić, Von Bergen managed to establish himself as a regular in Hertha's starting formation.
Italy
After his contract in Berlin expired in June 2010, Von Bergen agreed to join newly promoted Italian Serie A club Cesena on a free transfer. At the end of 2011–12 Serie A, Cesena relegated. In June 2012 he moved to Serie A club Genoa on a free transfer. His stay lasted only two months, as in August he was sold to fellow Serie A club Palermo for €1.7 million in an exchange deal for Greek goalkeeper Alexandros Tzorvas and Paraguayan striker Daniel Jara Martínez. At the end of season Palermo relegated.
Return to Switzerland
On 19 June 2013, he moved to Young Boys for €850,000. He was part of the Young Boys squad that won the 2017–18 Swiss Super League, their first league title in 32 years. Von Bergen retired at the end of the 2018–19 season.
International career
Von Bergen made his debut for the Swiss national team on 6 September 2006 against Costa Rica. He has been a constant member of the Swiss team, and was called to a preliminary squad for Euro 2008. Due to a broken hand, Von Bergen was not able to take part in the tournament. Von Bergen was part of the Swiss national team at the 2010 World Cup. In the first match, where Switzerland shocked Spain 1–0, Philippe Senderos was substituted in the 36th minute after injuring his ankle seriously in an accidental collision with a teammate, and Von Bergen replaced him. It was his first appearance in a major tournament. At the 2014 FIFA World Cup, Von Bergen was himself substituted for Senderos after being injured in the 8th minute of the second group game against France. A high foot from Olivier Giroud fractured his cheekbone and he left the tournament to receive treatment back in Switzerland.