After the US entered World War II, Fisher initially tried to enlist in the US Army, but was turned down because of poor eyesight and flat feet. He then enlisted in the United States Coast Guard, where he passed the physical examinations and was commissioned an ensign in 1942. His early assignments were organizing Port Security facilities in the Aleutian Islands of Alaska.
The team of 75 members of the US armed forces working with Fisher became informally known as "Task Force Fisher". Their assignment was to track down stolen valuables taken by the Nazis during the war. They were "...charged with locating and securing the enemy gold and loot before it could be destroyed or moved away." The small unit advanced with combat troops and were sometimes under artillery fire. Once they fired a volley at a German observation post. During the last few months of the war, the Task Force Fisher traveled 1,900 miles and located 6.65 tons of gold and 198,000 pounds of silver.
Recoveries
Task Force Fisher recovered gold, silver, gemstones and art from various locations in Germany as the Allied Forces advanced in the European Theater of Operations United States Army. Aue Reichsbank- 41 sacks of gold bars= 82 gold bars. All bars are re-smelted Belgian gold. Eschwege Reichsbank- 82 gold bars were found. Magdeburg Reichsbank- 6,074 bars of silver and 536 boxes of silver, which were looted from the Hungarian silver reserve. Merkers-Kieselbach- The Merkers mine held most of Germany's precious metals reserves. " Lt. Comdr. Joel H. Fisher, chief, Foreign Exchange and Property Control Section, Financial Branch, G-5 SHAEF... Fisher was assigned the responsibility for preparing inventories of all other mines in the immediate vicinity and to analyze all the testimony developed in interrogations to date with a view toward finding further gold and currency deposits as well as gathering financial and property control intelligence information." Nuremberg Reichsbank had burned 750 million French Francs, valued at the time at $17.2. Plauen- 35 bags of gold coins, 17 bags of US gold dollars, 1 million Swiss gold francs, 151,560 Norwegian gold kroner, 22 bags of German silver coins and 98,450 Dutch guilders. The gold coins, weighing over a ton, had been placed in the Reichsbank by orders of SS Chief Heinrich Himmler in April 1944.
Legal aid to displaced European Jews
In 1946, he married Mary Jane Johnson and, in 1947, the couple moved to Paris, where he was general counsel for the American Jewish Joint Distribution Committee in Europe, which helped to resettle Jewish refugees. Joel Fisher returned to Germany to assist in giving American legal aid to displaced Jews in Germany.
Post war law career
For nearly four decades until his retirement in 1989, Fisher headed Fisher, Sharlitt and Gelband, a legal firm that specialized in aviation law. From 1993 to 1997, Fisher also served as a consultant to the director general of UNESCO.