Loeb was born in New York City on May 2, 1930, to businessman John Langeloth Loeb Sr. (1902–1996) and Frances Lehman (1906–1996). His mother was a direct descendant of Meyer Lehman (1830–1897), one of the three original founders of Lehman Brothers banking family. He is a great-grandson of Adolph Lewisohn and grand-nephew of former New York Governor and U. S. Senator Herbert H. Lehman. Ambassador Loeb’s father and paternal-grandfather, Carl M. Loeb (1875–1955), were founders of Loeb, Rhoades & Co. Ambassador Loeb and his father share the middle name Langeloth. This comes in honor of family friend and businessman John Jacob Langeloth (1852–1914).
In 2011, Loeb married Sharon J. Handler. He was previously married to Nina Sundby and Meta Martindell Harrsen. His daughter, Alexandra Loeb Driscoll, was born to Loeb and Nina Sundby. His son, Nicholas Mears Loeb, is the child of Loeb and Meta Harrson. His daughter Alexandra and son-in-law Joseph Edward Driscoll have blessed him with his grandchildren Aiden Edward and Allegra Frances Driscoll.
The Loeb/Lehman family’s ancestry stretches back to the earliest Jewish colonial community in America. Ambassador Loeb is Honorary Chairman of the Sons of the American Revolution in the State of New York and is a member of both the Daughters of the American Revolution and the Society of Colonial Wars. Many family (including great-great-great-grandparents) are buried in the historic Colonial Jewish Burial Ground in Newport, Rhode Island. His family connections to Judah and Abraham Touro (America's first great Jewish philanthropists), and Moses Seixas (author of Congregation Yeshuat Israel's address to President Washington in 1790) are shown in the charts below.