Ashbee was born in Southwark, London. He was by occupation a textile trader, the senior partner in the London branch of the firm of Charles Lavy & Co. He travelled extensively during his life, including Europe, Japan, and San Francisco, collaborating with Alexander Graham on Travels in Tunisia, published in 1887. He was an avid book collector, with perhaps the world's most extensive collections of Cervantes and erotica. Influenced by a friendship with the Belgian diplomat Joseph Octave Delepierre, his erotica collecting proceeded with purchases in Amsterdam, Brussels and Paris.
Ashbee was a part of a loose intellectual fraternity of English gentlemen who discussed sexual matters with a freedom that was at odds with Victorian mores; this fraternity included Richard Francis Burton, Richard Monckton Milnes, Algernon Charles Swinburne, and others. He also amassed thousands of volumes of pornography in several languages. He wrote on sex under the pseudonyms "Fraxinus" (Ash) and "Apis" (Bee), and sometimes combined them as "Pisanus Fraxi".
Ashbee's will left his entire collection to the British Museum, with the condition that the erotic works had to be accepted along with the conventional items. Because the trustees wanted the materials related to Cervantes, they decided to accept the bequest. The trustees were allowed to destroy any of the books if they had a duplicate, but in practice went much further and destroyed six boxes "of offensive matter which is of no value or interest" including cheaply produced Victorian erotica. The remainder of the works formed the core of the Private Case which were kept hidden from readers in the British Library for many years; they include a work by William Simpson Potter.
Ashbee married Elisabeth Lavy in Hamburg, Germany in 1862. They had one son, Charles (the designer Charles Robert Ashbee, born 1863), and three daughters. His family life grew unhappier as he aged. As he became more conservative, his family followed the progressive movement of the era. "The 'excessive education' of his daughters irritated him, his Jewish wife's pro-suffragism infuriated him, and he became tragically estranged from his socialist homosexual son, Charles". Henry and Elisabeth separated in 1893. Henry Spencer Ashbee is buried in Kensal Green Cemetery.