Failed to get "never of it"-"never of what"?

Failed to get "never of it" in " in this motley throng though never of it". What does it refer to?

Context:

In San Francisco in the sixties, the popular promenade was through the streets Montgomery and Kearny from Jackson to Sutter. Here in the late afternoon might be seen as in a rapidly shifting kaleidoscope, a most unusual procession relieved here and there by the injected "characters," who lent life and color to the warp and woof of that most strangely variegated tapestry. A small army they were, each member living his own life and absorbed in his own mysterious schemes. Here were George Washington Coombs," known also as the "Great Matrimonial Candidate"; "Old Rosey"; "Money King"; "Robert Macaire"; the "Gutter-Snipe"; "Old Crisis," and others, all of whom long since have passed into oblivion. And in this motley throng though never of it, appeared "Emperor Norton."

Joshua A. Norton was his real name. He was of Hebrew parentage, born February 4, 1819, probably in Scotland. Of his earlier life nothing is known as he rarely spoke of it. Before coming to California he had been for some time at Algoa Bay, Cape of Good Hope, where he was a member of the Cape Mounted Riflemen. He finally reached San Francisco in December, 1849, having come from Rio de Janeiro on the Hamburg vessel Franzika.

MOre:
http://www.emperornorton.net/NortonI-Cowan.html