A nation of shopkeepers

Who originated the phrase "A nation of shopkeepers"? The usual answer is Adam Smith, but I think that is wrong. Smith was challenging the economic usefulness of the British Empire - often justified as providing millions of customers for British goods - and actually said something like "Having millions of customers would only be a good thing if we were a nation of shopkeepers". But Bartlett's Familiar Quotations says the phrase was used earlier, in the US congress, but frustratingly it does not name the speaker or give the context. Sometimes the phrase is ascribed to one of Napoleon's generals (Ney, I think) who said contemptuously "The British are a nation of shopkeepers". He may have said it, but he could not have originated it. Smith was earlier. Sometimes, this is confused further and the the quote is ascribed to Napoleon himself, but that certainly is false.

So, who was this Congressman? Was he talking about the British? Or did someone else originate the phrase?

And what's wrong with being a shopkeeper, anyway?