In USA, look at the quality of the diesel fuel as a partial explanation:
"There is no argument anywhere in the auto industry that more low-sulfur diesel fuels are needed to help bring cleaner, advanced, direct-injection diesel engines to market in the United States." -- "The Debate Over Diesel", by Warren Brown, Washington Post, September 6, 2002
http://www.washingtonpost.com/ac2/wp-dyn?pagename=
article&node=&contentId=A45702-2002Sep6¬Found=true
"One reason is the poor quality of diesel fuel sold in the US. A 1998 report on fuel lubricity worldwide found that diesel fuel sold in the US and Canada is some of the poorest quality fuel in the world. Fully 50% of the US fuel was found to be below the standards recommended by equipment manufacturers."
http://www.biodiesel.org/pdf_files/Lubricity.PDF
Big Oil has dragged it's feet in cleaning up diesel fuel in the US. Europe started producing cleaner, low-sulfur diesel fuel in 1990; the US plans to follow in 2006. And the new generation of clean diesels either can't meet the US emissions standards on the dirty US fuel, or they can't even use it. But they run just fine on biodiesel, with very low emissions.
Also, check out VW, Toyota, Ford (outside USA), and Opel diesel engine. It can be done cheaply and cleanly.
"Volkswagen AG is working on a novel 'engine-based' approach leveraging complex chemical and catalytic reactions to achieve the near-zero particulate emissions. And supplier and automaker R&D operations are running full-tilt to perfect prototype NOx-reduction technology like Toyota's promising Diesel Particulate-NOx Reduction (DPNR) catalyst, which also employs advanced chemical/catalytic processes to scrub out the last vestiges of diesel emissions nastiness..."
USA's Diesel Sales vs Europe:
Diesel engines power 37% of all new cars sold in Europe (62% in France), with the share predicted to rise to 45% by 2005 -- but fewer than 1% of new American cars have diesel engines.