I intentionally save long conversations for the car because when I'm not driving I can make my time more productive by working online. Driving is otherwise unproductive dead-time. I find the ability to conduct business and get other conversations done while driving a very productive business and personal tool. But I use a headset so I don't have to hold the phone to my head, and more importantly, I'm a very experienced and cautious driver, so I never have any problems.
I find having passengers in the car far more distracting than my cell phone and I also find that changing stations on the radio and getting change out for tolls more interfering with my visual contact with the road.
And I see a lot of drivers out there who can't drink coffee at the same time they drive, and I see women putting on their makeup while they drive.
I think there are a lot worse distractions out there than cell phones and with far less productive value.
TEXTING while drive is a whole other matter. Anything which forces your eyes off the road while driving will be the primary risk factor when driving. Auditory distractions are inevitable and low risk in comparison. Likewise, distractions that occur because you are thinking about something else (like a conversation) are something which different individuals handle with a different amount of ease. But most experienced drivers that I know don't think about the road even when they are just driving alone. Most experienced drivers drive on "brain auto-pilot" a vast majority of the time.
The problem with cell phones is that people are pushing the buttons, not that they are talking on them.