I can't comment on how hard it is going to be, but I know a fella who took the psychology route in his late twenties after studying history in B.A.
Here he is:
http://www.hhdev.psu.edu/hdfs/faculty/blair.html
And he's doing quite well.
In fact, I, too, plan to take a course in neurobiology, one of my passions, in my late twenties. Learning never stops, does it ever? Of course, if you have kids, balance is going to be an issue.