For myself it was trig. For the life of me that stuff would not make sense no matter how hard I tried.
Geometry was simple and straightforward, never gave me any trouble.
Algebra seemed tough for a time, so I doubled up on the class and it finally sank in.
Calculus did not cause major difficulty, had to work at it a little more; but, definitely worth it.
The only true answer is that the one which is hardest for you is the hardest. Similar to the sciences, some will excel in one branch and struggle in another, while a different student will have reverse experiences.