@Luay,
I am a software engineer. I have worked quite a bit with cryptography (as a user not a designer). I agree with the advice you were given.
Cryptography as a specialty is highly mathematical. The people who are experts in cryptography all have PhDs in mathematics and are experts in advanced topics in mathematics. If what you want to do is design, test, analyze and produce usable encryption... mathematics is the place to start.
If you want to be a software engineer who uses encryption designed by the Math experts, you still need to understand some of the math (math is all over software engineering). But in this case you might start with programming and computer science courses.
In general, in software engineering, the more advanced (and higher paying) the job, the more mathematics you need.
My advice, learn to love mathematics. The good news is that the more advanced you get, the more interesting the mathematics becomes.