I understand that Colin Baker was the one that everyone blamed for it going downhill...
Hmm, I would not blame this part of the series on the actor, but rather, the writers, production, and casting.
I actually thought that Colin Baker did a good job with what he was given, but the episodes where terribly written and repetitive.
If memory serves me right, his time as a timelord was dominated by The Trial of a Timelord episodes, which I thought was not up to par for the series as a whole. Many shows were connected, and it ended up being very repetitive and boring, in my opinion.
More recently, I thought it a terrible mistake (and I have no reason why it was done) to kill off all the timelords. In addition, getting rid of the best adversary, The Master, The Doctor ever had was just plain stupid. Anthony Ainley and Roger Delgado were superb.
Also, the characters The Rani and Romana were among my favourites.