Best times of the year for both air and rail travel are nowhere near holidays or kids' vacations. Few people travel in October; many, many more travel in July and December.
Odd times are good, particularly very early in the morning if you want to avoid crowds. That's as in 5 AM or so. On Amtrak, for example, there's a huge quantitative difference in terms of the number of bodies in the train when the train leaves at 10 AM versus if it leaves at 7 AM.
What Linkat says about the days of the week is also true. If the flight/train ride huddles around a weekend, it'll be crowded, but Saturday is less crowded because most people neither leave nor return on that day -- and because Saturday stays often reduce prices. See, unfortunately, avoiding crowds and saving money are often exclusive of each other, although not always.
Desirable destinations, of course, attract more travelers, but more services are available, in the form of more flights or extra planes or railroad cars. Less desirable destinations have few people going to them, so you end up often with one small plane or even a cancelled flight. Plus destinations are expensive (or not) depending upon airport taxes, which of course are included in the price.
For example, going to Boston is considerably more expensive than going to Providence, Rhode Island or Manchester, New Hampshire. This is because of airport taxes, and is why Southwest and the like (e. g. steep discount carriers) generally don't go to Boston's Logan Airport but instead go to a neighboring state or Hanscom Air Force Base if they can get permission to use it. I well recall a trip to somewhere in extreme Eastern Tennessee (Knoxville) was about $1,000 more than a flight to LA from Boston. This is due to the relative desirability of the two locations.
One more thing, you can often get discounts with AAA. Sign up, pay them the money (AAA memberships are less than $50) and the discounts can pay for themselves.
I know this, I used to travel for a living.