There is no such thing as spot weight loss. It sucks, but it's true. There are 2 things you can do -
1) Lose more weight and
2) Build up your abs and quads, which will at least firm up those areas.
You may want to try some weight training, for both of those things. Swimming is lovely cardio but weight training is very helpful. Also, since you are a year-round swimmer, your body may very well simply be used to your routine. Shake it up a bit, by changing up the cardio to something else on occasion. Lots of folks end up with stalled weight loss efforts when they get into too much of a rut.