Free will is not the ability to do anything you want, or the ability to do anything you will yourself to do. It is the ability to will yourself to do anything you want to will yourself to do. That is why it is free will and not free action.
If a man wills himself to fly unaided, he cannot, but this is not a violation of his free will, it is a violation of free action, which, so far as I know, no one argues we have. Likewise, If a man finds himself in a cage and wills himself to escape, but cannot, this is not a violation of his free will, it is a violation of his free action.
This is commonly referred to as a second order desire. Another way of positing the question of free will is to ask, "do we want what we want to want?".
Let me jump to examples. I don't want to go to the gym. However, I wish I wanted to go to the gym. In this case, my second order desire is not fulfilled. However, it could be that while I want to have the desire to go to the gym, I also have another stronger desire that prevents me from doing so. I want to have a desire to go to the gym, however my desire to do heroin is stronger, so I will stay home. (note: I don't condone the use of heroin, it makes for an easy example though) In this situation I can even want to not have the desire to use heroin, even though I actually have this desire, so this is a violation of my second order desire. It is in these second order desires that the question of free will lies. In the case where I am a stay at home heroin addict, my free will would be violated, because my second order desire (not wanting to desire heroin) is contrary to my first order desire (desiring heroin). This has also interfered with my other second order desires (wanting to desire going to the gym, but not actually wanting to go to the gym), thus I would argue heroin addiction has led to, at the very minimum a reduction of my free will.
Take the heroin out of the picture though. I want to desire to go to the gym (second order desire). I also want to desire to spend time with my children (second order desire). In this situation I can want to go to the gym and want to spend time with my kids (both first order desires), even if I don't actually go to the gym because I would rather spend time with my kids. My first order desires align with my second order desires even though my actions are limited by time constraints.
Another example is wanting to want to take an extravagant vacation and wanting to want to save money for early retirement (both second order desires). Again, I can want both as first order desires, but because of financial constraints, one will take priority over the other, but my desires align so there is no loss of free will.
More complicated examples would be situations of depression or anxiety. I want to desire to spend time with friends and family (second order) however, I don't in fact desire to do so (first order). Or, I want to desire a life of partying and socializing (second order), however I don't actually desire such a life (first order). Here there is a conflict between first and second order desires, so I would argue that both depression and anxiety limit free will because we are unable to desire what we want to desire.
I can't recall the title of the paper I read that argued this view, but I believe it was by Frankfurt. If you'd like I'm sure I can look back and find the paper and provide the title.