Well under the current economy you would have to be a fool not to be worried. However, there is only so much you can do and have control over.
On the positive side - I must be looked at well here - they made a lower level position into a higher level position to keep me here in Boston when I couldn't sell my house to relocate. With some of the changes, we anticipate that many people will no longer have direct reports but be individual contributors. My boss asked me which I would prefer - I believe that shows that at least a few people are thinking of me. I may not get the position I prefer - but it is nice that someone is behind you.