I believe the body uses up muscles (the physical muscle tissue) first when exercising. You need to exercise for over half an hour (or something like that) before the body starts using fat reserves.
I know what you mean though! If I don't exercise for a few days and it feels like I haven't for weeks!