Why do I feel bad and wake up early after only staying up a little late?

I usually go to sleep around 3 or 4 and wake up around noon or 1, but if I go to sleep even around 5 I'll wake up around 10 or 11 feeling like crap. I can't fall back to sleep when I wake up like this, it's like my body is uncomfortable and simply cannot do it. Is it maybe related to my bed or pillow or something?