Don't sweat it. Seriously. From personal experience (feel free to track down my stats in one of the posts from earlier this year) it is definitely possible to get into some great schools coming out of an otherwise run-of-the-mill undergraduate program. One reason is that generally the curriculum doesn't change a whole lot from school to school (students at MIT aren't being taught a different kind of quantum mechanics than you are). More importance seems to be placed on the "total package" (ie your whole application), and in particular how *YOU* took advantage of the opportunities afforded to you not only by your home institution, but also by the national university/laboratory system at large. For instance, just because a university with a physics program ranks outside the usual top 20, there is no doubt that many of the individual physicists/faculty employed by that department are conducting groundbreaking research. There's just not enough faculty openings at Caltech and Harvard to accommodate all of the brilliant scientists out there--I know my home institution has a number of reputed condensed matter physicists, biophysicists, and we even recently acquired the famous theoretical physicist Paul Davies, despite our being a state university ranked right around 50 as far as physics programs are concerned. Thus the message I always try to convey to anyone expressing the same concerns as you (in particular, high school students worried that going to a state university for undergrad because they can't afford to go to a private technical institution like MIT will keep them from getting into a top-rated graduate program) is that your ambition should shine through clear as day no matter where you go for your undergraduate education, and graduate committees at big-name schools will *often* (though not always) recognize that. Make sure you involve yourself in research, talk with your professors, try to assimilate experience and increased lab responsibilities, and if your institution happens to lack an (interesting) research group, there are more than enough opportunities to get into a good REU over the summer as long as your academic work is reasonably sound.
So don't fret...that is, unless you've just been cruising through your undergrad for three years with a low GPA and no research experience and only now have decided you want to get into a top-ten physics graduate program. Then your hurdle might be a bit more of a pole vault.