If you plan to remain in physics (in either academia or industry), then the most important aspect of a graduate school should be (in my opinion) whether there are research opportunities that interest you and are likely to be productive (i.e., good personality match with advisor, sufficient funding to enable you to spend your time on the research project, etc). Usually, these opportunities are more numerous at a higher ranked school (thus, the higher ranking), but your choice should be based on the availability of projects that are a good match to you, not the more generalized case. Depending on the schools, you may actually be better off at the lower ranked school if they have a strong research program in your area of interest.
However, rankings may matter if you plan to leave physics. Non-physicists are less likely to know that modest-name school X is a top-notch school in research area Y, and that big-name school Z has a rotten program in that particular research area. Thus, if your primary goal is to impress non-physicists (either a future employer or your friends and family), then you should consider the overall rankings in your decision.