I would say it's both, but it's the physics (or whatever department you are applying to) ranking that matters more. Ranking itself does not affect your admission chances directly because there aren't any rules like "If School X is a Tier Y school, then it must only accept students above GPA scores of Z". Instead, the reason why top tier schools are competitive is that their earned high ranking attracts more applicants and typically, more competitive applicants. And these applicants look at both school-wide and department rankings, but I'd say that in our field, department rankings matter more than school-wide ranking.
Also, you are an international student, like me, which means that you will probably have a tougher time at public schools. As you may know, international students cost a lot more than American students at public schools, so fewer of us are accepted at public schools. I would say that for us, this makes the competitiveness of the tiers more "smeared out" and I would probably group all schools ranked from 1 to about 25 as the same difficulty level. This is because the top 10 schools are generally private schools (where the extra cost for international students don't apply) and the top 25 schools generally include the best public universities. You will likely find yourself accepted at private schools ranked high and rejected from public schools ranked lower.
I realise that both example schools you mention (MIT and UPenn) are both private schools, but I'm just saying this since I don't know the rest of your list. In my opinion, if you are a competitive international student, then you should not be deterred at all by ranking when applying. If a school is a good fit for your research and personal life, then apply for it!