Actually what grae has brought to our attention is just the heart of the problem. I want to end the dotted sentence like this though "It's not where you go, it's who you work with and what you want to do". There are many possibilities on what you want to do. Do you want to be a top class data analyzer? Do you want to be a good string theorist? Do you want to be the administrative head of a particle experiment? Do you want to be at a high paying but easy going research institution? etc...
After the mid of the 20th century the meaning of "physics" as an academic endeavor has changed a great deal. Early on you just wanted to be a physicist and pondered on. Nobody actually cared on the quantity of your publications but on the quality. Nowadays it is different, you have to merely survive by publishing articles even if you like the content of your work or not. I do not know if you heard about J. Van Paradis you can look for his articles form the "Web of Science". You will notice that he is like an article publishing machine but after some point he shows a deceleration on his article publishing speed. What you will also notice is that this deceleration is due to his death. You did not misunderstand me; he first died and then his article publishing speed is decelerated, but not STOPPED. It is obvious that some other people wanted to use the reputation of J. Van Paradis and added his name to their articles providing that they used incomplete works by Van Paradis to complete this article.
J. Van Paradis is a well known astrophysicist, he has more articles than Einstein, Heisenberg, Bohr, etc... but what is the point. I am sure that you get what I mean.
After this example I want to go back to the original problem. It is actually what you want to do. And no matter what you want to do, you have to learn how to do, and you need a good mentor for that. If you will be working with someone and the only thing you do is to handle the calculations of this someones' researches without not even understanding the general idea behind the researches or where are these researches going; you are going to end up metamorphosed into a walking computer algebra program.
On the contrary, if you work with a person who shows you how to conduct a research, how to think through all the enormous publications work going on in physics, which idea to pick up as a reasonable and a promising one, how to connect the dots and how to triangulate then you are going to end up a physicist.
So it is first what you want to do, then who (or which group) can provide you with the abilities and resources necessary for your goal. You should have done this before you have applied though...
You can use help of both arxiv and web of science for choosing. You have to read the articles of the people you are considering. Even though you will not get the whole content of the paper, just take a brief look at it, read the abstract, and see if you enjoy the train of though process of that person or group.
For example if you want to be a high energy theorist but on phenomenology, then why are you applying to Harvard? Harvard High Energy professors are working on strings, they will be no use to you.
I hope that I managed to explain myself clearly.