CalTech. I know it has such a low female population (being I haven't dated in a year), but it is well-known for gravitational physics. Places like Princeton are also big on the mathematics of general relativity, but for some reason schools on the east coast just rub me the wrong way (I received a scholarship to attend a boarding school on the east coast back in high school and really didn't like the environment... might have just been the social-classes I was subjected to, but to me all schools out east give me that feeling). I'm from the west... I'll stick to the west.
Plus, walking through the same halls Kip Thorne and Feynman walked through would be amazing... they also have Sean Carroll