Most graduate schools have a few theoretical physics programs, and a handful have groups explicitly labeled "mathematical physics." Off the top of my head, Utah State University has a pretty active mathematical (not just theoretical, but mathematical) physics group, and they're not too tough to get into. UC Santa Barbara is a well-respected physics program, and they have a mathematical physics research group.
You might also look at some math departments. A lot of schools have a few math professors, particularly those studying differential equations or applied math, who do work that might be termed "mathematical physics."
Also, don't be worried about your classes not converging too much. You're just starting out, it sounds like, and your classes toward the end of your degree will appear much mathier, especially if your university expects their physics students to have a heavier math background. Introductory quantum mechanics is pretty much just a giant exercise in linear algebra with infinite-dimensional abstract vector spaces, and you might even see some group theory if you take a class on particle or condensed matter physics. GR requires some background in differential geometry, although a friend of mine reports that you kind of do things backwards compared to a mathematician (trying to understand a surface to get a metric, for example, rather than using a metric to characterize a surface). Complex analysis is really nice for solving hairy differential equations in E&M and quantum mechanics, although many of these will be glossed over in most undergraduate curricula.
It's my understanding that the trend only continues in graduate school. I've attended a lot of conferences and read papers in theoretical physics that are more interested in classifying a structure by its group or discovering whether or not a function is meromorphic. I even remember attending a colloquium on condensed matter that was basically a lecture in applied topology (somewhere, somehow, I just made a mathematician cry with that phrase).