My girlfriend did something similar, it wasn't a big problem. There is a bigger distinction between pure and applied maths than there is between theory and experiment for physics; which one are you interested in? If it's the modelling/systems research you did that interests you, you should look at programs in computational science. These programs seem to be heavy on maths subjects with intersections with physical/biological systems: check out, for example http://csc.ucdavis.edu/Welcome.html
at my (soon to be) grad school, or physics groups like this one at Pitt http://www.phyast.pitt.edu/people/fprofile.php?id=373
which work closely with statisticians. Groups like these will prefer someone with a physics background, and can be found at most schools from nos. 1 to 100.
If you want to do pure maths (proofs), though, then just rock the math GRE and apply to maths programs. You should have no trouble with the prerequisites. You also wont have trouble applying to applied math programs if you want to go that route, but the funding isn't as solid in that field, so sometimes it's better to do your applied maths under the umbrella of computational physics. The average faculty member in a maths program has many fewer students than in a physics program, so getting into maths programs are correspondingly more difficult (and you often TA for longer).
EDIT: One more thing: I'm (technically) an experimentalist, but I never work with an instrument, and my lab is just a computer. If it's just the time messing with the electronics that puts you off of experimental physics, consider something outside of accelerator physics or AMO, like (in my case) astrophysics, or complex systems research.
Interesting. Actually,maybe I'm better for theoretical physics. This is what I expect to learn about myself in the next year or two. Life is about the journey, I'm not in any rush. I think I've already made HUGE progress in that I've discovered that I hate aligning optics and fussing with amplifers. As for the astrophysics suggestion: just not my bag. I am much more a fan of the physics of the small. I've been to astrophysics talks before, and it's mostly just slides of stars with names like XNFGG9454954-osiris with auto-shapes drawn all over the place showing how a particular blurry pixel is the source of all that mighty. I just don't get it. Alas, I am naive re: astro so i hope this doesnt incite anger.
Does anyone know the relative competitiveness of theory programs to math programs? At, say, any given "top" school? I guess this question doesn't make much sense, but it appears to me that to get into a theory program at a top school, one could simply have worked very hard,etc, whereas to even be considered at the top 20 math schools you basically should be dominating the graduate math courses at your school senior year i.e. multi variable calculas in high school, etc.