Apply anyway. My GPA isn't any better but it's still worth trying. Pick reasonable schools, don't waste your time or money on top twenty schools. I agree also with HappyQuark, and add that you don't even have to do a whole master's program, you can start one and apply to grad schools again while you're in one, and transfer credits over to a new grad program (which is my personal back-up plan if I don't get in anywhere this round). A 3.2 in physics isn't something to shake a stick at. Maybe most of the people on this forum have better than a 3.2, but I would bet that most students in general, even among those that go to graduate school, are a little more "average."
If your advisors are discouraging you, if possible, get new advisors. If they have such little faith in your abilities then they might not write good letters for you, not to mention the effect that it's clearly having on your morale and motivation, which could have an impact on other parts of your application, like your SOP (if you're feeling down on yourself, or overdefensive) or organizing all your deadlines (if you're feeling unmotivated). My advisors have been honest and encouraging (which is one of the reasons I'm not applying to more than one top 20 school, and I don't actually expect to get in, but I have realistic chances in the 40 to 80 rank range, and that's good enough for me, and I bet you would have good chances in that range as well, not all of us get to go to Cornell/Stanford/Berkeley/etc), and I know that they will write good letters for me.
I test slow and lose focus and have anxiety issues too, but I refuse to be a victim of it.