first, there is a wide spectrum of physics graduate schools. From the top all the way to the unknown ones.
Please tell us your target schools, because you'll always be able to get into a
school just that you might not want to go there.
I am afraid to even ask the graduate students I am around because I am quite embarrassed by it.
Get rid of these thoughts. As with my other advice in this forum, I advise you to go talk to as many credible people as possible. Especially, talk to your PI because he/she knows the admission process and can objectively guide you on where you realistically can go. That will solve your anxiety much more effectively than any advice from us whom you don't know personally.