Yes, graduate admissions committees all know that the GRE is not really a good indicator of your ability in physics. However, because of the vast numbers of applications some of them receive, they are forced to use some quick-and-dirty method of narrowing down the candidates they will consider. When you are talking about a famous Ivy-League university that everyone here and abroad applies to, I'm gonna guess that this is a pretty extreme factor in their admissions process. I will say that I dont think an average score on the verbal gre will hurt too much when you balance it with such a high score in physics (630 is not low, it's about average, which is not bad considering this part of the test has to challenge people applying to humanities graduate programs in the same way the quantitative part was supposed to challenge us
Beyond that, it's really your letters of recommendation that will get you in. They have to answer two questions: "Are you a good researcher?" because that's what your graduate school will be paying you to do for five years, and "Have you demonstrated your research interests are in line with the research that goes on in this department?" Because obviously you want to go to a department where you can find plenty of stuff to do. Hopefully you know your recommenders well enough that you can sit down and talk to them about where they think you should apply, because their suggestions can be very helpful.
If you're that obsessed with your scores, I'll say this: do not bother re-taking the general GRE unless you think you can maintain your quantitative score, and raise your verbal score significantly
, which is to say by maybe 100 points. Keep in mind that this part of the general test is meant to be challenging to people who study words the way we study equations - the effort is probably not worth it.