I don't think you will be in a huge disadvantage. By the time you apply to grad school, you will have the same educational background as any other physics undergrad, except you'll be a bit older, but that's not necessarily a bad thing since you will likely be more mature and come with more life experience! So, I wouldn't even consider you as a "non-science field" because you will have a full Physics BS, right? And, you have a great profile--research publications (I am assuming you mean in major peer-reviewed Physics journals right?) and high GPA!
My advice would be to:
1. Don't bother getting a higher GRE Q score. 164 is very good, well above most minimums. Most top Physics programs don't even list General GRE minimum scores. As long as your scores are still valid (i.e. <5 years old) then don't retake the GRE. Also, I think even STEM programs will care more about the GRE V than the GRE Q (many opinion articles online will show that many people think the GRE V is a better predictor of grad school success than the GRE Q). Also, the GRE Q only covers high school math--things you should already know very well if you have a BS in Physics. Also, see the point below:
2. Do well on your Physics GRE. I think the PGRE is way more important than the general GRE score for physics grad programs, especially more important than the GRE Q. I remember the toughest questions on the GRE Q being some weird obscure geometry stuff. The PGRE will test math skills that are more commonly used in Physics research and Physics coursework problems.
3. Make sure you write about your past experience in a positive way and demonstrate how you will fit in with the places you're applying. This is true for everyone, but since you will have more previous experiences than the "traditional" applicant, you have more material to draw from and thus you may be able to make a stronger case than the younger applicants.