I was also in your position. During my 3rd undergraduate year, I decided to pursue physics. So I got around to taking the physics GRE and applying for admissions my final year. I am happy to tell you that I am starting physics graduate school this fall. The shift is definitely possible! The task of taking the physics GRE, however, becomes a more strenuous one... since you probably didn't take those courses. I spent a summer going through the typical undergraduate books (thorton, griffiths, etc) to learn the topics covered on the exam and did fairly well on the exam (which may be one of the biggest factors that got me in). I also had some previous research experience with an experimental physics group. My recommendations came from a physics professor, the research professor, and an electrical engineering professor.
It was a tough shot... and I applied to many schools (and was accepted to many!)... but ultimately, the admissions committee wants to see that you are capable of doing physics although you may not have majored in the field. Being in an engineering or a math field gives you that qualification. Do well on your GRE's (since you don't have many physics grades to offer) and get good recs (more than he/she did well in my class). Of course, some schools will be stricter on this issue of physics major than others. However, a lot of schools are moving towards an "interdisciplinary" field where they try to work with other fields to make progress. So in some cases, it might be a plus that you are from a different field.
Anyway, I hope that helped. Good luck!