Applied Physics programs clearly state on their websites that they accept students from related engineering and pure science degrees. That means you can have a math, chemistry, EE, ME, or whatever degree. You've taken a fair amount of physics, you've done great applied physics research already, I don't see why it would be a problem. Score well on your general GRE quantitative, get good grades and letters of recommendation, and I don't imagine you'll be at a significant disadvantage. You just might have to take a few extra classes, but they wouldn't welcome applications from other degrees if this was a problem.
As an example, go check out the Cornell applied physics phd program, or Harvard's. They tell you their class requirements. Because it is a broad field, the requirements are vague and flexible. You have to take some physics courses, but you pretty much get to pick which ones so you can stick to solid state and what you know for the most part. Further, in an applied physics program, your electives can be in any of the related departments so you can take electives in EE, or biophysics, or nanotech fabrication, or optics, or whatever floats your boat.
I know this because I went out there and started reading webpage after webpage for applied physics programs. This information is not hard to get.
EDIT: oops, I reread your post and it seems you don't mention research explicitly. This is almost essential for getting into a top program, but you may still be ok for applied physics in a lower tier school. However, if you don't have any research experience in applied physics, in my opinion it might be best to stick with EE. Check out the research they are doing, it overlaps significantly with applied physics research.