Given how much you'll be learning over the course of several semesters, I think that you should be able to catch on. After a several year hiatus the review that I did for going back to school was several months of quality time with my basic calculus book and a little university physics. When I started back in I applied for any posted research job I could find that was related to physics and astronomy, and even cold-contacted a few professors who work I found interesting at my school but hadn't posted a need for undergrad researchers. I got turned down by most, but the few that have been willing to talk to me have been great resources. I ended up in a research group and even though I'll ultimately go into a different field it has been a great experience. Additionally, I pestered the admin offices and some professors for TA/grader jobs and got re-exposed to physics I and III material that way. It was a great way to reconnect with the material, even though grading is sort of a pain. The upper level stuff is, in my opinion, significantly more difficult than the lower level stuff, but if you have a university physics text book that you can refer to to help with the concepts, you'll be able to handle the upper level stuff. I would take some math classes if you can, like diffeq and linear algebra, or if you're school has a math methods course start there, because you might be more likely to trip up on the math than the concepts (at least I was).
It will come back to you albeit slowly sometimes. I think you need to work on building your confidence in your ability because you seem to be looking for reasons for someone to tell you not to bother doing this. I won't lie, it is hard to go back. I've been broke financially for a while and I have no free time, when before I went back I was able to afford all sorts of extra stuff and go out a lot. But it is worthwhile. Get used to rejection from potential research advisors, and understand that, in all likelihood, someone will eventually say "yes." But it will be easier after you've been there for a semester or two. And try to do an REU. I had two summers available for that and was rejected for the all ones for the first summer because I didn't have strong letters and had only a few classes under my belt since my return, but had a few choices the second summer, and that helps a lot. Focus on what you *can* do at this point, what steps you need to take. Be patient with yourself and with your potential advisors, and be persistent. Once you've shown that you can do the work over a semester or two, you might find that they are more willing to talk.