I am a rising sophomore too, and I was in a similar situation when I took summer courses last year and got a 3.19 GPA. It definitely hindered my confidence, as I was never competent in high school (graduated with 3.0 GPA), and getting such grades to start off college with made me think that I am not good enough to do college level physics and math.
However, I worked really hard and disciplined myself last Fall and Spring, and earned really good grades. Though the initial drop in my GPA still hurts me, it was a good experience for me, as I it gave me confidence and taught me how to bounce back from failure.
Grad schools don't mind low grades in lower division general physics courses as long as you show competence in core upper division courses, and in fact, prefer a trend of improvement, so you shouldn't worry too much about it as I used to (a lot). I think studying over the summer and getting good grades your next semester will help you get back your confidence.