I'm new here
I'm not in the same exact position as you, but I'm in a similar enough one.
I went to an engineering school right after high school and failed out (I wasn't prepared well for it, got unmotivated, and spent a lot of time going to parties and playing with boys and not going to class). After that I got a 2 year community college degree in engineering, and went to school full time for mechanical engineering (but for some reason found myself working in electrical engineering firms doing design and drafting). During that I also took up a second major in physics, because I found that I really loved my university physics classes and wanted to learn more.
Engineering school was interesting, and as time went on I found it harder to keep up with both engineering and physics. The attitudes and styles and priorities in the two programs were so extremely different that switching from one way of thinking to the other was very difficult.
I remember talking to one of my engineering professors about this conflict, and that I was thinking about choosing one or the other. She told me to stick with engineering, because even if it is less interesting, I'll always have a job. For some reason, that thought served to convince me to stick with physics, because it was what I felt passionate about.
I ran out of money and finished neither degree, and went into industry for a while, electrical consulting engineering. I frequently fantasized about going back to school for physics. I got to the point with my employer where it was time for me to progress in the company, and they were willing to pay for me to finish a degree in electrical engineering.
Perhaps I'm not a practical person but that led me to quit my job and go back to school for physics. I knew that if I was going to put in several semesters of work to finish, and hard work at that, I wanted it to be in a subject I was passionate about. I was going to have about five semesters of work to do this.
I'm almost done and I don't regret quitting my job, turning down an engineering degree, and going back to school for physics. I don't regret it all. Even if the job marker is harder and the work is harder, I found that it has been worthwhile. It is one of the best decisions that I made for myself, and I derive a lot of joy in learning about physics and doing reasonably well in my coursework. I'm not a star student, but I'm good enough, and I'm confident that I'll get into a few graduate schools in the 40 to 70 rank range. I have good letters of recommendation and good research experience, respectable test scores (from what I can tell), and an average GPA. Not good enough for top 20, but that's ok, most of us don't end up at top 20 schools. My undergrad institution is ranked around 50. I'll be 33 when I graduate. I'll have significant loans to pay off, but that's what working is for.
So I might be biased, but if your passionate goal is reasonably realistic, if you have the ability to do physics, even if it isn't as practical as another career could be, I say go for it, give it everything you have, and you won't regret it, and you won't wonder what could have been.