Honestly, Cosmicomic has the answer pretty set, but I'll add my experience too.
I was in a very similar situation you are in. I graduated with a 2.9 GPA in physics (if you include my first 2 physics courses, ~3.1 if you exclude them). I applied last cycle to PhD programs in the top 40 (US news), and didn't get accepted anywhere. I reapplied this cycle.
I am now currently sitting on an offer from Purdue (#37 according to US news). I applied for their Master's program, but they "view all applications equally, and transitioning to a PhD is expected", so I'm likely going to change to the PhD program there.
The difference between last cycle and this one is 6 things: 1) I improved my PGRE from 41 %tile to 66 %tile. 2) I got an extra year of research and a letter of rec from the PI. 3) my name is on a paper that is currently in preparation to be submitted. 4) I beefed up my CV/Statement of Purpose. 5) Changed the range of schools I applied to. 6) I emailed at least 2 profs from every school I was applying to asking for a skype meeting.
I talked to my PI the past year extensively about how to improve my chances to get into grad schools. So, to go over the 6 things a little more in depth:
1) if you have a "crappy" GPA, then the PGRE is where you gotta put most of your effort. You need to show schools you can excel in graduate courses and not flunk out of the program. I was told to get my PGRE up to 80% or higher if I wanted to look at more top tier schools...but didn't make it lol
2 & 3) Research is a big deal, I was told getting your name on a paper is a big deal. It shows you can conduct research and contribute to a lab. I also spun my application towards computational physics and my HEPEX research was mostly data analysis and algorithm development/optimization. So the schools that accepted me seem to have a good match in research and skills that I have/can learn quickly.
4) I tailored my CV to support my computer science background, I listed projects I wrote, including AI and Machine learning projects (look at Andrew Ng's ML course on coursera). I also listed extra projects, I helped another lab in developing augmented reality code. So I just threw that on my CV. Unsure how much "prettying up" the CV helps, but figured it didn't hurt. My Statement of purpose spent...at most one sentence explaining my grades, the rest was spent detailing my research and academic experiences, and outlining my goals/research interests. Make sure your overall application has a "theme", try to show on your CV, and SOP that your research and ability to learn are top notch...don't talk about lack of determination lmao
5) I applied to a wide range of masters and PhD programs. Double check that the school actually has a "terminal" master's program. I recommend calling the department to make sure. I considered schools in the top 50 to be reach, between 50-100 as match, and 100+ as safety. I've gotten rejected from safety schools, and accepted to a reach school. So no idea if my listing is accurate.
6) Emailing profs can sometimes connect a face to the application, and if you can present yourself well, it'll only help your situation. This step was useful for me because I got to actually see what the labs were doing, the research interests of the group, and I got a first impression of the personality of the professors. Some of them also provided insight on the application process and what I needed to do to increase my chances of success.
Overall, I ended up making "improve my application" a second job. I'd try to take on that mentality if this is something you really want to do. But the biggest things are definitely PGRE, research, and an "interest match" with the schools you apply to.