Have you consulted your past PIs on where you can go?
Your undergrad institution is an Ivy, so with all your experiences, I presume it has a good graduate program too?
If one of them say, "sure, you're welcome to do PhD with my group". That would settle your anxiety, no?
In addition to asking them for a letter, you can also consult them on where you can apply to. They should know the playing field and can objectively assess your chances. Hearing from the ones in the field would be 100x more reassuring than from us, I guess.