If this is a US school, then they are almost certainly bound by the Council of Graduate Schools April 15 resolution: http://www.cgsnet.org/april-15-resolution
. As stated in the resolution: "In those instances in which a student accepts an offer before April 15, and subsequently desires to withdraw that acceptance, the student may submit in writing a resignation of the appointment at any time through April 15. However, an acceptance given or left in force after April 15 commits the student not to accept another offer without first obtaining a written release from the institution to which a commitment has been made." So, yes, you can decide not to attend even if you have already said that you would. Send a simple e-mail stating that you have reconsidered and no longer wish to attend Institution X and are thereby declining their offer of admission and financial support at this time. You might let them know why you have changed your mind, but there is no obligation to do so.
However, having said that, while you are not legally bound by your early decision to accept this offer, you should carefully consider whether withdrawing your acceptance is actually the right move. Physics is a small community. The actions you take now may be remembered later. Withdrawing an acceptance is a very rare occurrence. Do you want to be remembered for this action? The answer to that may be "yes" - it is far preferable to withdraw your acceptance now, then it is to attend and be miserable, or to drop out later. But if you are simply having second thoughts about attending this particular school, you may be better off starting a conversation with the director of graduate admissions about your second thoughts, rather than going directly to the withdrawal of acceptance. Faculty are people too. Many of us understand the complexities of making a good decision about which graduate school to attend. If you talk to the people at that school, it may make your decision easier (either way!). Start these conversations now, do not wait until the last minute. In fact, although it is legal, withdrawing your acceptance at the last minute (on April 15) without prior indication that you may choose to so is considered extremely unprofessional as this could negatively impact potential offers to students on the wait list. Thus, I strongly recommend that you initiate contact with the director of graduate admissions to indicate that you are having second thoughts as soon as possible. You may discover through that dialog that the school is still the correct choice for you, or it may more clearly elucidate why you are questioning your decision now.