A caveat: I will be attending Rutgers in the fall doing condensed matter. Maybe I met you at open house.
Are there particular parts of condensed matter that you find especially interesting (nanoscience, strongly correlated systems, quantum computing, etc...)? Look at the research interests of the faculty members at each place and see which place has more faculty members doing work you want to do. I know Rutgers has more faculty overall in condensed matter, but the overall spirit of the faculty there seems to be more on basic research in fundamental topics of condensed matter physics (i.e. emergent phenomena, quantum criticality and phase transitions, etc...) as opposed to more applied stuff like nanoscience or more direct materials science. See what place has more in what you want to do.
In particular, see how many of the faculty members are active, have publications within the past few years, and have sent students on to positions after graduate school. A school could have 10 people who do work you find interesting, but if 5 of them are emeritus and not taking students, they aren't relevant for your purposes.
Also, it definitely doesn't hurt to think about the fact that you are going to be living in this place for 5-6 years. Look at which place seems more fun, affordable and overall livable. This is up to you to decide what you value in a place that you live, and evaluate Madison vs. New Brunswick/Piscataway for which one is better. A good environment will help in producing good work. In a related line, look at how much they are paying you, and how much work you will have to do to get that money that is not directly related to your research.
I hope this helps you get started. Good luck! Maybe I will see you at Rutgers in the fall.