I presume you already got the offers from these schools...
I can understand the difficulty of choosing Harvard or MIT, but is it even a question to compare Harvard vs UCSB?
Choosing a smaller school just because there is one professor you are interested in is a very brave choice.
That one professor might move to a bigger institution during your 5-6 years of your PhD, or taking a sabbatical leave, or die, and you might be in tough spot then.
I personally would prefer places where there are 2-3 professors of same calibre, just in case they cannot take me in for the whole duration. Usually, these are the larger schools.