I would only consider cost of living / stipend offers as a "minimum" requirement. That is, if the stipend is high enough to live the lifestyle I want, then additional money on top of that doesn't make it "better", in my opinion.
But "enough" is very subjective and personal. It's not silly at all to ensure you want a minimum standard of living. Graduate school is not easy and if you make academia your goal, it will be a tough path with a lot of sacrifices. Therefore, it's not realistic to expect yourself to be able to sacrifice everything in order to make an academic career work. You have to decide where to draw the line--what is the limit for you.
For me, the "minimum stipend/cost of living" is fairly high. I want to have children near the end of graduate school, so I did not want to go to a place where I had just enough to make ends meet. My standard was a place where my stipend was high enough that I could live a modest, but comfortable lifestyle and still save about $2000-$3000 per year (for future children and also for vacations and travel to visit family/friends). So there were definitely many places I didn't apply to at all because of cost of living considerations and many places that I declined offers because the stipend was too low!
That said, between the final 3 schools I was considering, I ended up going to the one that gave me the lowest stipend offer after adjusting for cost of living. As long as I could live comfortably, I don't think having a few extra thousand dollars per year is worth it in the long run. Unless you are in some very special case where the difference is more than $10k/year! (But this is pretty rare in grad school!)
For what it's worth, my supervisor had the same philosophy when applying to graduate schools and postdocs too. And I plan to do the same! If/when I am a postdoc, I will have children to take care of and this means I want to have a certain salary and live in certain places. If I can't find a postdoc in a place that meets these criteria, I'd be happy to leave academia to do something else.