For me it really depends on the fit and overall comfort of the heel, whether the front is sharp or rounded (I have slightly broad feet so I can wear only rounded front heels). I love my heels and have many pairs in the range of 3.5 - 4 inches. My highest pair is about 5 inches with platform in front, so it's not that bad, but I only wore that like 3 times..
The type of activity counts too. If I plan to shop the whole day, I most definitely won't wear my heels. If I'm going to weddings etc where most of the time will be spent sitting down then it's alright. The brand of shoes is important too. Some brands just churn out horribly uncomfy high heels so I stay far far away from them.