I always look for a few things: Reliability, Support/Knowledge, Price - you get what you pay for really.
I have to say the support at
Libertyvps.net is awesome! I'm an older musician, trying to wrap my mind around web design and the whole internet thing, so I need a lot of support help. I've used their chat support quite a bit, as well as phone support, and they have always been patient and informative.