I look for sincerity, positive reviews, due diligence, features and flexibility.
I don't care if the host is one guy and a server... I want the hosting business to be upfront... you know, do they own the datacentre, do they have a cage, do they co-locate their own servers or are they resellers. Always good to know whether the host buys new equipment, leases or using old cycled equipment (i.e. retired after 3 years of 247 enterprise usage). I don't want a host using the "cloud" term when all it is redundant power supplies (try buying a rack unit server with a standard ATX PSU! OR 2u/4u servers with just one PSU) or RAID configuration.
I have to admit I normally search for reviews in regards to whether a host is recommendable. I don't want the fantasists that create problems just so they can write on a blog how they fixed the issues within an hour. I don't care for good support either... nothing should go wrong; maybe the support is polite, helpful and fast but being able to have control directly via a control panel is much better than just having to open a support ticket for every little thing.
I always do due diligence... who host their server (their website in particular)? where is it located? Its not unusual for their main website to be hosted elsewhere to their clients, with small hosting companies. If its a UK limited company, I check to see their financial situation, CCJs etc
Features are important, what type of control panel (if any), try of VPS, port speed, installation etc. I recently cancelled some hosting... clearly said one-click install wordpress from the control panel, pretty proudly! Don't get me wrong, its not too difficult downloading wordpress, setting up the database, uploading the files to the server and doing a manual install... but if your host is selling you a package for wordpress but the scriptlicious (?) software has every blog software imaginable other than wordpress... there is surely more issues down the road with the hosting.
I like flexible payments... preferably paypal, monthly payment option (why get locked in for a year or two, rather not get the 2 months free) and money back guarantee where possible. Sometimes you can only find out how good a host is by actually using them. Bigger hosts suffer usually by having a couple of servers with constant issues whereas the rest typically work fine... just depends which server you end up on.
Also... Not cheeky beggars like Heart Internet that take payment before your order has gone to checkout if you add your card details to your account whilst a hosting package is in your basket! I changed my mind due to escalating costs... so didn't go through checkout (despite creating an account in advance), was surprised when I checked bank statement! They did email me begging to continue the transaction, just in case I forgot, or something. Not good.