A fast-loading website improves user experience and SEO. Choose a host with SSD storage, CDN integration, and optimized servers.
Check out GTHost.com and Hostingsource.com cheap vps solutions. They are reliable and trustworthy, so I suppose this might be suitable for your request.
When the time comes to back up your server, they transfers those changes to their multi-terabyte central backup system. It can be configured to backup as frequently as every hour.