The cheapest way to start a business is to think of a business that has extremely low start up costs. Businesses that don't require any equipment or machinery are obviously good ideas.
Internet based businesses are often a low cost way of starting a business. Take my own software business, Winnersh Triangle Web Solutions. I started this business to sell some computer software products I'd developed myself. I used my home PC to write the software. There weren't any other business start up costs aside from starting a Limited Company (which cost around £50 to open), and buying the website domain name (which cost around £10 a year to register and host). The biggest business expense I then had was handling the credit cards customers used to order by software products. However, I managed to find a company that would charge for their cash handling processes by taking a percentage of the software's price. So this meant I could effectively start taking orders without having to pay another company a cent for their services.
If you're not sure how to build a website or you need a much more interactive website then you might need to pay somebody to build a website for you. However, if you have a basic information based website or a simple ordering process then you can often get by with just the popular WordPress website content management software. I use WordPress for many of my websites and it's pretty quick and easy to design a fabulous looking site and upload your text and photos to the site.
Banks can be good sources of finance for startup companies. However, bank loans aren't always very cheap, especially if you don't have much of a track record in business. So it's always worth asking friends and family to see if they're prepared to invest a small amount of money in your new venture. Just be careful about involving friends and family in anything that could be considered a risky venture.
Another place to find some cheap sources of funding are through business angel investors. You can often find these at local business networking events. Sometimes you can even get an experienced mentor to help you through the early stages of starting up your business.