The problem with development consultancies

Many of the start-ups we work with have, at one time or another, taken work out to a development consultancy, web shop, off-shore developer, or creative agency. Coming from a consultancy background ourselves (but no longer being one) it's interesting to see how engineering, media and web consultancies fail time and again to serve the needs of early-stage companies.

It wouldn't be fair to say that these are all failures of the consultancies themselves: several are not strictly the fault of the third party, but rather that many founders are ill-equipped to engage with them. Taking on development resource, be it through another company or directly, is difficult, and it takes a certain set of skills to manage this. Agencies will rarely throw their hands up and warn you.

If we take a sample of the thirty or so entrepreneurs we've talked with over the last six weeks, very few are happy with their development partners. Of those who are relatively happy, most are still engaged with them and still awaiting the launch of their project. Signs of Stockholm syndrome are common. From what we've seen, there are six main areas of failure.

Compliance & security

One of our clients had outsourced some work to Pakistan, and whilst looking at the (publicly accessible) development servers of that company, we found we had full access to a variety of highly confidential documentation from their other clients, not to mention a full, unencrypted database backup which provided details of all of a UK retail business' customers.

Serious security failures like this are uncommon, but failure to comply with relevant legislation is surprisingly prevalent. Even if the third party had secured their servers, the fact that they were storing complete sets of user data outside the EU sets them in breach of the Data Protection Act, and their storage of customer billing details breaches PCI-DSS (payment card industry) regulation. The customer's site was itself in breach of the Companies Act.

Sending work to Pakistan, India or China can reduce your costs, although managers may struggle to realise a net saving. If you do manage to secure that tantalising sub-£1,000 man-month, will the people you outsource to understand the legal and regulatory constraints you have? Will they know what they are doing well enough to secure your site, or will they be in a part of the world where Microsoft heavily sponsor their training and they are limited to point-and-click on their servers? If you outsource abroad, retain a competent local expert.

Quality

Delivery quality management is a discipline that countless development managers fail to manage. Engineers and designers have a confusing plethora of tools and techniques to use, and it is not always clear to them which are best practice for the application that they are building. There are some principles which are almost always good to use: agile, design patterns, avoiding fixed scope, source code management, web frameworks, Open Source, and so on, but is isn't black and white. If you're going agile, which methodology should they use? Extreme programming? Scrum? Should they be test-driven? Should they use test-first and continuous integration?

Start-up customers usually have high hopes and low budgets. If it doesn't get delivered at a level of quality to get investment or revenue, they're sunk. A lot of agencies are used to working with established businesses and just don't understand this. Get an independent third-party to take a view on their methodology before they start. If the start-up has an independent technical face to follow the build and chase developers, agencies will be much more likely to deliver well.

Specification & contract control

Start off in control of the specification, and share it through user stories or something similar to create joint ownership. Founders know this, and they start off with a specification, but often as the cost of delivery increases and the product nears release, they start to tune their requirements. Most of the pre-money business models we see change three times before they get either investment or revenue. As such, it's not a good idea to commission your project fixed-scope in its entirety right from the word go. In the event that the developers actually build what was asked for, it's all but guaranteed that what was asked for is then not now what was required. When this starts to become clear, often the agency project manager steps back and lets the customer work directly with the programmers. All of a sudden there's no clear spec, the user stories have been abandoned, and the agency abandons its methodology.

Most agencies and consultancies that read this will say "of course we'd never do that, we're [agile/professional/highly competent]". Realistically, they're not. And it's often not their fault. It's mostly the fault of those engaging them.

Another issue with specification control concerns technical specifications. Sometimes the most remarkable things happen, and we've seen an agency that has delivered beautiful search engine optimised Ajax-driven sites release static Web 1.0 monstrosities. When we intervened and asked them why they built it to be SEO unfriendly the reply was "we haven't optimised it as there are hundreds of techniques and it's completely ready for SEO enhancement". Yes, anything that isn't search engine optimised is ready for being optimised...

Project specification should be done with a view to any compliance issues that business might face, but don't forget to specify the platform metrics. Web and user metrics are one thing, but what are the operational metrics that will be reported on to the investors, and which are key to report on to monitor business performance? Specifying which of these should be monitored up-front should save a costly or inaccurate data-mining process.

Finally, contract metrics are important. Your contract with the agency should handle what happens when only X% of the functionality is delivered, and consideration of what defines a successful implementation is important.

Cost and cost management

Odds are, you should double the cost if the project involves more than customisation of an existing platform. If they're good it may only be 50% more than quoted cost. If you build a prototype first you're much more likely to know what you really want, and to get a feel for build-cost creep. Micro-proofing, or building inexpensive prototypes as often as possible, is a great principle.

Intellectual property

It's really important to understand whether a business needs to establish technical IP. Contracts and specifications need to be clear. Are the agency developing a product that the customer will own? Or will they have a perpetual license, or an ongoing licensing cost? Are they building something, adapting Open Source or customising a proprietary white-label product? We've seen an agency turn around (after billing for the full project build) and when asked to clarify an unclear contract, they said "you have a perpetual non-transferable license". In short: customer has no technical IP and doesn't even own the product they thought they bought, investors flee to hills.

Even if the project is being built for ownership of the customer, is it being built to run on a proprietary framework? Are they offering escrow on the source of that framework in the event of the agency's liquidation (& could it realistically be managed if that happened)? Is the software being built using Microsoft servers or software, which will introduce additional licensing cost if hosting providers are changed?

Disengagement

Rather than spell out how disengagement from the agency can work, here's an oft-repeated series of steps which should provide food for thought.

1. Founder engages consultancy with specification. Low cost fixed-scope agreement. Contract formal but not always well-understood or as wide-ranging as necessary.
2. The project starts agile with an experienced team or experienced team lead.
3. The team start to deliver functionality, and the founder is able to try it out... and changes the specification.
4. The cycle continues with increasing changes to specification, leading to closer interaction between founder and developers. The project manager's grip weakens on project as it turns from fixed to unfixed scope. As the scope changes, so does the potential for cost, which isn't always clear.
5. Development speed is decreased: when the specification is changing a lot, it's more sensible to have fewer developers exploring the build, the idea being that it will get back up to speed once settled. Realistically, this is the point where agencies put their good staff onto whatever new business they have, and push their "work experience" staff onto the founder's project.
6. Technical complexity increases massively, and best practices are discarded. Developers now understand the specification to be coming from the founder, ad hoc. Cost may now be increasing beyond original quote on day rate of £600 or more.
7. Something gets delivered. There are a series of problems:
1. It's complicated and technically unsound. Maintenance or further development will be much more expensive.
2. A lack of product documentation becomes apparent. Excuses include "we allocated all resource on development", "the code is commented", "the specification is the documentation", "it's too simple to need it".
3. The product isn't search engine optimised / Web 2.0 / Ajaxified / Insert missing feature, but founder saw examples of the agency's work with that functionality. We've heard "we also do SEO, but we didn't design it into yours... you didn't ask"!
4. The agency presents a bill which is larger than expected. The founder isn't clear on what needs to be done to get the project to phase 2, although phase 2 is often academic as phase 1 has since changed. The bill will wipe out the founder's investment or more, and they're faced with a dilemma. If they can't pay the bill, do they play brinkmanship and wind down their start-up? The agency may offer to reduce the bill for an equity stake in the start-up, or a commitment to further work being put their way.

The really important bit: how to work with developers if you're a start-up:

• Keep it simple. If you think it's complicated or they tell you it's complicated, consider this: are you building a space ship? If you're not, it's probably not complicated. (Just because it hasn't been done before doesn't mean it's complicated.)
• Double their estimates and their time-costs. Ask them what their man-month cost works out as. If they start to tell you they can't break down their quote that way, put your fingers in your ears until they're finished speaking, and then ask them again. The metric isn't vital, but it's good to have, and it's also good to let the agency know you want metrics on your terms.
• Micro-proof: build a prototype first. Avoid fixed scope contracts (you will change, they will learn, there will be several phases).
• Create a plan for business technology for the next three years. Put in three or four milestones and phases. If you don't know what you'll want in Y3, make some assumptions. The milestones will change rapidly, but it's important to be cognisant of the likely need for further technical work and cost. Investors want to you think about this.
• When you're in the mess, you might be seized by Stockholm syndrome: but it's probably better not the devil developers you know. Get help.
• Make sure the developers are "agile" and use best practices, and get an independant third party to vet or manage them. This need not be costly.
• If programmers aren't project or development managers (and odds are they're not), and you haven't managed programmers before, don't talk to them about specification. Sure, it seems you're on the same page, but really you're not.
• Get help from people who have started web businesses before.

We'd love to incorporate any feedback into this article.

PS. If you found this useful and we might be able to help, get in touch. Also, check out Daniel Kehoe, a friendly American freelance CTO that we're in touch with every now and then. He wrote a great post on start-up disasters when getting a team together.