Painful lessons from IT outsourcing gone bad

In tough times, companies look to shift tech work to outsiders, whether offshore or down the street. Be careful: This "cure" could be deadly.

By Ephraim Schwartz, IDG News Service

August 25, 2008

As companies look to economize in a weak economy worsened by rising energy costs, it may be more tempting than ever to consider outsourcing your IT -- whether to a cloud-based provider, to a shop in your town, or to a provider in some far-off land. Certainly, outsourcing has worked well for many companies, but it can also lead to business-damaging nightmares, says Larry Harding, founder and president of High Street Partners, a global consultancy that advises company on how to expand overseas. After all, if outsourcers fail, you're left holding the bag without the resources to fix the problem.

In his consulting, Harding has seen many outsourcing horror stories, from corrupt general managers "with all sorts of conflicts of interest" (such as service providers getting kickbacks from landlords on the leased space) to projects torn apart by huge turnover rates. "You end up with project teams that are hugely inconsistent. You might have a good team in place, but a month later, three-quarters of the team has transitioned," Harding says.

[ If  your IT job is moving overseas, maybe you should too. Find out where the hot jobs are abroad and what it takes to move where those jobs are, in InfoWorld's "offshore yourself" special report. ]

"Only when executed well can it pull out hundreds of millions in cost and transform organizations," says Brian Keane, CEO of Dextrys, an outsourcing service provider that focuses mainly on China.

In the sometimes panicked desire to save money -- especially with the powerful lure of "half-price" workers in places like India, China, and the Philippines -- good execution flies out the window. And that's where the problems flock in. Outsourcing is not for the faint-hearted or the ill-prepared. It just doesn't "happen."

That's why understanding what can go wrong before you jump into outsourcing is a great way to reduce your risk, because then you can approach outsourcing with eyes wide open, Harding notes. The companies who've lived through outsourcing horrors have two things in common: lack of preparedness going into a new relationship and lack of communication once the projects gets under way. Other factors can make these worse, of course.

Outsourcing's biggest horror show
In the pantheon of outsourcing horror stories, the $4 billion deal between the U.S. Navy and global services provider EDS stands out as one of the most horrific. It started back in 2003 when the Plano, Texas, vendor beat out the likes of IBM and Accenture for the contract. The deal was to manage voice, video, networking, training, and desktops for 350,000 Navy and Marine Corps users. But just one year later, EDS was writing off close to $350 million due to its inability to come even close to fulfilling its obligations.

The reasons behind the failure are complex, but suffice it to say that one of the major causes behind the debacle was that EDS, perhaps anxious to win the prize, never realized that the Navy and Marine Corps had tens of thousands of legacy and custom applications for which it was expected to either integrate or rip and replace. An EDS spokesperson said at the time the company's goal was to get the number of legacy apps down to a mere 10,000 to 12,000.

While there was plenty of blame to go around at EDS, the Navy took its share of blame as well. One of the major issues with the Navy was that the buck stopped nowhere. There was no single person or entity that could help EDS determine what legacy applications were needed and what applications could be excised. EDS, for example, found 11 different label-making applications, but there was no one who could say which 10 to eliminate.

[ If  you're planning to outsource, get the collective wisdom of those who've had success in InfoWorld's "13 best practices for outsourcing." ]

Most companies will never face outsourcing problems on the scale of the Navy and EDS. But many will face their own horrors on systems and projects just as critical. Consider these four modern examples and what lessons the companies painfully learned.

Horror No. 1: A medical firm's configuration management surprise
When Fran Schmidt, now a configuration engineer with Orasi Consulting ,was told at her previous job in the medical industry to head up a team to outsource the existing in-house development and quality assurance IT departments, she faced the usual problems.

"There was one Indian fellow no one could understand over the phone. It took us months to figure out what he was saying," Schmidt recalls with a smile.

That was expected. But what the medical firm didn't count on was that its existing configuration management tool, Microsoft Visual SourceSafe, which worked fine locally, would be a total bust when used collaboratively between two groups 8,000 miles apart. It took the remote teams in India an average of 13 hours to get updates on source code. And with a time difference of about 11 hours, the outsourcers were behind a full day's work.

"When we hit the [Send] button, there was no code done by the previous shift the entire time they were at work," recalls Schmidt. Not having immediate access to what was previously done the day before caused major problems for in-house developers. "All our progress schedules were behind. It's a domino effect with everyone playing catch-up." And the firm's customers paid the price: They were upset because they were not getting the same level of care that they expected.

The medical firm ultimately switched its configuration management tool to AccuRev, cutting the transoceanic updating from 13 hours to about an hour and a half. All told, it took around six months to recover from the disaster, Schmidt recalls.

The obvious lesson was the need to test your infrastructure before going live in an offshoring scenario. But the medical firm also learned another hard lesson: The desire to save big bucks so blinded the executives that they didn't realize they were replacing a group of people experienced with using a product to a group of people who were looking at it for the first time. "We underestimated the loss of knowledge that would take place during the transition," Schmidt says.

[ At many companies, IT is under extreme pressure, and may be at the breaking point. Find out how under-the-gun IT is responding to such duress in our special report "IT workers pushed to the limits." ]

Horror No. 2: Manufacturing efficiency doesn't extend to marketing
Executives in charge of a small consumer product group at Hewlett-Packard were under the gun. They were told in no uncertain terms to cut all costs related to getting the product into the big-box stores like Best Buy and Circuit City, recalls Margaret McDonald, then marketing manager for the HP department and now president of her own company, McDonald Wordsmith Communications. (McDonald would not name the product and would only say that it is sold today at places like Best Buy.)

"We were trying to get as much work as possible over to the Taiwan manufacturer with the goal to get the cost for these products down as low as possible," McDonald recalls. The Taiwanese outsourcer had a great deal of experience in getting the bill-of-materials costs lower, and HP was seeing that benefit. So managers started pushing for more savings elsewhere, insisting that the entire project be handed over to Taiwan -- everything from manufacture to writing the instruction manuals to all the marketing materials.

"These execs were being evaluated on cost, not on the quality of the brand," says McDonald. When she tried to tell her managers that what they wanted was unreasonable for an outsourced manufacturer to deliver, they accused her of just trying to hold on to her job.

As she predicted, the project turned out to be a disaster. Take this example of the Taiwan-produced marketing materials: "This glamour of new product will perfectly fit to your daily life from any of locations!" Of course, non-native English prose like that never saw the light of day, but it wasted six months until the higher-ups finally realized what was happening.

McDonald isn't sure her managers learned a lesson. She sees the failure not due to the offshore firm hired or even the miscommunication between the U.S. and Taiwan firms. Instead, she sees the problems as a failure within HP, between its own internal organizations. "The main [HP] branding people had no idea was going on." And the local managers reacted to the extreme cost pressures in a vacuum, with no concern for protecting the brand, McDonald says. The fact that the job was outsourced simply created the right circumstances for these internal flaws to finally become evident.

Horror No. 3: Giant telecom stumbles in transition to offshore
Steve Martin, a consultant and partner at Pace Harmon, a company that is often called in to help fix outsourcing deals gone bad, recalls the giant telecommunications company headed for disaster: It never considered the fact that although its new offshore provider, though good at coding, did not understand the business side of telecommunications.

The outsourcing project was divided into two phases. In phase one, all the internally managed operations were moved to an outsourced service provider (in this case, based in the United States). The idea was to test and stabilize the outsourcing approach with a local provider first, before taking the riskier step of moving the application development offshore.

The first phase went fairly well, so the telecom initiated phase two, shifting the effort to India. That didn't proceed so smoothly. The Indian provider simply didn't understand the telecom business, so lost in the transition halfway across the globe was all the telecom's inherent knowledge of the business applications -- what it is supposed to do and why. "All of that knowledge got left in the U.S.," Martin recalls.

Because the Indian firm didn't understand what it was coding, it took much longer to develop the applications. And they didn't work well, resulting in even more time and effort to figure out where they went wrong and fix them. It got so bad that the telecom canceled the offshoring midway and brought the effort back home.

Of course, there were lingering problems to resolve, such as how to handle the disputes over tens of millions of dollars in service credits the telecom believed it was due from the Indian outsourcer, which argued that it delivered what it had been asked to do. "An amazingly large amount of costs had to be reconciled," Martin notes. The two companies eschewed a legal battle to avoid the bad publicity, ultimately settling the dispute privately.

What the telecom company learned the hard way was that there is more to a deal than signing the contract. In the original deal, pricing took precedence over every other consideration because the executives wanted to show that they saved millions of dollars. Shortchanged in the process were the details of the transition, the development processes, and the governance. Adequate thought was not given to the obligations of the people who were responsible for executing the transition.

"The contract was executed from a business perspective, where it looked great, but not enough thought was given to how to programmatically move to the new environment," Martin says.

Horror No. 4: Service provider blacks out the client
James Hills, president of marketing firm MarketingHelpNet, probably had one of the most terrifying offshore experiences of all. When a dispute between his new company and the Web site developers grew heated, he came into the office one day -- only to discover the developers had shut his client's site down.

"I came in and checked e-mail. No e-mail, no Web site. They had simply turned it off. It was all gone," Hills recalls. While he was shocked to discover this, in some ways, he was not surprised. After all, the relationship with the offshore provider had been troubled from the start.

It started when Hills took on an assignment from a major client. Rather than trying to develop Web design skills needed to complete the client's project, Hills decided to farm that part of the job out to an offshore provider in the Philippines, at a savings of half of the cost of working with a local Web site designer, says Hills.

As soon as the offshore provider began sending back completed work, Hills knew there was trouble: "Functionality and community features didn't mesh properly and the design wasn't what we were looking for." On top of that, the offshore provider continually missed deadlines.

Becoming increasingly frustrated, Hills didn't make the final payment. The result, of course, was a panicked wake-up call from his client telling him there was no e-mail and no Web site.

Looking back, Hills says that if had he to do it over, he would have been more diligent in checking references. He did only a perfunctory check of references, unfortunately taking it for granted that the offshore design firm actually created the Web sites they claimed.

Time differences also played a key role in the soured relationship. "We weren't able to communicate directly, only through IM," Hills says. And as a small startup at the time, he couldn't support multiple shifts at home to get overlap with India, nor ask his staff to work 20 hours a day to cover both time zones. And sending a manager to the Philippines was out of the question, Hills says.

Hills doubts he'll ever outsource again, but if he did, he would insist that the job be done with a U.S.-based company that puts its offshore staff onto the company's payroll. "No contract workers," Hills says tersely.