Gartner Vendor Report Card project: Evaluating large ESPs

Oct 4, 2000
Gartner

By Lisa Stone and Linda Cohen

Gartner's Vendor Report Card project evaluates the leading North American IT ESPs to see how well they are delivering services to their clients. We explain the project and what clients can expect to see in terms of published results.

Gartner Research Notes

Gartner Research Notes offer valuable data that is compiled and written by analysts at Gartner, a business technology advisor based in Stamford, CT. (TechRepublic is a subsidiary of Gartner.) Every week in IT Consultant Republic, you'll find a new Research Note like this one, which will provide you with a snapshot of information about a particular mission-critical topic. To see more Gartner research, click here.

Key issue: What criteria should be used to determine the appropriate external sourcing options?

Gartner will soon complete the initial phase of its Vendor Report Card project, which evaluated 13 of the largest external services providers (ESPs) in the North American market. This project was designed to apply rigorous analysis on two levels to determine how well these vendors are delivering services to their clients. The first level of evaluation focused on enterprise-level attributes, while the second level of evaluation looked at the vendors in terms of specific service categories.

Vendor selection criteria

To be included in Gartner's Vendor Report Card project, a vendor must have had more than $1 billion in services revenue (based on 1998 numbers as reported to Gartner Dataquest). Additionally, the vendors selected reflect service capabilities that span multiple technology, process, or consulting services across more than five verticals. The vendors included in the project at the enterprise level are: Affiliated Computer Service, American Management Systems, Andersen Consulting, Cap Gemini/ Ernst & Young, Compaq Computer, Computer Sciences Corp., Deloitte Consulting, Electronic Data Systems, IBM, Keane, KPMG, PricewaterhouseCoopers, and Unisys. Several of the vendors elected to participate in less than six categories for the drilldown analysis. As the project continues, service categories will be updated and deleted, and new ones may be added.

Enterprise-level ESP attributes
At the enterprise level, the attributes considered include organizational focus, financial stability, strategic vision, recruiting, retention and hiring, project resourcing, and internal knowledge management. All these attributes impact how well an ESP is positioned to deliver services to its clients today as well as in the future. Vendor briefings related to enterprise-level attributes will be completed at the end of September 2000. Gartner will publish a Strategic Analysis Report that explores those attributes and how the attributes need to be considered for various types of service offerings.

Gartner does not believe there is a single best "systems integrator, consultant, or outsourcing" ESP and does not recommend that selecting ESPs for any specific project should be based on these enterprise-level attributes. However, it is appropriate to use the enterprise-level information as differentiators when two vendors appear equal in terms of their ability to deliver and as supplemental information to the following service categories.

Service delivery categories
Concurrent with the enterprise-level analysis, Gartner has been conducting in-depth analysis on how well service is being delivered in these categories:

IT strategy, planning, and architecture. This includes IT strategic plans, business-to-IT alignment strategies, and IT architectural planning.
E-strategy, design, development, and implementation. This includes the creation of an e-strategy, the design and development of Web applications that support that strategy, and the implementation of the intended functionality, including any necessary integration with existing systems and processes.
Full-service outsourcing. This includes data center management services, as well as application management.
Applications outsourcing. This includes the ongoing maintenance, enhancement, conversion, and support of an application.
Supply chain management (SCM). This includes SCM solutions, as well as processes and enabling technology.
Business process re-engineering/change management. This includes planning and implementation of new business processes and enabling technologies that represent significant change to existing business models.

The service delivery analysis included the development of a comprehensive evaluation for each service category, vendor briefings, and reference checking via a consistent set of questions for each service category with clients. Not every vendor invited to participate in the study elected to participate in every category. By year-end 2000, the evaluation criteria used at the category level, as well as the results of the analysis by service category level, will be published. Gartner recommends that enterprises use these criteria to evaluate ESPs in terms of the specific project/competency need they are trying to fulfill.

Bottom line
How ESPs manage their organizations internally must be considered when evaluating the various candidates for services. A rigorous, standardized approach to interviewing vendors helps determine their relative capabilities, strengths, and weaknesses. Past performance and, more precisely, recent past performance are significant indicators in terms of how an ESP will perform for any services needed. The results of the Vendor Report Card project will provide data on market directions and trends, and on how successful services are being delivered.

This report was originally published by Gartner on Sept. 18, 2000.

Join the discussion

What criteria should be used to evaluate a consultancy's performance? To share your opinion, post a comment below, or send us a note.