Creating a System to Share User Experience Best Practices at eBay
As presented at SIGCHI 2005 Conference [DOI Bookmark]
Increasingly, organizations are looking for ways to use technology to document and share information in order to increase effectiveness and foster a sense of community [1, 3]. As one example of a “community of practice” [4, 10], the eBay User Experience & Design group recently created a knowledge management system known as the Playbook to enable the eBay design community to share design best practices and other information.
This paper describes the considerations that went into creating the Playbook and how the working team enabled the rest of the eBay design community throughout the world to contribute their best practices. We also discuss how the group as a whole has benefited from this system as well as lessons learned during this process which may help other design communities outside of eBay.
eBay is an online, worldwide marketplace that enables people to buy and sell goods and services. eBay’s User Experience & Design (UED) group is responsible for researching and designing the user experience for the various eBay sites around the world. Over the last several years, the UED group has experienced dramatic growth and now has over 125 people in ten offices around the world (including Asia, Australia, Europe, and North America) working on the user experience for sites in over 30 countries.
To enable and support this growth rate, over the years the group has evaluated and sought to improve its processes and communication. For example, in early 2004 we heard from UED members in offices outside the US who wanted greater visibly into existing best practices and who wanted to share their best practices with group members in other countries. In general, we found that best practices, including project processes, were not being shared effectively within design offices or between offices.
The process for sharing best practices was inefficient for the person looking for information as well as the person who had the information. Since the best practices were often hard to find, we heard several cases where a designer would document best practices, assuming it had never been done before, only to discover later that in fact someone else had done so previously.
Also, as the group grew it was increasingly likely that multi-disciplinary product design teams would be made up of several people who were new to eBay and/or who did not have experience working together. As a result, teams needed clearer definitions of each role at each stage of the process.
To address these issues, we started the process of creating a knowledge management system  which we called the Playbook. Our goals were to:
- Enable the group to document and communicate best practices for how the group works, address problems, and takes advantage of opportunities.
- Enable everyone in the group to contribute to this information so that it always reflects our most current best practices.
- We believed that achieving these goals would:
- Ensure that best practices would be communicated more quickly so that others could benefit from lessons learned.
- Enable new group members to become productive faster and more easily.
- Enable existing members to share their experience with the entire group quickly and easily.
- Foster a sense of community among our growing set of UED members located throughout the world .
2. Forming the team
UED is a cross-functional group that includes five disciplines: Content Management, Creative Design, Prototyping, User Experience Research, and User Interface Design. We identified one person in each discipline to be on the working team. This had the benefit that each discipline’s needs would be addressed. In addition, it ensured that knowledge, expertise, and support for the project would exist across the organization
The team’s objectives were to create the Playbook within three months so that the rest of UED could start benefiting from it and to prove the benefits of knowledge management for the eBay design community.
3. Understanding users’ needs and creating requirements
The team started by applying a user-centered design process and considering user needs for such a system, including understanding each user segment, creating use cases, determining what content people needed, how they wanted to access it, how they would look for it, and how they would add, edit, and delete content. We also interviewed colleagues at eBay who had experience using knowledge management systems at previous companies to gain insight into what worked and what didn’t in these instances.
As a result of our research, we created the following requirements for the Playbook. It had to:
- Be easy of find information.
- Be easy to share information.
- Be scalable for the future, both from a user experience point of view and technical point of view.
- Be engaging. To be successful, we needed to cause a voluntary change in people’s habits. Therefore, the system needed to be sufficiently useful and dynamic to draw users back on a regular basis .
- Fit with eBay’s culture [5, 9]. eBay’s corporate and community values included two statements that influenced our requirements for the Playbook: “We believe everyone has something to contribute” and “We believe that an honest, open environment can bring out the best in people”. As a result, we decided that the Playbook needed to enable everyone in the group to contribute content without prior review or approval. It also had to enable anyone in the group to edit any content at any time.
There was one additional requirement we placed upon the initial system: as a practical matter it had to be cost-effective. Since we considered this initial system to be a test, the initial budget we requested was relatively small. We saw the initial version as the way to measure the benefits of the Playbook and to warrant future investment in what we thought would be an expanding enterprise .
4. Designing the system and adding content
Next, we investigated software solutions that would enable us to meet these requirements. UED’s Prototyping group, made up of several software engineers, provided the necessary expertise to consider a range of technical solutions, including whether to build the software ourselves, buy it from a third-party, or whether to adapt an existing open-source content management solution.
After considering a number of technical solutions, we found that Plone , an open source solution, met our user needs and requirements. It came with a user interface that enabled people to easily search for, add, edit, and delete information. The team also created modules to search for information in a variety of common file formats (e.g. PowerPoint and Word). In addition, the team made the system more dynamic by adding a module to automatically highlight information that had been added since the user’s most recent visit.
The working team also created the high-level information architecture for the Playbook. A recently created binder for new group members was a helpful guide as to what members needed to know. In addition, we interviewed people who had recently joined the group to determine their most common questions, what information they found useful, and what information they had difficulty finding. We also interviewed more experienced members of the group to determine what questions they were most frequently asked and what information they thought would be most useful to others.
As a result of these interviews, we identified the major content sections, including:
- Design principles, guidelines, and frameworks. For example, it’s important for a designer to know the design principles that are applicable across the eBay user experience as well as the design framework that applies to the specific area of user experience they are focusing.
- Business process , including how UED members can initiate proposals for new projects to benefit the eBay community and the business.
- Product process, including an explanation of each stage of the process, as well as the role and expectations for each job discipline.
- Administration information, e.g. how to order business cards, how to make travel arrangements.
- Member Profiles and Fun. To enable the Playbook to foster a sense of community, this section enables each person to create their own profile, including their photo, job role, areas of expertise, and their interests outside of work .
It was important to launch the Playbook with basic content so that, at launch, it would already be useful to group members. The working team identified content experts in each area and enabled them to add their content to the Playbook before we launched it to the wider group. Much of the information, such as the design principles, already existed so it was relatively easy to add this to the Playbook so everyone in each design office would have easy access to it.
5. Launching the system
The team launched the Playbook on schedule to the entire UED group. In order to ensure that the Playbook grew beyond the initial set of content, we raised awareness of the Playbook by giving presentations, sending email announcements to group, and announcing week-to-week growth rates of the number of users and contributions. In addition, the Creative Design group within UED created a series of posters that advertised the benefits of using the Playbook.
Six weeks after launch, the team (led by members of the User Experience Research group within UED) conducted an online survey to gain a better understanding of how the Playbook was serving needs and where further improvements were needed. The results of this survey informed our decisions on what to improve and we plan to repeat the survey at regular intervals.
In addition, the team leveraged a global UED summit, involving members from around the world, to officially launch the Playbook to all eBay design offices. During the Playbook session, members were provided an opportunity to:
- Brainstorm best practices they would like to share.
- Foster a sense of community with counterparts from offices around the world.
- Build awareness of how the Playbook would enable them to continue sharing best practices beyond the annually held summit.
To date, we have focused on understanding and addressing user needs, launching the initial version, adding content, and raising awareness of the Playbook to ensure it is used. As the Playbook is gaining acceptance and maturing, our next challenge includes finding a way to recognize not just quantity of contributions but also quality .
Based on the number of users, return visitors, content added, survey results, and anecdotal feedback, the launch of the Playbook has achieved the requirements we specified at the beginning of the project. We believe this success is due to several factors, including:
- Understanding user needs via upfront research and following a user-centered design process.
- Leveraging the skills from each discipline in the group and the willingness of the entire community to contribute content.
- Ensuring that those who are responsible for the success of the Playbook are the same people who will benefit from an effective system to share information [6, 7].
- Building and maintaining momentum by recognizing and rewarding people who are adding the most content to the Playbook.
- Creating a solution that is in synch with the existing culture.
We recognize that the Playbook will continue to be successful only if we ensure that the group works together to keep the information up-to-date and that those who need information are aware of the Playbook and find it easy and effective to use. Finally, we believe the lessons learned and summarized above are not unique to the success of the Playbook at eBay. We believe that other communities who follow a process with similar attributes will increase the chances that their system will be adopted by everyone involved and leveraged to serve the entire community.
The initial UED Playbook working team was composed of the authors as well as Ketan Barbaria, Elaine Fung, John Giang, Frank Gruger, Scott Joaquim, Yathish Sarathy, Steve Tse, and Craig Villamor. The following people also provided significant contributions and support: Kate Aronowitz, Kelly Braun, Larry Cornett, Nancy Dickenson, Andre Haddad, Beth Leber, Ishantha Lokuge, Justin Miller, Suja Raju, Lynn Reedy, John Sanborn, Luke Wroblewski, and everyone in eBay’s User Experience & Design group.
- Ackerman, M. S. Augmenting Organizational Memory: A Field Study of Answer Garden. ACM Transactions on Information Systems, Vol. 16, No. 3, July 1998, 203 -224.
- Ackerman, M. S. and Palen, L. (1996) The Zephyr Help Instance: Promoting Ongoing Activity in a CSCW System. In Human Factors in Computing Systems: The Proceedings of CHI ’96. New York: ACM Press, 268 – 275
- Erickson, T. and Kellogg, W. A. “Knowledge Communities: Online Environments for Supporting Knowledge Management and its Social Context.” In Sharing Expertise: Beyond Knowledge Management. (eds. Ackerman, M., Pipek, V., and Wulf, V.). MIT Press, Cambridge/London, 2002.
- Hildreth, P.M. and Kimble, C. Knowledge Networks: Innovation Through Communities of Practice. Idea Group Publishing, London, 2004.
- Kim, A.J. Community-Building on the Web. Peach Pit Press, Berkeley, 2000.
- Grudin, J., 1987. Social evaluation of the user interface: Who does the work and who gets the benefit? In Proc. INTERACT’87 (Stuttgart, September I-4, 1987), 805-811.
- Grudin, J. Why CSCW applications fail: Problems in the design and evaluation of organizational interfaces. In Proceedings of the 2nd Conference on Computer-Supported Cooperative Work. ACM Press, New York, 85-93.
- Herman, J. Process for Creating the Business Case for User Experience Projects. In Proc. CHI 2004, 1413-1416.
- Herrmann, T., Hoffmann, M., Jahnke, I., Kienle, A., Kunau, G., Loser, K., Menold, N. Concepts for Usable Patterns of Groupware Applications. In Pendergast, M., Schmidt, K., Simone, C., Tremaine, M. (Eds.): Proceedings of the 2003 International ACM SIGGROUP Conference on Supporting Group Work. ACM: New York, 349-358.
- Millen, D.R. and Fontaine, M.A. Improving individual and organizational performance through communities of practice. In Proc. 2003 international ACM SIGGROUP conference on Supporting group work, 205-211.
- Millen, D.R., Fontaine, M., and Muller, M. J. Understanding the Costs and Benefits of Communities of Practice, Communications of Association of Computing Machinery (CACM), April, 2002.
- Plone. http://www.plone.org.
- Resnick, P., Iacovou, N., Suchak, M., Bergstrom, P., Riedl, J. (1994) GroupLens: An Open Architecture for Collaborative Filtering of Netnews. Center for Coordination Science, MIT Sloan School of Management Report WP #3666-94.