Professors Karim R. Lakhani and David A. Garvin and Research Associate Eric Lonstein prepared this case. HBS cases are developed solely asthe basis for class discussion. Cases are not intended to serve as endorsements, sources of primary data, or illustrations of effective or ineffectivemanagement.Copyright © 2010, 2011, 2012 President and Fellows of Harvard College. To order copies or request permission to reproduce materials, call 1-­800-­545-­7685, write Harvard Business School Publishing, Boston, MA 02163, or go to www.hbsp.harvard.edu/educators. This publication may not bedigitized, photocopied, or otherwise reproduced, posted, or transmitted, without the permission of Harvard Business School.KARIM R. LAKHANIDAVID A. GARVINERIC LONSTE INTopCoder (A): Developing Software throughCrowdsourcingIn December 2009, Jack Hughes, CEO and founder of TopCoder Inc., entered his ????????????????????headquarters in Glastonbury, Connecticut, eager to review a particularly complex softwaredevelopment project for an ????????????????????????????dynamic power pricing system. Eight years after foundingTopCoder, Hughes still enjoyed detailed project reviews. He was particularly proud that hiscompany could produce high-­quality software solutions for which his own employees did not haveto write a single line of code. Instead, the firm nurtured a global community of more than 225,000programmers who competed to design and create software modules for TopCoder clients, a processthat the popular press called crowdsourcing.1??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????-­free and operationalon its first day, a rarity in the software industry. Especially impressive to Hughes was that in fourmonths, 65 participants from 11 countries on six continents had competed in 57 contests to create thiscritical pricing system for the client (see Exhibit 1). As of 2009, TopCoder routinely producedsoftware solutions for over 45 clients, including AOL, Best Buy, Eli Lilly, ESPN, GEICO, and theRoyal Bank of Scotland.??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????changes in the software industry, while also pursuing its unique competition-­based softwaredevelopment approach. He had transitioned his business from a model that helped other software??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????traditional IT consulting services and competitions, mobilizing developers world-­wide to solveclients?? problems.The shift to a greater emphasis on competitions, encompassing all aspects of softwaredevelopment, however, meant that project volume was a growing issue for TopCoder. Hughes had tothink through how a competition-­based business model, which increasingly stressed contests as anorganizing as well as money-­making approach, could handle increases in numbers of competitions,clients, and participants. Hughes considered his own goal: attaining $200 million in revenue from ahigh of just over $18 million in 2008. He fundamentally believed that contest demand would spur thesupply of TopCoder participants, who would in turn create high-­quality software solutions. But, was1 ????????????????????????????????????????????????????????????????????????????????Wired Magazine 14.06, June 2006.For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .610-­032 TopCoder (A): Developing Software through Crowdsourcing2$200 million in revenue a reasonable goal? Did his assumptions make sense? If so, what would it taketo increase revenues by over an order of magnitude?Background and Current OperationsBefore he founded TopCoder in 2001, Hughes had built a custom software development2company, Business Data Services, in 1985;; the company name changed to Tallan in 1991. Tallanemployed some 600 people before being sold to CMGI in 2000.3 As he was completing thetransaction, Hughes reflected on what he had learned from his experiences at Tallan??the experiencesthat would inspire the core tenets of the TopCoder business model. Although Hughes enjoyed histime at Tallan, the company struggled in some areas. For example, recruitment was an expensive andfrustrating process because finding qualified programmers was time-­consuming and talent wasdifficult to assess???? ???????? ?????? ?????????????????????? ?????????????????? ???????????????????????????? ?????????????????????????? ???????????? ?????????? ???????????? ??????????????obsolete after only a few years of productive service, leading to high levels of employee turnover.?????????????????????????? ???????????????? ?????????????????? ?????????? ?????? ?????????????????????? ?????????????????? ?????????????? ?????????????? ?????????????????? ???????????? ?????????????????????????????????????? ?????? ?????????? ???????????????? ?????????? ???????? ???????????? ???????? ???????? ?????????????????? ???????????????? ?????????????????? ???????????????????? ????????????components instead of building each application from scratch.Drawing upon these and other insights, Hughes set about creating a new kind of organization thatwould build a ??community?? of programmers to help address the issues he had identified. Theseprogrammers would compete??as well as affiliate??by building and using components that hadalready been tested and found workable. The idea of reusing software components for new projectswould become the core of the solutions the new company, called TopCoder, provided. Hughes?????????????????????? ???????? ???????????????? ?????? ???? ??????o-­???????????? ???????????????????? ???????? ?????????????????? ???????????????????????? One side of theplatform would be clients, firms that needed software developed, who would work with his staff tospecify programming challenges. The other side would be community members who would competein contests to create solutions to the challenges for money and skill ratings. TopCoder would be in themiddle as the platform host, designing and enforcing the rules of engagement between clients and???????? ???????????????????? ?????????????????? ?????????? ?????????????????? ?????????????????????? ?????????? ????????????????d that the company needed toexcel at five core tasks: breaking down large client software projects into components, taking in andprocessing client project specifications, determining appropriate contest prizes, having a consistentand unbiased way of selecting contest winners, and fixing bugs at the back end of development.Setting out to amass an initial collection of highly skilled programmers, from 2001 to 2003TopCoder asked established software development companies to sponsor world-­wide web-­basedp????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????competition platform and provided the company with access to talented programmers from aroundthe world. In return, the sponsors, including Sun Microsystems and Google, used the contests to??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????vice president of finance, explained thatduring the sponsorship phase, TopCoder offered unusually large prizes??as much as $5,000 to$10,000 per match for tournament winners??to attract competitors and expand the community. Inaddition, every contestant that participated received an objective numerical rating for their2 Custom software development by specialist firms in the global IT consulting and services sector (for example, Accenture andIBM) was an over $500 billion segment in 2008. (Source: ???????????????? ?????? ?????????????????????? ???? ???????????? ???????????????????? ?????????????????? ???????????????????? DataMonitor, March 2009.)3 ???????????? ?????????????????? ???????????? ?????????????????? ?????????????? ???????? ??????????M???????????????? InternetNews.com, February 14, 2000, http://www.internetnews.com/ec-­news/article.php/303771/CMGI-­Acquires-­Tallan-­for-­920-­Million.htm.For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .TopCoder (A): Developing Software through Crowdsourcing 610-­0323performance against the global talent pool, providing a clear signal to TopCoder and others about thetalent in the community.By the end of 2004, the TopCoder community was 50,000 members strong. In its early efforts touse the community to generate revenue, TopCoder acted as a placement firm, matching top-­ratedcommunity members with firms seeking new talent. Hughes, however, was ????????????????????????????????????????????????????????the idea of TopCoder becoming a placement firm. That was not my end ????????????????????????????????????????????????In 2005, TopCoder began to use its community to develop software components and applications.Hughes first tested this model by having highly rated community members compete to redesign and???????????????? ???????? ?????????????? ???????? ?????????????????? ???????????????????? ???????? ???????????????????? ?????????? ???????? ?????????????? ???????????????? ???????? ?????????? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????itive proof thatcomplex software systems could be built through competitions.Initially, TopCoder adopted a model to create solutions for clients by contracting with communitymembers, running competitions, and providing consulting services. The company broke down thesoftware development process into seven distinct but interrelated tasks: 1) conceptualization, 2)specification, 3) architecture, 4) component production, 5) application assembly, 6) certification, and7) deployment. Most revenue came from consulting services: TopCoder billed clients for the time the???????????????????? ?????????????????? ?????????????????? ???????????? ???????????????????????????????? ???????? ?????????????????????? ?????????????? ???????????????????? ???????????????? ??????component design and development competitions, assembling components, and delivering finishedsolutions.Shortly after TopCoder started developing software for clients, the company identified reusablecomponents from the software it was creating and collected the components in a catalog. These????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????ue proposition to its clients. Manyof the custom applications could be produced by combining existing catalog components with newcomponents built through competition. TopCoder had also received eight U.S. patents for variousaspects of running online programming contests in a distributed community setting and had otherpatents pending domestically and internationally.?????????????????????? ?????????????? ?????????????????????? ???????????? ???????? ?????? ???????????? ???????????????????? ?????? ???????????????????? ?????????????????? ?????????????? ??????????????????????????????????????????????????????????????????????????????????????-­??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????2007 and 2008, TopCoder produced nearly $20 million in revenue, but platform manager costsremained high (see Exhibit 2 for information on revenue and platform manager costs). Attempting toalleviate costs, in 2007 TopCoder introduced competition tracks for component architecture andassembly. With these new competition tracks in place, the work traditionally done by platformmanagers would now be done by the community. In 2008, the company also added competitions insoftware development tasks, such as conceptualization and specification, as well as deployment andbug fixing.By early 2009, TopCoder had moved increasingly away from the hybrid consulting model. It nowfocused on completing all tasks in software development through competitions. Instead of paying fortime and materials for TopCoder platform managers, clients paid a monthly platform fee based onthe complexity of their software requirements and the estimated number of competitions they wouldrun through the TopCoder platform each month. The platform fee also provided clients withunlimited access to the ?????????????????????????????????????????????????????????????????????????????????????? Roughly 60% of most clients??projects could be accomplished through reusing components from the catalog. The company coupledthe move from the hybrid consultancy model to a competition model with the reduction of manyplatform manager positions, leaving the company with 16 project managers servicing 35 clients bythe end of 2009.For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .610-­032 TopCoder (A): Developing Software through Crowdsourcing4As of late 2009, TopCoder ran two different types of competitions on its platform: algorithm andclient software development. Algorithm competitions served as the primary means for attracting newmembers and retaining existing members. These competitions required members to develop creativesoftware solutions to relatively difficult programming challenges. All members were assessed againsteach other through an automated computer scoring system;; they then received a TopCoder rating fortheir performance. Some algorithm competitions also had cash prizes for winners.The second type of competition targeted developing software applications for specific clientneeds. A TopCoder platform manager initially worked with the client staff to develop a ??game plan????(see Exhibit 3 for a representative game plan) or a project road map for building the software. Thefirst step typically involved a contest where the general client problem was presented to theTopCoder community in a conceptualization contest. Here contestants publicly cross-­examined theclient staff as to their actual needs and then submitted a business requirements document and high-­level use cases. The client chose the submission or submissions that best represented ???????? ??????????????????needs as the basis for further development. Then a series of specification contests was held to createthe application??s requirements documents, application wireframes (i.e. the logical flow of theapplication), and storyboards (detailed cases of the user experience). Next, the output of thespecification contests was fed into several architecture contests to create the overall system andcomponent level designs. At this point, the TopCoder platform manager would work with the clientto either select components from the catalog or commission the creation of new components throughdesign and development competitions. After the component production phase, all the relevantcomponents were put together through an assembly competition with the objective of creating aworking system. Assembly was then followed by certification and testing contests and then,eventually, deployment. Throughout the execution of the game plan, TopCoder retained flexibility in???????????????????????? ?????? ???????????????? ??bug r???????????? ?????? ???????????????????????? ?????????????????? ?????????????? ??????????????????????????s or unforeseenerrors.To determine winners and assess quality in client software development, TopCoder used acommunity-­based peer-­review system. In particular, expert and experienced TopCoder communitymembers were paid to grade and comment on all contest submissions using detailed scorecards,ultimately picking the contest winners. The winning competitors for each contest then receivedmonetary prizes, and all participants received updated ratings for their performance. TopCoder alsoran studio contests if an application required logos or graphics;; in those cases, clients chose thewinners.Evolution of the TopCoder CommunityGrowth and CompositionFrom 2001 to 2009, TopCoder added an average of 25,000 new computer programmers to itscommunity each year. After filling out a short online registration form, anybody in the world couldparticipate in a software development competition;; by spring 2009, the TopCoder community hadover 200,000 members (see Exhibit 4 for community growth). Although the size of the overallcommunity was large, the number of people within that community who actively participated incontests and posted in forums was much smaller. The majority of community members at TopCoderregistered as members of the community but never competed in any contests. In fact, by 2009, only35,000 unique individuals had competed in contests. To Mik???? ?????????????? ?????????????????????? chief technologyo???????????????? ???????? ???????????????????? ???????????? ?????? ???????? ???????????????????? ???????? ???????? ???????????????? ?????????????? ?????????????? ???????? ?????????? ??????????????????????For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .TopCoder (A): Developing Software through Crowdsourcing 610-­0325enough in the TopCoder platform to register and had the potential to provide TopCoder withincreased development under the right conditions.A second group within the TopCoder community comprised those members who at one timeparticipated in TopCoder contests but then stopped participating. Lydon noted that, after TopCoderdecreased prize values in 2008, many competitors from the United States and Canada left theTopCoder community. Yet another group included people who participated in TopCoder contestsbut did not win. TopCoder saw those competitors as the ???????????? ????????????people who primarilycompeted for the sake of learning????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????less-­skilled competitors could improve over time and increase their levels of contribution. Lastly,?????????????????????? ?????????? ?????????????????? ???????????? ?????? ???????????????????????? ?????????????????? ???????? ?????????????????? ?????????????????? ???????? ?????????????? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????for 0.5% of the total TopCoder population.The ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? in their 20s.According to Michael Paweska, a six-­???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ?????? ???????? ???????? ?????????? ?????????? ???????? ???????????????????????? ?????? ?????????? ?????????? ?????????????? ?????????????????? ?????? ???? ?????????????????????? ?????????????? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????TopCoder attractedcompetitors from developed nations such as the United States, Canada, South Korea, and Japan, aswell as from emerging economies such as China, Russia, Poland, India, and Ukraine. Wu Yanbo, aChinese TopCoder community member studying abroad in Australia, explained that mostcompetitors in the lower-­paid contests were from developing countries. According to Wu, the prizeswere not large enough for many individuals from developed countries to compete, since they couldspend their time better elsewhere.Justin Gasper, a member since 2001, began experimenting with the TopCoder platform whileworking for a traditional software engineering company. After winning significant money withTopCoder, Gasper decided to quit his job in 2005 and devote 40 to 50 hours a week to TopCoder.Gasper explained: ???????????????????? ?????? ?????? ????????-­?????????? ?????????? ???? ???????????? ?????????? ???? ???????? ?????????? Gasper was one of?????????????????????? ???????????????? ?????????????????? ???? ?????????????? ?????? ???????? ???????????????? ?????????????? ?????? ?????????????????????????? ?????? ??????????????????????????competitions, Gasper won at least second place 95% of the time and had a win percentage of 69.23%.Competitors at TopCoder could choose which contests and what type of contests to join (seeExhibit 5 for participation and prize data by contest type).Profiles and RatingsEach programmer in the TopCoder community maintained a public profile that displayed his orher user name, contest history, and basic personal information. Another part of the member profile??????????????????????????????????????????????????numeric rating for each type of contest. The rating system was modeled onthe one used to rank grandmaster chess players engaged in worldwide competition. A ?????????? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? within the community and a high skill level.Yellow, blue, and green color ratings represented descending skill levels. ????????????????????????????????????countryrank, total community rank, success rates for contests, and reliability??or percentage of times thecontestant joined a competition and submitted a passing solution??were featured in their profiles.TopCoder members could also choose whether or not to display their total earnings on their profiles(see Exhibit 6 for an example member profile).For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .610-­032 TopCoder (A): Developing Software through Crowdsourcing6Motivating MembersBetween 2001 and 2009, TopCoder paid out over $20 million in prizes and peer review money toits community of developers. However, prize money was not evenly distributed throughout theTopCoder community. The top 5% of prize earners received approximately 80% of the total prizepool, while the majority of TopCoder community members earned little or no money fromcompetitions. Some competitors were extremely successful. For example, from 2006 to 2008, Paweskaearned $200,000 to $300,000 per year, while Gasper averaged over $100,000 annually. Wucommented: ??????????????????????????????????????????is the most attractive thing. The prize is very good compared to theincome of my friends who are working in some local companies in China. Even though the economyis not very good and TopCoder reduced its prizes, I can still earn around $1000 per month in my??????????????????????????TopCoder typically awarded prizes to the top two submissions in each contest, with the??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????Besides prizes awarded on a contest-­by-­contest basis, another main source of income for memberswas the Digital Run. In the Digital Run system, the top five ranked competitors for each contest wereawarded points based on contest rank and performance. At the end of each month, TopCoder tallied?????????????????????????? ???????????? ?????????????? ???????? ???????????????? ???????? ???????? ???????????? ??arners thousands of dollars in bonus prizes.Paweska explained that success in the Digital Run was not all about who was the best programmerbut more about who could handle the most all-­nighters. Other competitors, such as Gasper, alsomade money through contracted projects that TopCoder assigned.In addition to their cash earnings, many community members reported that their TopCoder ratingwas very important because it provided an objective assessment of ability. Wu commented that it wasnot easy to maintain a very high rating as it required familiarity with many kinds of technologies,quick thinking, the ability to learn independently, a strong work ethic, and attention to detail.According to W?????? ???? ?????????????????? ?????????????? ???????????? ?????? ???????????????????? ???????? ???? ?????????????????????????? ?????????????? ???????????????? ???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? Gasper noted thatTopCoder ratings were also symbols of status and prestige for many programmers: ??If you have red???????????????????????????????????????????????????????????????? Indeed, many prestigious software firms asked potential recruits toget a TopCoder rating before applying for a job. To others, however, the rating system was lessimportant. Gasper, for example, explained that winning and making money meant more to him thanratings.Although there were differences of opinion regarding the importance of ratings, almost allcommunity members agreed that competing at TopCoder provided numerous opportunities to learnand improve. In fact, for many programmers, a TopCoder career often began with failure, but post-­contest evaluation and peer review of each submission helped them grow and improve. Gaspernoted: ?????????????????????????????????? in my first competition. But the reviewers were really good at pointing me inthe right direction???????????????????????????????????????????????????????????????????????????? ?? ??????????????????????????????????????????????????????????????????????????????????peer reviewed by people who are better at programming ????????????????????????????????????????????????????????????????????????????????????they hurt your feelings;; they are ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????that getting feedback from reviewers was crucial and added that community members could alsolearn from acting as a reviewer for contests. For scientists and developers, Wu believed thatalgorithm contests were particularly helpful at sharpening research skills and improving criticalthinking abilities. In all cases, continual learning opportunities from peers were an important reasonfor participation.Gasper described the appeal of working at home on a web-­based platform instead of in a?????????????????????????????????????????????????????????????????????? ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .TopCoder (A): Developing Software through Crowdsourcing 610-­0327a half an hour to work each day and can do the same work at home. If I want to take off a day to playgolf, I just do i?????? ???? ?????????? ???????????? ?????????? ?????? ???????? from 2:00 to 6:00 p.m.???? ?????? ?????????? ?????????????????????????? ??????????????Sharing similar sentiments, Paweska liked that while working at TopCoder he did not have asupervisor looking over his shoulder.???????????????????????????????????????? ?????? convenient but also challenging, as competitors had to actively managetheir individual levels of participation. Gasper constantly balanced effort and reward to maximize?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? too much work for???????????????????????????????????????????????????? . . . ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????worthwhile to solve???????????????? skill that ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????Wu noted that, although the firm was competitive in sprit, competition at TopCoder was neverdisrespectful or nasty and that people liked to help each other, even when they competed in the samearena. TopCoder forums were the main source for collaboration. In the forums, less-­experiencedcommunity members asked for assistance on certain problems and received instant feedback frommore-­experienced competitors.At TopCoder, conversations and relationships extended beyond the scope of softwaredevelopment. Hughes reflected on a particularly remarkable exhibit of communal strength and?????????????? ???????? ?????????????? ???????????????????? ???????????????? ???????????????? ?????? ?????????????????? ?????????????????????????? ???????????? ???????? ?????? ???????????????????????????? ???????????????? ?????????????? ?????? ???????????? ?????????? ?????????????????????? ?????? ???????????????? ???????? ?????????? ?????????? ???? ?????????????? ?????? ????????community members took ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????Once a year, TopCoder paid for all of the best talent from the community to travel to Las Vegas,Nevada, to compete in the TopCoder Open (TCO). In addition to serving as a proving ground for thebest programmers in the world, the TCO provided community members with the opportunity tonetwork professionally and socially.The TopCoder community had a distinctive culture, with identifiable personalities. Wu explained:?????? ???????????????? ?????????? ?????????????????????? ?????????? ???????? ???????????????? ???????? ???????? ???????? ?????????????????? Clearly, the members built it upcontinuously. When I joined the community, there were already some leading members who wereactive in competitions and forums, brought out good suggestions, and started up interesting and???????????????????? ???????????????????????????? ?????? ?????????? ?????????????? ???????? ?????????? ?????? ???????????????????? ???????????????? ?????????????????? well beyondTopCoder. For example, Tomasz Czajka??????????????????????????????????????????????????rock star???????????????????????????????????????? pictureplastered on billboards throughout Warsaw after he won the TopCoder Open in 2006.????????????????????????????????????????????????Clients came to TopCoder to have high-­quality software developed in a cost-­effective and time-­efficient manner. TopCoder positioned itself to serve both large firms and medium-­ to small-­sizedbusiness that wanted to see systems developed. Keith Moore, a TopCoder client and former seniorvice president at LendingTree.com, believed that, regardless of size, any company could takeadvantage of TopCoder, whether it was a five-­person operation or large outsourcing vendor. Formany CIOs, the process of software development and talent recruitment was a major headache, andmissed deadlines and large cost overruns were common worries. According to Stephen Laster, the???????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? over 48% of itsemployees every three years. This process is very costly. The same problem exists with ourFor the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .610-­032 TopCoder (A): Developing Software through Crowdsourcing8outsourcing consultants. When selecting consultant teams, we tried out 60 programmers beforefinding our team of 20. With TopCoder, I pay for performance and the CIO sees Nirvan????????As of 2009, TopCoder had developed a strong relationship with existing clients for deliveringhigh-­quality software solutions and superior customer service. After completing their first projectwith TopCoder, 82% of clients signed up for a second round of contests. These clients cited severaladvantages.BenefitsBetter Ideas Before sinking thousands of dollars into a project, a client could run aconceptualization contest through which TopCoder members helped identify bad ideas and generatebetter approaches early in the development cycle. When the client introduced a business problem tothe community, members asked hundreds of questions. Nic Perez, a former technical director atAOL, ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ???? ?????????? ???????? ?????????????? ?????? ???????????? ?????? ???????????? ???????? ?????? ???????????? ???????????????????? ???????????????????? ???????????? ?????????????? ????????????????clients answered questions for all competitors only once, avoiding repeated efforts. In some cases,clients scrapped product ideas entirely after the community raised concern???? ???????????? ???????? ????????????????????likely success or usability in the marketplace.?????????????????????? ??????????????-­based development system consistently produced highly creative ideas andsolutions. According to Darren Smith, a solution architect for the e-­commerce division at FergusonEnterprises, ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????, ??????????????????????????????comes back with many options. It really has surprised us. You never know what you are you going toget. The creative side allows us to go to the marketing management team and say, ????e could do X, Y,and Z that we may not have previously considered.?? ?????????????? adding value to our business becausethey bring us solutions that quite frankly we may not have considered or were not resourced todeliver??????Superior Quality, Cost, Speed, and Flexibility ???????????????? ???????????????? ?????????????????????? ??????????????????evaluation and documentation process for being well above industry standards. Reflecting on hisexperience working on the Google Talk interface to AOL Instant Messenger, Perez stated thatTopCoder and its community had a strong desire to deliver bug-­free code and that even the mostcomplex systems always had fewer than 100 identified bugs. According to Perez, the same sizedprojects, developed internally, at AOL would have had five to eight times that number of bugs.Another TopCoder client, a Web-­based startup business, noted that it would have had to pay$350,000 to a large IT consulting firm, $200,000 to a small IT consulting firm, or $80,000 to individualcontractors ????????????????????????????????????????????????????????????. Using TopCoder, the client only spent $35,000. This sameclient proclaimed: ???????? ?????????????????? ?????????? ?????????????????????? ???????????? ?????? ?????? ???????????? ?????????? ???????? ???????????????? A different clientnoted that based on its experience working with almost every type of software developmentcompany, TopCoder charged approximately half of the fee of a large, tier-­one IT consulting firm.Using the community for parallel problem solving, TopCoder marketed itself as faster thanother software development shops. This was true for back-­end bug races and system checks, asTopCoder took 72 hours to complete the same bug testing that a traditional development firmfinished in 10 business days. However, for other steps in the software development process, reportson speed were mixed. Some clients said that TopCoder worked at about the same speed as a large ITconsulting firm, while others lauded TopCoder for speed of completion.For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .TopCoder (A): Developing Software through Crowdsourcing 610-­0329Especially appea?????????? ?????? ???????????????? ???????? ?????????????????????? ???????????????? ?????? ?????????????? ?????????????????? ?????????????????? ????????????????????????capacity. In particular, a TopCoder client could expand or reduce its business requirements anddevelopment capabilities without having to hire or fire programmers. According to one client, a basicin-­house computer programmer cost $120,000 a year, after accounting for benefits, sick time, andvacation. Working with TopCoder??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????downtime.ConcernsAlthough CIOs were im???????????????? ?????? ?????????????????????? ???????????????????? ?????????????????????????? ???????? ????????-­saving potential,??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????model.Intellectual Property (IP) and Security According to Ira Heffan, ???????????????????? chief legalc???????????????? ?????????? ???????? ???????????????? ?????????????????????? ?????????? ?????????????????????? ?????????????? ?????? ???????? ?????????????????? ?????????????????? can be aninitial point of resistance. Until they understand the documentation and processes we have in placewith the community members, they see IP and security as potential barriers to working with a???????????????????????? ???????? ?????????????????? ?????????? ???????????????? ?????????? ?????????????????????????? ?????????? ???? ?????????????????? ???????????????????? ??????????????might divulge proprietary ideas, business plans, or operations to their competitors. In addition, someclients worried that once a component became an integral part of their IT systems, the communitymember who built the component might attempt to prohibit its use or ask the client to payconsiderable royalties. Lastly, some clients were concerned that a solution submitted by a communitymember could be stolen, copyrighted, or taken from open-­source software projects, thus potentiallyopening the door for intellectual property disputes.TopCoder had in place a number of initiatives targeted at addressing these concerns and reducingthe risk level for clients, and also took steps to communicate its processes. To ease clients?? intellectualproperty and security concerns, TopCoder produced a white paper that detailed confidentiality???????????????????? ?????????????????????????? ?????????????????? ?????????????????????? ?????????????? ???????? ?????????????????????? ???????????????? ?????????????????? ?????? ??????????????????development. In addition, TopCoder allowed clients to keep their company names anonymousduring competitions and helped clients generate test data sets to avoid the exposure of sensitive?????????????????????????? ?????? ???????? ?????????????????? ?????????????????? ?????????????? ???? ???????????????????? ?????????????? ???????? ???????????????? ???????????? ?????????? ????competition, all competitors could be required to sign a standard competition confidentialityagreement.The peer-­review process was another means to ensure code security and quality. Peer reviewerswere selected and vetted by TopCoder employees based on their superior performance on priorcompetitions. TopCoder clients also had the option of running testing competitions at the back end ofsoftware production, serving as an additional means of checking code security and quality.?????????????????????? ???????????????????????????????????? ?????????????????? ???????????????????????? ???????????????? ??????o made it difficult for a singlecompetitor to insert harmful code into a program, since individual contests only addressed one smallpiece of the overall program.Cultural Change Many clients realized that working with TopCoder would be difficultculturally for their company. In particular, CIOs believed that internal employees would viewTopCoder as a threat to their job security. One new client observed: ?????????????????????????????????????????????????????????????????? ?????????????????????????? ???????????? ?????????????????????? I fully expect that if this goes well and if my programmers seegood quality work coming out of TopCoder, fear will ???????????????? ???????????????? ?????????????????????? ???????? ????????????????????Although using TopCoder could help a company scale and reduce the programming staff costs,companies still had to reta?????? ???????? ?????????? ???????????????????? ??????????????the employees who could guide theFor the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .610-­032 TopCoder (A): Developing Software through Crowdsourcing10TopCoder development process. The managers at TopCoder clients also had to adjust to a perceivedloss of control over the software development process. Smith ??????????????????????????????????????????????????????????????????????,but they manage the whole process. Our project management group works with the TopCodermanager to ensure delivery according to pre-­determined service level agreements (SLAs)??????Additionally, some clients found a few community members to be pushy and rude during pre-­competition question-­and-­answer sessions.Coding Challenges ?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ???????????? ?????????????? ?????? ?????????? ?????? ?????????????????? ???????????? ?????? ???????????????????? ???????? ???????????????????????? ?????????? ???????? ?????????????????? ??xistingsystems, review the code for security issues, and adjust and fix code as systems changed over time.???????? ?????????????????? ???????????????? ?????????? ?????? ?????????????????? ???????????? ???? ???????????????????????? ?????????????? ?????? ?????????? ???????????????????????? ?????????????????? ?????????????????????????????? ?????????????????????? ?????????? ?????? ?????????? ?????????? ?????????? ?????????? were no security threats or bugs. In somecontests, TopCoder clients also spent time evaluating ideas and approaches from multiple winningsolutions.Another ongoing issue for clients was finding the right types of problems and providing theappropriate a???????????? ?????? ???????????????? ?????????????? ???????? ???????? ?????????????????? ?????????????????????? ?????? ???????????? ???????????????????? ???????? ??????????want neither too much nor too little detail. You do not want to quell innovation but also want a??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????icipation decreased ifthey were unclear about what problems they wanted to solve or presented problems that were toocomplex or vast in scope;; in those cases, the TopCoder community struggled to produce anacceptable solution. Clients also found that community members worked best when contests lastedless than two weeks. If projects took too long to complete, contestants would lose interest and notmake submissions.Managing TopCoderThe Supply SideA management job at TopCoder was unique. Along with supervising internal TopCoderemployees, managers at the firm had to oversee a community of over 200,000 members and direct theprocess of competition-­based software development. According to senior vice president GeorgeTsipolitis, the key to success was effective process management: ???????????? ?????????????? ?????????????????? ?????????????????????????? ???????? ???????? ?????? ?????????????? ?????????????????? ?????????????????????????? ?????????????? ?????????????????? ???? ?????????????? We ???????????? ????????????????individuals. We can only control the process of their participation.???? ???????????????? ???????? employees alikebelieved that the sustainable value of the ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????community participation and foster community growth. Lydon described the risks: ??From thebeginning, we focused on the community. We knew they could be unforgiving. If you did the wrongthing, ????????????????????????????????????????Attraction To run many competitions simultaneously and produce solutions for many clientsat the same time, TopCoder needed to have access to a critical mass of talent and coding capacity.?????????????????????? ???????????????? ???????????? ?????? ?????????????????????? ???????? ???????????????? ?????????? ???????? ???????????????????? ???????? ???????? ?????????????? ????????challenge of the algorithm contests. In addition, TopCoder occasionally advertised its onlinecompetition platform by paying for Google keyword searches using terms such as ??????????????????????????????????????A third mechanism for attracting talent ???????? ???????????????? ???????????????????????? ?????????????? ???????????????????? ?????? ???? ????????????team of TopCoder employees, member development days were held at Chinese and otherinternational universities. At a member development day, a student representative would post signsaround the school and explain the TopCoder system. A primary goal of these member developmentFor the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .TopCoder (A): Developing Software through Crowdsourcing 610-­03211days was to encourage participation in the higher-­revenue-­producing development and designcontests. During one member development day in China, TopCoder registered over one thousandnew community members. Bourdon noted that TopCoder had achieved critical mass once it crossedthe 200,000 member threshold, as there were now many members with deep and narrow skills over arange of software development challenges (See Exhibit 7 for the number of participants by contesttype).Norms As the community grew, TopCoder paid close attention to establishing communitynorms. As contest administrator, the company had to maintain the highest standards of contestintegrity, fairness, transparency, and quality. For example, TopCoder personnel strictly monitoredcompetitions and tolerated no form of cheating. Community members who peeked at other?????????????????????????? ??olutions, shared ideas during competition, or used unauthorized code wereimmediately eliminated from the contest. Often they were kicked out of the community entirely.If any uncertainty or disagreement arose about which competitor won a particular contest,TopCoder would spend extra money to re-­run the competition. Another part of contest integrity,Tsipolitis explained, was TopCoder???? emphasis on maintaining consistency of rules and procedures:?????????? ?????????????? that participants ???????????? ?????????????? ???????? ???????? ?????? ?????????? ???????????????? ?????????? ?????????????????????????????? ?????? ?????? ????????????change the rules of a competition mid-­?????????????????? TopCoder also guaranteed complete contesttransparency by storing all contest and competitor statistics, peer reviews, and solutions in a datawarehouse. The data were publicly available to registered community members, accessed via theTopCoder website.???????????????????? ???????? ?????????????????? ?????????? ?????????????????? ?????? ?????????????????????? ???????????????????? ???????????????????????? ???????? ????????????????????compensation philosophy. In particular, TopCoder was up-­front with the community over itsintention to make money. When TopCoder made a decision to change corporate direction orcompetition procedures, Hughes posted the information in the forums and explained the businessreasons behind his decisions. Hughes also believed that, since the company benefited from the???????????????????????????????????????????????????????????????????????????????????????????? community members was essential.Governance Although TopCoder executives were responsible for final decisions, theyfrequently incorpo?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? ????????????????????????????community as the driver for everything we do. If we have enough dissent from members, we always????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????embers will also be????????????????Community member Gasper shared a similar perspective: ????????????????????????????????????????????????????????????????????into the forums to get feedback. Seventy-­five percent of the time, they listen to the community. ButTopCoder also has its own business interests to consider. Sometimes the community and business???????????????????????????????????????? ??????????Similarly, if competitors were unhappy with a peer-­review scoring outcome, TopCoder allowedthem to appeal the decision. Over 90% of contests featured at least one appeal. If a member appealed,peer reviewers had to provide specific reasons why the appeal was accepted or rejected. Ifdisagreement remained between contestant and reviewer, TopCoder employees often investigated.Contestants could also appeal directly and privately to TopCoder personnel or post complaintspublicly on the TopCoder forums.TopCoder managers inevitably made decisions that sometimes disturbed and upset the TopCodercommunity. For example, facing a very difficult economic environment in the summer and fall of2008, TopCoder reduced the contest prize amounts, cut payments to peer reviewers, and reduced thenumber of algorithm competitions. During this period, some TopCoder competitors left thecommunity entirely and others dramatically reduced their participation levels. Gasper argued thatFor the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .610-­032 TopCoder (A): Developing Software through Crowdsourcing12the payment cuts also led to many superficial reviews because the best reviewers were no longerdoing the work, which then required additional cycles to achieve acceptable quality.Resource Allocation Another part of the TopCoder managerial role was allocating??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????how to distribute the number of people who want to participate across the number of contests that?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? the costs ofevaluation, stimulate effort through competition, and get at least one solution that was acceptable tothe client. To achieve the ideal number of submissions and participants, TopCoder adjusted the prizeamount, the duration and timing of the contest, the number of other contests running concurrently,???????? ???????? ???????????????????? ?????????????????????? ???????? ?????????????? ?????????? ?????????????????? ?????? ???????? ???????????? ?????????????? ?????? ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????directly to individual community members if other methods did not lead to the desired participationlevels.Although TopCoder managers could pull many levers to influence contest participation, theybelieved it was important not to act like ???????? ?????????????????????? boss. Hughes explained his communitymanagement philosophy???? ???????? ???????????? ???????? ?????????? ?????????????????????? ?????? ?????????? ?????????????? ?????? ?????? ?????????? ?????????? ??????????want to be here. You are just going to get much better results when you let people do what they??????????????????????????????????????Retention At the same time, TopCoder executives worked to retain community members andencourage future contest participation. At least one client raised concerns in this area: ??I think thatcommunities are fickle. Community members could start to ask, why do I need them? For example,what happens if an imitator comes along and offers twice ???????? ???????????? ???????????????? To avoid suchproblems, TopCoder tried to supply community members with consistent work streams and prizemoney. TopCoder also encouraged community members to engage in the community as much aspossible by dedicating significant resources to facilitating forum discussions and inviting contestants????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????TopCoder community members differed on their level of loyalty to the TopCoder community.?????????????????? ?????????????????????????? ?????? ???????? ???????? ???????????????????????????? ?????????????????? ???????? ???????????????????? ?????????????????????? ?????? ?????????? ??????????loyalty. I think it would take a lot for me to leave. Only if there were no projects would I leave????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????[For me] to defect, the payment and work would have to outweigh the paymentand flexibility I have at Top????????????????The Demand SidePlatform Managers The other side of management at TopCoder was guiding clients throughthe contest-­based software development process. ?????????? ???????? ???????? ?????????????????????????????? ?????? ???????? ??????????????????platform managers, whose job was to induce the appropriate amount of community participation,make suggestions for contest prize amounts, gather feedback between contests, and provide projectstatus updates to clients. Before starting the next step in the game plan, platform managers alsoadjusted contest requirements based on the work already completed. Once the product was deliveredto the client, TopCoder platform managers were required to act in a support and service role. If therewas a technical problem with a solution, the platform manager often contacted the communitymembers who developed the component and worked with the community members to fix the issue.For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .TopCoder (A): Developing Software through Crowdsourcing 610-­03213Most enterprise-­level clients believed the platform manager was pivotal to ????????????????????????????????????????????????????large client like LendingTree, the platform manager was on site three to four days a week, conductingdaily meetings with the internal teams. ???? ???????????? ???????????????? ???????? ?????????????????? ???????????????????? ?????????? ???????? ??????????????????client expectations and serving as a sounding board for client concerns. At the back end of projects,?????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????? the platformmanager was also an expert at combining the small software pieces. The component integration rolesaved the client hours of work trying to figure out how all the pieces fit together. At Ferguson, Smithconsidered the TopCoder personnel working on site to be an integral part of his team.TopCoder Direct However, each ????????????????????????????????????????????????????????????????????????????????????????????????????????????????narrowed profit margins. As of 2009, a typical platform manager at TopCoder cost $100,000 a yearincluding ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????To avoid a potentially large increase in expenses as TopCoder added clients and projects, Hughes?????????? ?????? ?????????? ???????? ???????????????? ?????? ???????????????????? ?????????????????? in which the client used the ?????????????????? platformwith little to no intervention from its employees. Under this self-­service model, platform managerswould educate clients on how to use the TopCoder platform to manage the contest-­based softwaredevelopment process themselves. ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????to an experienced community member or an external ??????????????????????????????????????????????????????????????????????????????????????????someone who would serve as a ??????-­???????????? to assist the client staff. With co-­pilots taking the role ofplatform managers, ????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????ect wouldshrink from 40 hours to two, thus saving the client and TopCoder considerable time and money.The FutureAs of December 2009, no competitors had elected to copy ????????????????????????????????????????????????????by offeringfull-­service software development through a competition-­based approach. Instead, companies suchas RentACoder, Elance, and oDesk served as online liaisons between clients and freelance softwaredevelopers. Unlike TopCoder, whose clients only paid for solutions, clients at these firms used a ?????????????????????? approach: they selected one or more programmers to solve their problem. More similar toTopCoder, uTest used crowdsourcing to find bugs and check the functional usability of web, mobile,desktop, and gaming applications, but did not engage in software development. According toHughes, this lack of direct competition reflected the technical difficulties and costs associated withbuilding a full-­fledged community and platform.After a significant downturn in the global economy in 2008 and 2009, Hughes believed thatTopCoder was primed for growth. Sales staff were forecasting aggressive targets for the volume ofcompetitions and revenues, and several strategic partnerships were under consideration. However,significant challenges and uncertainty remained. In particular, Hughes wondered whether thecommunity, as well as the company, could grow to meet increasing demand.Stakeholders had divergent views. Mike Morris, vice president of sales, saw unlimited potential:????f sales grow at a linear rate, membership grows at an exponential rate. The supply of communitymembers is not going to limit growth. If you throw enough money out there, you will get enough?????????????????????????? Community members Paweska and Wu agreed that offering more money per contestwould increase participation among existing members. Paweska also believed, however, that holdingmany more contests than usual in a given week would result in inexperienced competitors competingactively for the prizes, possibly reducing code quality. Furthermore, Lydon noted that that during??????????????????????????????????????????????????????-­up in 2007, review quality suffered during a transitional period of a fewFor the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .610-­032 TopCoder (A): Developing Software through Crowdsourcing14months. As more contests became available, the usual reviewers wanted to compete in the contests,rather than review them, leaving TopCoder scrambling to find replacements. In addition, a fewclients worried that as the number of avenues of competition at TopCoder grew, attracting the samegroup of competitors would prove much more difficult, reducing contest consistency and continuity,which were especially critical for addressing legacy systems.Hughes also worried about client service. If the number of TopCoder clients expandedsignificantly, TopCoder???? staff might face increasing difficulties responding to all ???????????? ??????????????????questions and concerns. For large clients, expansion might require adding more platform managers,??????????????????????????????????????????????????????COO, was concerned that too many platform managers might make thefirm appear to be like any other large IT consulting company, with the risk of losing its uniquebusiness model.????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????position. In particular, Hughes wondered if community members would stick with TopCoder if anew competition-­based software development company emerged. What would happen if a companylike Accenture started to develop software in the same way as TopCoder? Would the TopCodercommunity remain intact?For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .610-­032 -­15-­Exhibit 1 TopCoder Members Involved in Creating a Power Pricing System for an Energy CompanySource: Company documents;; developed via a TopCoder Studio competition.For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .610-­032 -­16-­Exhibit 2 Number of Clients, Revenue, Number of Platform Managers, and Platform Manager Costs by Quarter2007Q1Q2Q3Q42008Q1Q2Q3Q42009Q1Q2Q3Q4Number of ClientsTotal Revenue($MM)Number ofPlatformManagersCost of PlatformManagers ($MM)a324.66511.23324.50511.24343.80441.12255.35521.30245.80521.36385.50461.19374.85441.13472.60360.91462.25200.53471.92190.49381.82180.46352.45160.40Source: Company statistics.a Includes platform managers?? salaries, benefits, and other expenses.For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .610-­032 -­17-­Exhibit 3 Sample Game PlanSource: Company documents.Phase# ofContestsEstimatedCostsMT WT F MT WT F MT WT F MT WT F MT WT F MT WT F MT WT F MT WT F MT WT F MT WT F MT WT F MT WT F MT WT F MT WT F MT WT F MT WT FConceptualizationLogo – Tournament 0 $ –Concept Contest – Tournament x x x x x 1 $ 4,500.00Wireframes – Tournament x x x x x 1 $ 3,000.00Storyboard – Tournament x x x x x x x x x 1 $ 3,500.00Application BuildSystem ArchitectureSystem Architecture x x x x x x x x x x 1 $ 4,500.00Component Design x x x x x x x 1 $ 2,300.00Component Development x x x x x x x x 1 $ 2,050.00Catalog Components 0 $ –System Assembly x x x x x x x x x x 1 $ 4,900.00 ModuleModule Specification x x x x x x x x x x 1 $ 4,000.00Module Architecture x x x x x x x x x x 1 $ 4,000.00Component Design x x x x x x x x 3 $ 6,900.00Component Development x x x x x x x x x 3 $ 6,600.00Catalog Components 0 $ –Module Assembly x x x x x x x x x 1 $ 4,900.00Prototype Assembly x x x x x x x x x x 1 $ 4,900.00TestingTest Scenarios x x x x x x x x x x 2 $ 3,900.00Test Cases x x x x x x x x x x 2 $ 3,900.00Bug Hunt x x x x x x x x 1 $ 2,750.00DeploymentDeployment x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x 20 $ 4,000.00UpdatesBug Races x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x 30 $ 4,200.00Total $ 74,800.00Timeline4/27 5/4 5/11 5/18 5/25 6/1 6/8 6/15 6/22 6/29 7/6 7/13 7/20 7/27 8/3 8/10For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .610-­032 TopCoder (A): Developing Software through Crowdsourcing18Exhibit 4 Community GrowthSource: Company statistics.Exhibit 5 Average Contest Registration, Submission, and Prize Amount for Client Contests in 2008and 2009Contest TypeNumber ofRegistrantsper Contest2008Submissionsper ContestPrizeAmount perContestaNumber ofRegistrantsper Contest2009Submissionsper ContestPrizeAmount perContestaConceptualizationSpecificationArchitectureComponentDesignComponentDevelopmentAssemblyStudion/abn/ab16.39.8315.416.0927.55n/abn/ab1.642.852.691.1214.57n/abn/ab$1,590$899$733$1,628$79517.5714.2019.3416.2625.5918.3827.573.601.941.751.942.561.1820.04$1,314$1,017$1,095$559$465$913$1,015Source: Company statistics.a Prize per contest ?? Prize for first and second places and reserve for Digital Run.b n/a ?? Data not available for most of 2008.050,000100,000150,000200,000250,000TotalCommunityMembersFor the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .TopCoder (A): Developing Software through Crowdsourcing 610-­03219Exhibit 6 Example Community ProfileSource: http://www.topcoder.com/tc?module=MemberProfile&cr=287614, accessed December 23, 2009.For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .610-­032 -­20-­Exhibit 7 Number of Unique Participants by Contest Type per Year and Month/Total Number of Official Contests per Year2005 2006 2007 2008 2009Average Submitters Contests Average Submitters Contests Average Submitters Contests Average Submitters Contests Average Submitters ContestsContest Type per Month per Year per Year per Month per Year per Year per Month per Year per Year per Month per Year per Year per Month per Year per YearAlgorithm ContestsSingle Rounda 1,319 5,287 81 1,930 7,525 105 2,638 8,994 113 2,945 10,433 66 2,558 9,616 51Marathon Matchb 500c 500c 1c 273 1,532 20 281 1,588 30 253 1,621 29 314 2,150 35Client SoftwareDevelopment ContestsConceptualization 7c 11c 9c 6 34 70Specification 5c 7c 16c 6 32 71Architecture 1c 4c 4c 4 29 65 9 36 145Component Design 33 157 362 45 225 615 61 243 698 38 144 488 20 93 300Component Dev. 58 316 287 88 451 484 135 605 780 91 434 733 41 204 337Assembly 7 47 86 10 59 191 20 104 416Design ContestsStudio 65c 223c 17c 81 453 118 66 279 451 102 429 456Totald 1,370 5,565 730 2,146 8,517 1,224 2,867 10,072 1,825 3,198 11,487 2,023 2,911 11,122 1,881Source: Company statistics.a 75-­minute programming contest.b Programming contests that run from 3??30 days.c Partial year data??contest track officially did not officially start until the middle or end of the year.d Represents the unique number of participants during a given time period. Columns are not additive.For the exclusive use of S. Weisband, .This document is authorized for use only by Suzanne Weisband in .

Type of paper	Academic level	Subject area
Essay Term Paper Research Paper Coursework Book Report Book Review Movie Review Dissertation Thesis Thesis Proposal Research Proposal Dissertation Chapter - Abstract Dissertation Chapter - Introduction Chapter Dissertation Chapter - Literature Review Dissertation Chapter - Methodology Dissertation Chapter - Results Dissertation Chapter - Discussion Dissertation Services - Editing Dissertation Services - Proofreading Formatting Admission Services - Admission Essay Admission Services - Scholarship Essay Admission Services - Personal Statement Admission Services - Editing Editing Proofreading Case Study Lab Report Speech Presentation Math Problem Article Article Critique Annotated Bibliography Reaction Paper PowerPoint Presentation Statistics Project Multiple Choice Questions (None-Time-Framed) Other (Not listed)	High School Undergraduate Master Ph. D.	Art   Architecture   Dance   Design Analysis   Drama   Movies   Music   Paintings   Theatre Biology Business Chemistry Communications and Media   Advertising   Communication Strategies   Journalism   Public Relations Creative writing Economics   Accounting   Case Study   Company Analysis   E-Commerce   Finance   Investment   Logistics   Trade Education   Application Essay   Education Theories   Pedagogy   Teacher's Career Engineering English Ethics History   African-American Studies   American History   Asian Studies   Canadian Studies   East European Studies   Holocaust   Latin-American Studies   Native-American Studies   West European Studies Law   Criminology   Legal Issues Linguistics Literature   American Literature   Antique Literature   Asian Literature   English Literature   Shakespeare Studies Management Marketing Mathematics Medicine and Health   Alternative Medicine   Healthcare   Nursing   Nutrition   Pharmacology   Sport Nature   Agricultural Studies   Anthropology   Astronomy   Environmental Issues   Geography   Geology Philosophy Physics Political Science Psychology Religion and Theology Sociology Technology   Aeronautics   Aviation   Computer Science   Internet   IT Management   Web Design Tourism
Number of pages	Paper urgency	Cost per page:
1 pages/275 words 2 pages/550 words 3 pages/825 words 4 pages/1100 words 5 pages/1375 words 6 pages/1650 words 7 pages/1925 words 8 pages/2200 words 9 pages/2475 words 10 pages/2750 words 11 pages/3025 words 12 pages/3300 words 13 pages/3575 words 14 pages/3850 words 15 pages/4125 words 16 pages/4400 words 17 pages/4675 words 18 pages/4950 words 19 pages/5225 words 20 pages/5500 words 21 pages/5775 words 22 pages/6050 words 23 pages/6325 words 24 pages/6600 words 25 pages/6875 words 26 pages/7150 words 27 pages/7425 words 28 pages/7700 words 29 pages/7975 words 30 pages/8250 words 31 pages/8525 words 32 pages/8800 words 33 pages/9075 words 34 pages/9350 words 35 pages/9625 words 36 pages/9900 words 37 pages/10175 words 38 pages/10450 words 39 pages/10725 words 40 pages/11000 words 41 pages/11275 words 42 pages/11550 words 43 pages/11825 words 44 pages/12100 words 45 pages/12375 words 46 pages/12650 words 47 pages/12925 words 48 pages/13200 words 49 pages/13475 words 50 pages/13750 words 51 pages/14025 words 52 pages/14300 words 53 pages/14575 words 54 pages/14850 words 55 pages/15125 words 56 pages/15400 words 57 pages/15675 words 58 pages/15950 words 59 pages/16225 words 60 pages/16500 words 61 pages/16775 words 62 pages/17050 words 63 pages/17325 words 64 pages/17600 words 65 pages/17875 words 66 pages/18150 words 67 pages/18425 words 68 pages/18700 words 69 pages/18975 words 70 pages/19250 words 71 pages/19525 words 72 pages/19800 words 73 pages/20075 words 74 pages/20350 words 75 pages/20625 words 76 pages/20900 words 77 pages/21175 words 78 pages/21450 words 79 pages/21725 words 80 pages/22000 words 81 pages/22275 words 82 pages/22550 words 83 pages/22825 words 84 pages/23100 words 85 pages/23375 words 86 pages/23650 words 87 pages/23925 words 88 pages/24200 words 89 pages/24475 words 90 pages/24750 words 91 pages/25025 words 92 pages/25300 words 93 pages/25575 words 94 pages/25850 words 95 pages/26125 words 96 pages/26400 words 97 pages/26675 words 98 pages/26950 words 99 pages/27225 words 100 pages/27500 words 101 pages/27775 words 102 pages/28050 words 103 pages/28325 words 104 pages/28600 words 105 pages/28875 words 106 pages/29150 words 107 pages/29425 words 108 pages/29700 words 109 pages/29975 words 110 pages/30250 words 111 pages/30525 words 112 pages/30800 words 113 pages/31075 words 114 pages/31350 words 115 pages/31625 words 116 pages/31900 words 117 pages/32175 words 118 pages/32450 words 119 pages/32725 words 120 pages/33000 words 121 pages/33275 words 122 pages/33550 words 123 pages/33825 words 124 pages/34100 words 125 pages/34375 words 126 pages/34650 words 127 pages/34925 words 128 pages/35200 words 129 pages/35475 words 130 pages/35750 words 131 pages/36025 words 132 pages/36300 words 133 pages/36575 words 134 pages/36850 words 135 pages/37125 words 136 pages/37400 words 137 pages/37675 words 138 pages/37950 words 139 pages/38225 words 140 pages/38500 words 141 pages/38775 words 142 pages/39050 words 143 pages/39325 words 144 pages/39600 words 145 pages/39875 words 146 pages/40150 words 147 pages/40425 words 148 pages/40700 words 149 pages/40975 words 150 pages/41250 words 151 pages/41525 words 152 pages/41800 words 153 pages/42075 words 154 pages/42350 words 155 pages/42625 words 156 pages/42900 words 157 pages/43175 words 158 pages/43450 words 159 pages/43725 words 160 pages/44000 words 161 pages/44275 words 162 pages/44550 words 163 pages/44825 words 164 pages/45100 words 165 pages/45375 words 166 pages/45650 words 167 pages/45925 words 168 pages/46200 words 169 pages/46475 words 170 pages/46750 words 171 pages/47025 words 172 pages/47300 words 173 pages/47575 words 174 pages/47850 words 175 pages/48125 words 176 pages/48400 words 177 pages/48675 words 178 pages/48950 words 179 pages/49225 words 180 pages/49500 words 181 pages/49775 words 182 pages/50050 words 183 pages/50325 words 184 pages/50600 words 185 pages/50875 words 186 pages/51150 words 187 pages/51425 words 188 pages/51700 words 189 pages/51975 words 190 pages/52250 words 191 pages/52525 words 192 pages/52800 words 193 pages/53075 words 194 pages/53350 words 195 pages/53625 words 196 pages/53900 words 197 pages/54175 words 198 pages/54450 words 199 pages/54725 words 200 pages/55000 words	30 days 6 hours 12 hours 24 hours 48 hours 3 days 4 days 5 days 7 days 10 days 20 days	USD GBP CAD AUD EUR  Total: