Crowdsourcing News, Events, and Resources -- University of Texas as Austin
The University of Texas at Austin     The University of Texas at Austin   The University of Texas at Austin

Crowdsourcing News, Events, and Resources

Maintained by Matt Lease and UT Austin's Information Retrieval & Crowdsourcing Lab as a community resource for information on crowdsourcing & human computation research. Please email suggestions for additional links to share with others.

Blogs and Communities · Shared Task Evaluations · Conferences & Workshops · Journals · Tutorials and Talks · Panels · Books · Additional Blogs · Sites · University Courses · Software · Amazon Mechanical Turk (& related blogs) · Other Vendors · UT Austin Personnel and Resources · A Brief Introduction to Crowdsourcing

Research Blogs and Communities

Shared Task Evaluations

Conferences and Workshops

Journals

Tutorials and Talks

Panels

Books

Additional Blogs

Sites

University Courses

Software and Data

  • SQUARE: open-source methods and data for label aggregation / conensus
  • Alex Sorokin
  • BATC: interactive tool for benchmarking aggregation under varying simulation parameters
  • Boto (source): Python bindings to Amazon Web Services
  • Boto (source): Python bindings to Amazon Web Services
  • ClockWork Raven (Twitter)
  • CrowdForge (source, Boris Smus)
  • Get Another Label (Panos Ipeirotis)
  • GLAD (Jacob Whitehill)
  • MTurk Active Learning (Florian Laws)
  • Quick TurkIt (Jeffrey Bigham)
  • Soylent (source, Michael Bernstein)
  • Truck (Stephen Meylan, delivering linked sequences of HITs on Amazon's Mechanical Turk)
  • TurkIt (source, Greg Little), see also turkit-online and turktime)
  • TurkPipe: a quick way to batch Amazon Mechanical Turk jobs at the command line
  • TurkServer (Andrew Mao)
  • Turk Surveyor (Adam Kapelner)
  • Ushahidi: Crowdsourcing Crisis Information
  • Not available: Jabberwocky, Adrenaline, CrowdWeaver, ...

    Amazon Mechanical Turk Resources

    Other Vendors / Platforms / Marketplaces (See also: list by Panos Ipeirotis)

    UT Austin Personnel and Resources

    Projects

    Examples

    Additional examples

    • Artificial Intelligence, With Help From the Humans (NY Times, March 25, 2007)
    • Inflated Expectations: Crowd-Sourcing Comes of Age in the DARPA Network Challenge (Scientific American, December 1 2009)
    • Can Crowdsourcing Prevent Another Financial Meltdown?
    • Afghanistan war logs: how Wikileaks turned crowd-sourcing into journalism (Telegraph.co.uk) -- see WikiLeaks for yourself
    • Serious games for social change, etc.
    • X PRIZE Takes on Oil Spills with $1.4 Million Open Innovation Challenge
    • Musopen

      A Brief Introduction to Crowdsourcing

      Crowdsourcing (aka "human computation", "distributed work") has emerged in recent years as an exciting new avenue for leveraging the tremendous potential and resources of today's digitally-connected, diverse, distributed population. Crowdsourcing describes outsourcing tasks to large numbers of people in order to leverage the wisdom of crowds. Crowdsourcing platforms such as Amazon Mechanical Turk and CrowdFlower have gained particular attention for connecting employers with the largely under-utilized global workforce. Crowdsourcing offers intriguing new opportunities for accomplishing different kinds of tasks or achieving broader participation than previously possible, as well as completing standard tasks more accurately in less time and at lower cost. Crowdsourcing simultaneously providing new opportunities to workers and non-workers alike (e.g. to have fun, to to find employment in economically-depressed or politically-unstable geographical areas, etc.). See SamaSource, a non-profit founded by a former PeaceCorps member after spending time in African refugee camps.

      Crowdsourcing represents a new intersection of people and technology with corresponding new challenges and opportunities. Since crowdsourcing is ultimately about working with people, it incorporates issues of developing effective design for human factors and human-computer interaction (HCI), as well as issues of economics, ethics, legal policy, etc. With regard to computing, crowdsourcing creates fascinating new opportunities for leveraging real-time human computation for a range of diverse tasks: data annotation, data processing, system evaluation, and "closing the loop" in developing complementary, hybrid human-machine systems. The so-called human processing unit (or "HPU") must be integrated with existing principles and practices for computer architecture and application design, giving rise to a new class of software applications which blend traditional automation with human computation (potentially in real-time) to provide new functionality or accuracy not previously possible with purely automated systems (e.g. Soylent).

      Unlocking the potential of crowdsourcing in practice requires a multi-facted understanding of principles, platforms, and best practices spanning different design verticals: pay-based (real or virtual) marketplaces like Amazon Mechanical Turk, entertainment-based Games with a purpose, and other motivational paradigms based on socialization, prestige, and/or contributing to society (e.g. Wikipedia, Aardvark, Community Q&A sites like WikiAnswers, charity sites like FreeRice, etc.).

      In the 5 years since Amazon.com introduced Mechanical Turk, it has quickly become a phenomenon in academic research across disciplines. A quick search of "mechanical turk" on Google Scholar, for example, returns (an estimated) 1600 results. This academic work reflects part of a much larger, societal trend in which traditional industrial outsourcing work is being increasingly supplanted by crowdsourcing. With crowdsourcing, anyone with an internet connection anywhere in the world can pick and choose what and how much work to do. This shift is a game-changer for employers and workers alike, and reflects a major shift in societal practice as a function of an increasingly well-education global population being connected to the internet.

      Crowdsourcing research has focused around two areas (with overlap): those who study it and those who use it. The first camp might broadly incude areas such as business, economics, legal policy, ethics, and sociology. Whenever any significant shift takes place, academics want to study it, document it, investigate its implications for future practice, and impact its future directions identifying open challenges and unrealized opportunities.

      A second primary area of academic research has been with researchers who are less concerned with studying crowdsourcing but whose field methdologies are being affected by it (as might be expected wityh with any singificant shift in practice). Crowdsourcing is radically changing the methodology of how various kinds of research are now being carried out for greater responsiveness, effectiveness, and affordability. This is seen most clearly in areas like electrical and computer engineering, computer science, linguistics, and psychology, where crowdsourcing is enabling new forms of data collection and user studies.