Flash Teams at UIST 2014

Today I am going to tell you about how we computationally gather and guide experts from the crowd with flash teams.

While technology enables groups of individuals to come together and rapidly complete large tasks, the dominant paradigm in crowdsourcing has been to use many non-expert workers from platforms such as mechanical turk to complete homogeneous microtasks, such as image labeling and text shortening.

This approach to crowd work does not easily scale to larger open-ended and creative tasks, such as design and video production. Specifically, when task units are big, they become much harder to decompose into homogenous micro tasks, especially when they require a certain level of expertise to complete. Overcoming these barriers is critical in order for crowd work to reach its potential and support complex and interdependent work.

Our objective is to explore the feasibility of solving complex, interdependent problems BY STRUCTURED COLLABORATIONS BETWEEN EXPERTS FROM THE CROWD.

For example, could we bring together a team of crowd workers, give them a napkin sketch of an idea and have them create a user-tested interactive web app in a single day?

How about an animated video in 48 hours?

Better yet, could we push the boundaries of crowdsourcing and bring together multiple teams of crowd workers to create an entire MOOC platform along with content within a day or two?

To accomplish these goals, we shift away from paid micro task work and introduce diverse experts from crowd platforms such as oDesk. oDesk is an emerging online marketplace where you can find paid experts and professionals with a variety of skills, including programmers, designers, video editors and musicians.

Experts on these platforms mostly work as uncoordinated independent contractors. This limits the goals that they can pursue.

And microtask crowdsourcing techniques that assume homogenous crowds do not effectively leverage participants’ diverse expertise.

It might be tempting to just bring crowd workers together and have them self-manage but it turns out these teams are riddled with problems.

However, we know from organizational behavior research, that even temporary groups can coordinate complex work effectively if they have what are called lightweight team scaffolds. These scaffolds define who is working together and who is responsible for which tasks.

Our goal is to combine the strengths of team scaffolds with the scale and interactivity of computation to dynamically assemble and manage teams of expert crowd workers, which we refer to as computational organizational behavior.

To do this, we introduce Flash Teams, which are computationally-guided teams of crowd experts supported by lightweight, reproducible and scalable team structures.

For example, computation might help a flash team transform a napkin sketch of an idea into a user-tested, interactive web prototype in under a day.

Project requesters assemble and manage a flash team by chaining blocks together to transform a starter input into a desired final output.

The complete set of blocks determines which crowd experts need to be recruited and when they are likely to be needed. With offline teams, Bunderson and Boumgarden found that these kinds of structures succeed by encoding individuals’ responsibilities and dependencies.

Each block is a self-contained, reusable representation that has computational properties. It requires the skills needed, an input, target duration, instructions, and an output type.

This kind of modularity isn't just useful for programming: management modularity theory makes it clear that loosely coupled team elements, connected via standard interfaces, are able to interact in many different configurations. And that's exactly what we intend to enable.

For example, suppose you are a project requester building a team to execute the user-centered design process starting from a napkin sketch of an idea. You might connect the Balsamiq mockups from the first block to any other block that takes low-fi mockups as input, such as a revised mockup block or a heuristic evaluation block.

To put this into context, here you can see what this flash team’s complete sequence of blocks and inputs and outputs might look like.

Most work teams suffer from what's known as "coordination neglect”: they are skilled at splitting up work but much worse at coordinating and combining their efforts at the end.

Because flash teams are machine readable, they are computationally malleable. This lets us minimize coordination neglect and create entirely new kinds of organizational forms, which let teams replicate, grow and shrink, and more.

First, flash teams benefit from modularity, which allows them to be replicated and used for different projects.

For example, the same napkin sketch design flash team can be replicated and used for many different types of web or mobile applications. For the most part, organizational behavior research has focused on using these kinds of team scaffolds for fluid membership. We focus on how they allow us to capitalize on the scale of the crowd.

Another benefit of modularity is that, flash teams can be combined like Lego blocks to create larger organizations.

For example, to create an online learning platform, a flash team might combine several teams that can design and implement software with additional teams that can develop educational materials and animated videos.

Second, traditional teams are pretty static: you’re either on the team, or you're off. But, because crowdsourcing platforms allow for quick recruitment, flash teams also benefit from elastic growth, which enables teams to grow and shrink as needed. 

Third, flash teams can also be pipelined rather than waiting for the entire task to complete. Suppose the system knew whether a task could accept in progress input or produce in-progress output. For instance, the UI designer is idle while the Heuristic Evaluation is done,

but he could start working on the revised version of the low-fi prototype as soon as the UX researcher reports the first heuristic violation. This allows computation to automatically optimize these teams.

Finally, so far, we have assumed that project requesters are fluent with creating teams. Most people, however, have a goal in mind but may not have a strong idea of how to break it down.

We can search through the space of known inputs and outputs to automatically create a new team — one that may have never previously existed! — using the components of previous teams.

To do this, we represent all known flash team blocks in the STRIPS planning language and use a planning algorithm to find a set of actions that connect from original input to final output. While the general planning problem is NP-complete, boolean satisfiability solvers can scale these systems to thousands of actions.

Existing project management and micro task management tools do not capitalize on the unique affordances and scale of distributed expert crowd work. This means that flash teams require new kinds of tools and infrastructure.

We created Foundry, a web platform tailored to the design affordances of distributed expert crowds. It allows project requesters to create flash team structures and helps team members track their progress and send notifications when a handoff has been issued.

With Foundry, requesters can create a flash team from scratch or compose or modify previous team structures from the Foundry flash teams library.

To give you a sense of how Foundry works, let’s take the example I showed earlier and create a team to execute the user-centered design process.

As a requester, you would first add the team roles by reaching into the nearly 2,500 oDesk skill categories.

You would then add blocks to the timeline as you would on a calendar interface, including all of the metadata around members, inputs and outputs.

You can make dependencies visible by explicitly marking handoffs that connect tasks based on inputs and outputs.

When each block is complete, worker(s) upload their output materials.

Foundry also makes available a library of all blocks that previous teams have used for drag-and-drop.

In runtime mode, Foundry adds a red timer playhead that starts advancing along the timeline.

Each worker on your team receives a unique link that logs them in to the Foundry runtime with their tasks highlighted in the timeline.

When a worker marks a block as complete, Foundry notifies workers for the next blocks that their task is active.

If a task is delayed or completed early, Foundry recalculates the start time of subsequent tasks, updates the timeline and notifies workers.

In order to better understand the design space of flash teams and what they would enable us to do, we recruited experts from oDesk and ran a series of example projects, ranging from design to education.

The first series of projects that we ran was with the napkin sketch design team.

We gave three different flash teams sketches that we took from Stanford student assignments in the Introduction to HCI class. Their goal was to produce a fully functional and user-tested web prototype.

Here you can see a summary of the napkin sketch outcomes along with some screenshots of the different web prototypes. The completion times ranged from 18 hours to 31 hours and 30 minutes depending on the amount of pipelining and elasticity. All of the outputs were fully functional and reflected the original intent. Therefore, these flash teams took a project that takes weeks in an intro HCI class and completed it in a fraction of the time.

After completing the napkin sketch design teams, we asked ourselves: … … “can flash teams support creative outputs and non engineering domains?”.

To test this, we decided to use a flash team to create an animated video based on a rough script idea, which was to elebrate the retirement of Terry Winograd. In under two days, the flash team came up with a plot and produced the video, which captured a funny anecdote from his childhood, when Terry built a computer in his garage.

Let’s have a look at the video that the flash team produced.

https://www.youtube.com/watch?v=183pmArc_mQ

The napkin sketch design teams and animations team illustrated that a flash teams approach to expert crowdsourcing works.

Our final objective was to combine multiple modular team structures to create a new online learning (MOOC) platform and to generate three short lessons, which included introductions to using the diaphragm while singing, portrait photography and the Towers of Hanoi puzzle.

We added one more “education” flash team that combined a curriculum design expert with a topic expert in each area.

Then, by replicating and combining these teams like Lego blocks, we could create a macro-scale organization.

For each course, one education team created the course content and quiz materials and one animation team produced the video. Simultaneously, three napkin sketch teams worked on the web platform used to upload and share the courses.

Taken together, it took the nine flash teams 19 hours and 20 minutes to create the MOOC platform and three interactive lessons. You can see excerpts from each of the three courses at the bottom of the slide.

We have one final question. Does this infrastructure help people collaborate? We conducted a controlled experiment to compare flash teams to a self-managed team approach where the crowd experts choose their own path. We collected data with 3 napkin sketch design teams in each condition, totaling 22 crowd experts from oDesk.

We asked all of the teams to create a simple party planning task manager from an input sketch.

The main difference between the two conditions was the existence of the flash team workflow, which defined the sequences of tasks to the team members, and allowed Foundry to support coordination via handoffs and notifications.

While all six of the teams completed the task, the flash teams took roughly HALF AS MANY WORK HOURS, which means half the cost. They also followed the iterative design process more closely, and required less coordination.

In fact, by active work time, even the slowest flash team completed the task faster than the fastest team in the control condition.

Most crowdsourcing research has focused on coordinating workers that are homogeneous and replaceable.

Flash teams offer an opportunity to shift the crowdsourcing narrative towards teams of diverse experts from the crowd.

More importantly, we strongly believe that flash teams and platforms like foundry will enable the combination of computation and organizational behavior to shape the future of crowd work.