We have a client that is looking for an e-learning course on "Dealing
with Difficult People". They've posted an $8,000 open challenge on the
LearnCreate website here:
http://www.learncreate.com/dealing-difficult-people/ to have this created.
This is the first public example of a crowdsourced e-learning course that I know of, so I'm looking forward to your comments.
We'll definitely post the winning entries, and they final course will also be provided free of charge to LINGOs (Learning in Non Governmental Organizations) who will make it available to groups such as CARE, Save the Children, Habitat for Humanity, The Nature Conservancy, the World Wildlife Federation and many others.
Thanks for the excellent feedback and the opportunity to respond to your comments. I’ll only cover the first link, since the second and third are also related to the same company.
In the first link, a company called Forty- a direct competitor to a company called 99designs (crowdsourcing pioneer in graphic design)- claims that a business is better off by using a student instead of crowdsourcing to have a logo created.
First of all, coming from a direct competitor it’s not very objective, and secondly, I think a better comparison would be using a professional agency (such as Forty) verses crowdsourcing. After all, there are quite a few students in the “crowd”. A professional agency will charge an average of $5,000 for a logo, and 99designs charges $295. The entire LearnCreate website was built using crowdsourcing: the logo came from a company called choosa (www.choosa.net) for $295 and the buttons came from 99designs (http://www.99designs.com),also for $295. In contrast, I had a logo created ten years ago with a traditional agency for $5,000. Most people like the $295 logo better than the$5,000 logo. Why? Because 2 people from one company contributed their talents and effort to the $5,000 logo and over 100 people contributed their talents to the $295 logo.
Here are a few examples of what’s happening now with crowdsourcing: 99designs paid $1.445million to their “crowd” of developers last month (June, 2012) and they received over $30million in funding last year to expand their range of services. They’re clearly doing something right. Tongal (http://www.tongal.com) makes it possible for companies to buy amazing video advertisements made for TV for $30,000 instead of the typical $400,000 charged by a specialty agency.Innocentive is running a $160,000 challenge right now for the US Government called My Air, My Health (https://www.innocentive.com/ar/challenge/9932947).
I see a lot of hard evidence that crowdsourcing works across the board, and no evidence to the contrary. Nevertheless, there are real issues associated with crowdsourcing, such as loss of confidentiality, missed deadlines and costs of adaptation for the inexperienced. I cover all of these in my blogs, and we do our best to mitigate these for our clients.
Can you explain a little more about how this works? Are multiple designers contributing to your projects? How does the crowd build a course? I find this interesting but don't think I understand how it works.
Here’s a very bad answer to your very good questions: It depends.
We provide a variety of options for our clients to work with the “crowd”, from Open Collaborative (all instructional designers and developers see what others have done and suggest or make improvements) to Closed Private (instructional designers and developers don’t have access to what others have posted and submit concepts and designs for the client to review). This particular posting is Closed Private.
Once the client reviews the submissions,they have the option to proceed with one vendor, or engage a combination of vendors for an additional fee.
I’d be happy to email a comprehensive description of the various crowdsourcing models we support if you’d like. You can also get a high-level overview from my blog here: http://bit.ly/MaDBrA
It's an interesting concept. I don't think there's one right answer but as a service provider and occasional consumer of services I can't help but think there is more risk for the provider and the consumer than the broker of services. The broker takes on the least risk, followed by the consumer of services with the service provider rolling the dice.
Spec work is particularly distasteful to professionals that just want to provide quality services of a "profession." That's what the articles Brian linked above are referencing. "Crowdspec" seems a little wasteful of effort and fairly exploitative of providers that consider themselves professionals. So instead of a one-to-one relationship of effort to outcome, you have X:1 where X is the number of competitors in the spec space working to provide a service for selection. Imagine doing this with other professional crafts like plumbing or electrical. Come do some work for me, if I like the 40 hours that you put in I'll pay you for it and pick you for more work. By the way, I'd like this done for a 30% discount. You're OK with that, right?
This probably works very well in a crowded global market with customers that don't necessarily know what they want or don't know well enough to judge the outcome. Probably works out quite well for the party that takes the least risk, the broker that connects the willing spec worker(s) with a customer that doesn't understand the value of the services they are asking for.
I can see some potential benefit (particularly since it's going to LINGOs and open to all) but it's still competitive spec work. Spec work isn't in the best interest of a professional craft, even if it is proven lucrative for those that broker the services. I wouldn't encourage folks to work for a lottery ticket. Bad for business. Bad for the field.
That's not to say that a model that reduces the risk for the service provider and respects the profession can't work. I'm just not sure this model describes those conditions.
We're always open to ideas on how to better reduce the risk for service providers and better respect the profession, so would love your thoughts on these.
I do disagree that this type of work is "wasteful of effort and fairly exploitative" though. Going through a typical corporate procurement process with RFP's and follow up work will almost always consume more effort than participating in a design challenge, and there's nothing exploitative about it. We don't allow our clients to ask for anything unreasonable, and nothing can be used that isn't paid for.
Thanks again, and looking forward to your feedback on how we can make this work best for all parties!
Whether you agree or disagree about crowd sourcing as a way to develop content, I think this could potentially raise the bar on what can be done. Here are a couple of reasons why:
1. There are plenty of instructional designers who are forced to create page turners and talking head pieces of instruction. This is a great opportunity for them to show some new and more interesting ways of delivering instruction.
2. Most instructional designers are not good programmers but often have to create the content as well as design it. In my opinion this limits what many people design. This is a great opportunity to match up the best of both design, graphics and coding.
3. Whether or not you believe that crowd sourcing or open source is a viable way of developing stuff should not be the debate. It does work in some industries and some efforts have failed miserably. The one thing crowd sourcing consistently does is introduce new ideas and innovation which maybe painful, but is always good for the industry.
I can't wait to see some new and innovative approaches to this topic!
I'm glad you're open to exploring the model. Maybe my perception is off (ICBW - I could be wrong). Either way, this is a really interesting discussion.
I think crowd sourced efforts work better at very small scales or increments. Mechanical Turk isn't making most service providers any money, it's more of a game / hobby but this is the extreme small end of the effort to value scale. The examples described above of crowd sourced logos and buttons are an entirely different scale of effort than a storyboard or course production effort. I assume some correlation of effort to risk. There's probably some formula that can be applied to clarify this
For an example of wasteful of effort:
Let's say you get 10 folks to participate. Each team contributes 15 hours on average to the planning of a design concept for which they (I assume) have limited interaction with the client of the effort. At the close of the competition, it appears that three top folks will receive $250 for these efforts and the top 1 will receive the work to produce the storyboards with a $3250 reward.
So, applying some simple math. That's 150 hours invested. Assuming the winning entries only spent 15 hours on the submission, that's $16 / hour and the winner receives a $3250 contract for an unknown scope for the remaining delivery. 150 - 45 is 105 wasted or uncompensated hours. Any crowdsourced creativity and input benefited the winners. This process creates a bubble or imbalance of equitable compensation for professional effort (see my plumbing and electrical example above, I don't think a designer's time is worth less than a service craftsman).
This becomes a less acceptable risk the larger the scale of the effort. What works for smaller deliverables or in rare cases for providers that work in a depressed local economies doesn't consistently scale well, in my opinion. There's really a potentially cool concept here in progressive enhancement of inputs, but I think small scales of input need to be compensated in some way and so all parties benefit equally. This doesn't necessarily mean $$ currency. In any economy, imbalances of value will eventually lead to collapse.
I don't think you should think of money as the only motivation to participate. People are motivated by things when they participate in an effort like this including recognition, opportunity, experience and even ego. If I just graduated from college and was looking for a job in ISD, I would jump at this purely for the experience. If I was an unemployed designer, I would consider this for opportunity. If I was brilliant I would have thought of this myself years ago...
The only thing I might add to this effort to make it even more enticing might be some formal award and recognition for the top runner ups.
Yeah, this is a great discussion. For what it's worth, Steve and Phil are a couple of the best elearning peeps around. If they wouldn't consider taking this type of work, then who are the designers who would?
Quick question: Will LearnCreate retain exclusive and complete copyrights to all submissions, or only submissions purchased?
Its a great initiative and you never know if it will work or not if you don't try.
In my opinion, I have to agree with Steve and Phil, this type of crowd sourcing will never get me to participate as it does require quite a significant amount of work for me to risk to potentially win. What you will end up attracting are people who either do not have a lot of experience (New college grads) or unemployed designers. I am not saying they cannot deliver, but there will be a lot of work required from the client to weed through all those submissions. Ed you are right, this is a great portfolio building opportunity for new people to ID, but I do not think this will attract seasoned professionals.
As far as recognition, I would personally prefer recognition from competitions run by e-learning software development firms (Articulate or Adobe). Thats my personal preference, because it would have a much wider recognition as they are more global brands.
Crowdsourcing with the level of expertise required in e-learning I think would work better for smaller units of work such as digital assets, voice overs etc.
Crowd sourcing is not new and has been part of the start up world for generations. The only difference is the fact that in a startup environment, the potential of returns are exponential; and hence why I would be more inclined to take a higher degree of risk.
My personal experience in using crowd sourcing has been terrible (rentacoder etc.), and I ended up going with a professional specialist firm.
Thanks for the great comments, and please keep them coming. We knew we wouldn’t get it right the very first time, so all of your comments will have a direct impact on the evolution of our business model and supporting technology!
The comments seem to be focused on three main areas: Level of effort to enter, type of participants, and negative experiences with crowdsourcing.
Level of effort to enter- I think that you’re overestimating the level of effort required to create a submission, especially if you compare this to the level of effort required for a typical corporate or government procurement process where you have the hassles of RFP’s and reference calls. I think it’s safe to say that there is always some level of effort required to compete for new business, and I believe we’ve greatly reduced that effort compared to a normal procurement process. The project documentation contains an eight-page high level design document that a brilliant instructional designer put together for our client, so quite a bit of work has already been done. Also, this is the first in a series of courses, and the likelihood is very high that when the client picks a winning team of designer and developer, they’ll automatically get repeat business without having to compete.
In fact, we make it possible for individuals and small content shops to compete for opportunities that they’d normally never have access to. Our client for this particular challenge is a large automotive manufacturer in Germany, and not a single participant to date would be able to compete because they’re not on the preferred supplier list.
Type of participants- While I agree that all of your assumptions on the level of experience are in line with our expectations as well, I can confirm that every participant in this challenge so far has at least ten years of experience, and one is even the head of a department with 30 instructional designers working for him! All participants are either individual freelancers or small content shops located in the US and Canada. Why do they do it? The content shops have people on the bench that need to be productive and the individuals have capacity and a passion for design and development.
Negative Experience with Crowdsourcing- I totally agree! Our first project was a total failure and the inspiration for my blog series “How to Survive in the Virtual World Without Losing your Health, Hair and Happiness”, which can be found on our website. This is also one of the reasons we exist- we totally mitigate all risk for paying clients and carefully structure and manage the projects, which saves them time and money.
Steve- I’d really be interested in hearing your ideas on micro tasking and compensation. This is a fairly tough nut to crack but we’re actively exploring this.
Will- All IP stays with the developers unless they are paid for their efforts.
Ed- Thanks for the kind words!
In the spirit of transparency, I’ll have an infographic created covering all data points for this challenge, such as average age, education and location of participants, NDA’s requested, project documentation requested and submissions… and of course average time spent to participate
I don't think you should think of money as the only motivation to participate. People are motivated by things when they participate in an effort like this including recognition, opportunity, experience and even ego. If I just graduated from college and was looking for a job in ISD, I would jump at this purely for the experience. If I was an unemployed designer, I would consider this for opportunity. If I was brilliant I would have thought of this myself years ago...
The only thing I might add to this effort to make it even more enticing might be some formal award and recognition for the top runner ups.
Ed
Whilst I do not have the bandwidth to participate, I agree that this is a great way for people to (potentially) increase their profile, "give forward", and perhaps be able to put something interesting on their website/marketing.
18 Replies
What a neat idea. I hope they share the winning entries - it'll be cool to see what they come up with.
We'll definitely post the winning entries, and they final course will also be provided free of charge to LINGOs (Learning in Non Governmental Organizations) who will make it available to groups such as CARE, Save the Children, Habitat for Humanity, The Nature Conservancy, the World Wildlife Federation and many others.
What an awesome concept,.
The problem with crowdsourcing and taking part in such activities:
http://fortyagency.com/insights/crowdsourcing-your-brand
http://gruntmonkey.tumblr.com/post/26777988636/we-did-it-to-ourselves
https://www.facebook.com/notes/creative-go-round/building-without-a-foundation/414702815231474
Thanks for the excellent feedback and the opportunity to respond to your comments. I’ll only cover the first link, since the second and third are also related to the same company.
In the first link, a company called Forty- a direct competitor to a company called 99designs (crowdsourcing pioneer in graphic design)- claims that a business is better off by using a student instead of crowdsourcing to have a logo created.
First of all, coming from a direct competitor it’s not very objective, and secondly, I think a better comparison would be using a professional agency (such as Forty) verses crowdsourcing. After all, there are quite a few students in the “crowd”. A professional agency will charge an average of $5,000 for a logo, and 99designs charges $295. The entire LearnCreate website was built using crowdsourcing: the logo came from a company called choosa (www.choosa.net) for $295 and the buttons came from 99designs (http://www.99designs.com),also for $295. In contrast, I had a logo created ten years ago with a traditional agency for $5,000. Most people like the $295 logo better than the$5,000 logo. Why? Because 2 people from one company contributed their talents and effort to the $5,000 logo and over 100 people contributed their talents to the $295 logo.
Here are a few examples of what’s happening now with crowdsourcing: 99designs paid $1.445 million to their “crowd” of developers last month (June, 2012) and they received over $30million in funding last year to expand their range of services. They’re clearly doing something right. Tongal (http://www.tongal.com) makes it possible for companies to buy amazing video advertisements made for TV for $30,000 instead of the typical $400,000 charged by a specialty agency. Innocentive is running a $160,000 challenge right now for the US Government called My Air, My Health (https://www.innocentive.com/ar/challenge/9932947).
I see a lot of hard evidence that crowdsourcing works across the board, and no evidence to the contrary. Nevertheless, there are real issues associated with crowdsourcing, such as loss of confidentiality, missed deadlines and costs of adaptation for the inexperienced. I cover all of these in my blogs, and we do our best to mitigate these for our clients.
Don,
Can you explain a little more about how this works? Are multiple designers contributing to your projects? How does the crowd build a course? I find this interesting but don't think I understand how it works.
Thank you!
Carla,
Here’s a very bad answer to your very good questions: It depends.
We provide a variety of options for our clients to work with the “crowd”, from Open Collaborative (all instructional designers and developers see what others have done and suggest or make improvements) to Closed Private (instructional designers and developers don’t have access to what others have posted and submit concepts and designs for the client to review). This particular posting is Closed Private.
Once the client reviews the submissions,they have the option to proceed with one vendor, or engage a combination of vendors for an additional fee.
I’d be happy to email a comprehensive description of the various crowdsourcing models we support if you’d like. You can also get a high-level overview from my blog here: http://bit.ly/MaDBrA
Thanks for your questions!
It's an interesting concept. I don't think there's one right answer but as a service provider and occasional consumer of services I can't help but think there is more risk for the provider and the consumer than the broker of services. The broker takes on the least risk, followed by the consumer of services with the service provider rolling the dice.
Spec work is particularly distasteful to professionals that just want to provide quality services of a "profession." That's what the articles Brian linked above are referencing. "Crowdspec" seems a little wasteful of effort and fairly exploitative of providers that consider themselves professionals. So instead of a one-to-one relationship of effort to outcome, you have X:1 where X is the number of competitors in the spec space working to provide a service for selection. Imagine doing this with other professional crafts like plumbing or electrical. Come do some work for me, if I like the 40 hours that you put in I'll pay you for it and pick you for more work. By the way, I'd like this done for a 30% discount. You're OK with that, right?
This probably works very well in a crowded global market with customers that don't necessarily know what they want or don't know well enough to judge the outcome. Probably works out quite well for the party that takes the least risk, the broker that connects the willing spec worker(s) with a customer that doesn't understand the value of the services they are asking for.
I can see some potential benefit (particularly since it's going to LINGOs and open to all) but it's still competitive spec work. Spec work isn't in the best interest of a professional craft, even if it is proven lucrative for those that broker the services. I wouldn't encourage folks to work for a lottery ticket. Bad for business. Bad for the field.
That's not to say that a model that reduces the risk for the service provider and respects the profession can't work. I'm just not sure this model describes those conditions.
Agree with Steve and Brian regarding spec work, I really dont like crowdsourcing as a route to develop and create content.
I can see that it is for a good cause, but I am afraid I cannot commit time to a project like this.
Steve,
We're always open to ideas on how to better reduce the risk for service providers and better respect the profession, so would love your thoughts on these.
I do disagree that this type of work is "wasteful of effort and fairly exploitative" though. Going through a typical corporate procurement process with RFP's and follow up work will almost always consume more effort than participating in a design challenge, and there's nothing exploitative about it. We don't allow our clients to ask for anything unreasonable, and nothing can be used that isn't paid for.
Thanks again, and looking forward to your feedback on how we can make this work best for all parties!
Whether you agree or disagree about crowd sourcing as a way to develop content, I think this could potentially raise the bar on what can be done. Here are a couple of reasons why:
1. There are plenty of instructional designers who are forced to create page turners and talking head pieces of instruction. This is a great opportunity for them to show some new and more interesting ways of delivering instruction.
2. Most instructional designers are not good programmers but often have to create the content as well as design it. In my opinion this limits what many people design. This is a great opportunity to match up the best of both design, graphics and coding.
3. Whether or not you believe that crowd sourcing or open source is a viable way of developing stuff should not be the debate. It does work in some industries and some efforts have failed miserably. The one thing crowd sourcing consistently does is introduce new ideas and innovation which maybe painful, but is always good for the industry.
I can't wait to see some new and innovative approaches to this topic!
Good discussion, Don -
I'm glad you're open to exploring the model. Maybe my perception is off (ICBW - I could be wrong). Either way, this is a really interesting discussion.
I think crowd sourced efforts work better at very small scales or increments. Mechanical Turk isn't making most service providers any money, it's more of a game / hobby but this is the extreme small end of the effort to value scale. The examples described above of crowd sourced logos and buttons are an entirely different scale of effort than a storyboard or course production effort. I assume some correlation of effort to risk. There's probably some formula that can be applied to clarify this
For an example of wasteful of effort:
Let's say you get 10 folks to participate. Each team contributes 15 hours on average to the planning of a design concept for which they (I assume) have limited interaction with the client of the effort. At the close of the competition, it appears that three top folks will receive $250 for these efforts and the top 1 will receive the work to produce the storyboards with a $3250 reward.
So, applying some simple math. That's 150 hours invested. Assuming the winning entries only spent 15 hours on the submission, that's $16 / hour and the winner receives a $3250 contract for an unknown scope for the remaining delivery. 150 - 45 is 105 wasted or uncompensated hours. Any crowdsourced creativity and input benefited the winners. This process creates a bubble or imbalance of equitable compensation for professional effort (see my plumbing and electrical example above, I don't think a designer's time is worth less than a service craftsman).
This becomes a less acceptable risk the larger the scale of the effort. What works for smaller deliverables or in rare cases for providers that work in a depressed local economies doesn't consistently scale well, in my opinion. There's really a potentially cool concept here in progressive enhancement of inputs, but I think small scales of input need to be compensated in some way and so all parties benefit equally. This doesn't necessarily mean $$ currency. In any economy, imbalances of value will eventually lead to collapse.
I don't think you should think of money as the only motivation to participate. People are motivated by things when they participate in an effort like this including recognition, opportunity, experience and even ego. If I just graduated from college and was looking for a job in ISD, I would jump at this purely for the experience. If I was an unemployed designer, I would consider this for opportunity. If I was brilliant I would have thought of this myself years ago...
The only thing I might add to this effort to make it even more enticing might be some formal award and recognition for the top runner ups.
Ed
Yeah, this is a great discussion. For what it's worth, Steve and Phil are a couple of the best elearning peeps around. If they wouldn't consider taking this type of work, then who are the designers who would?
Quick question: Will LearnCreate retain exclusive and complete copyrights to all submissions, or only submissions purchased?
Great topic!
Great topic.
Its a great initiative and you never know if it will work or not if you don't try.
In my opinion, I have to agree with Steve and Phil, this type of crowd sourcing will never get me to participate as it does require quite a significant amount of work for me to risk to potentially win. What you will end up attracting are people who either do not have a lot of experience (New college grads) or unemployed designers. I am not saying they cannot deliver, but there will be a lot of work required from the client to weed through all those submissions. Ed you are right, this is a great portfolio building opportunity for new people to ID, but I do not think this will attract seasoned professionals.
As far as recognition, I would personally prefer recognition from competitions run by e-learning software development firms (Articulate or Adobe). Thats my personal preference, because it would have a much wider recognition as they are more global brands.
Crowdsourcing with the level of expertise required in e-learning I think would work better for smaller units of work such as digital assets, voice overs etc.
Crowd sourcing is not new and has been part of the start up world for generations. The only difference is the fact that in a startup environment, the potential of returns are exponential; and hence why I would be more inclined to take a higher degree of risk.
My personal experience in using crowd sourcing has been terrible (rentacoder etc.), and I ended up going with a professional specialist firm.
Thanks for the great comments, and please keep them coming. We knew we wouldn’t get it right the very first time, so all of your comments will have a direct impact on the evolution of our business model and supporting technology!
The comments seem to be focused on three main areas: Level of effort to enter, type of participants, and negative experiences with crowdsourcing.
Level of effort to enter- I think that you’re overestimating the level of effort required to create a submission, especially if you compare this to the level of effort required for a typical corporate or government procurement process where you have the hassles of RFP’s and reference calls. I think it’s safe to say that there is always some level of effort required to compete for new business, and I believe we’ve greatly reduced that effort compared to a normal procurement process. The project documentation contains an eight-page high level design document that a brilliant instructional designer put together for our client, so quite a bit of work has already been done. Also, this is the first in a series of courses, and the likelihood is very high that when the client picks a winning team of designer and developer, they’ll automatically get repeat business without having to compete.
In fact, we make it possible for individuals and small content shops to compete for opportunities that they’d normally never have access to. Our client for this particular challenge is a large automotive manufacturer in Germany, and not a single participant to date would be able to compete because they’re not on the preferred supplier list.
Type of participants- While I agree that all of your assumptions on the level of experience are in line with our expectations as well, I can confirm that every participant in this challenge so far has at least ten years of experience, and one is even the head of a department with 30 instructional designers working for him! All participants are either individual freelancers or small content shops located in the US and Canada. Why do they do it? The content shops have people on the bench that need to be productive and the individuals have capacity and a passion for design and development.
Negative Experience with Crowdsourcing- I totally agree! Our first project was a total failure and the inspiration for my blog series “How to Survive in the Virtual World Without Losing your Health, Hair and Happiness”, which can be found on our website. This is also one of the reasons we exist- we totally mitigate all risk for paying clients and carefully structure and manage the projects, which saves them time and money.
Steve- I’d really be interested in hearing your ideas on micro tasking and compensation. This is a fairly tough nut to crack but we’re actively exploring this.
Will- All IP stays with the developers unless they are paid for their efforts.
Ed- Thanks for the kind words!
In the spirit of transparency, I’ll have an infographic created covering all data points for this challenge, such as average age, education and location of participants, NDA’s requested, project documentation requested and submissions… and of course average time spent to participate
Whilst I do not have the bandwidth to participate, I agree that this is a great way for people to (potentially) increase their profile, "give forward", and perhaps be able to put something interesting on their website/marketing.
Interesting concept.
Bruce
Thanks for all the great storyboard submissions, and congratulations to the winners- Christina, Melissa and Andree!
http://bit.ly/Nn714Z
This discussion is closed. You can start a new discussion or contact Articulate Support.