Using Feedback Army
1. Where are my results?
When you checkout from Feedback Army, you are presented with a link to a page that will contain these results. Your results will show on this page as they come in.
Your results are also available as an RSS feed that you're welcome to subscribe to.
Feedback Army also sends a link to your results page via email. If you don't see this email, it's possible it went to your spam folder. Be sure to check there too.
Feedback Army does not send a final email when all of your results are in. Make sure you check your results page regularly.
If you ordered a review and do not have the results link, contact me and I will get it to you.
2. How long until I see results?
Each request is posted for eight days. Generally you'll start to see results right away. I've seen many website owners receive their initial ten responses within one to three hours.
The reviewers pick and choose which sites they want to review. Requests with the following traits tend to discourage reviewers:
- Poorly worded questions
- Excessive number of questions
- Requiring personal information/email address to "use" site
I find three to six questions is fair. Read the usability testing tips for more information.
3. How can I get more responses than what you're offering?
Feedback Army now offers up to 50 responses from unique reviewers. From dialog I've had with many of you this is enough responses.
I'm only offering 50 responses at this time because it is what I know Feedback Army can deliver. I don't recommend purchasing the same set of questions twice to get more responses.
4. Are my results private?
Requests made after 15 Jun 09 are. Feedback Army now generates a special code in the link to view your results. Your results are not viewable without this code. If you want to share the link (with code), this is your choice. For those of you who want your results to be private... you're all set.
5. Are my questions and link private?
No. Feedback Army is built on Amazon's Mechanical Turk web service. Your questions and link are publicly available on that service for as long as it takes for the reviewers to review your site. After your site receives the specified number of reviews, your link and questions are pulled from the service.
6. Can I ask questions in a language other than English?
Unfortunately, Feedback Army is English only at this time. There are two reasons for this. The service I submit your questions to doesn't like accented characters. Also, this service is meant for folks who speak English. There is no way for me to separate reviewers who speak Greek or Finnish from those who speak only English.
That said, I have seen a Spanish language site submitted with English questions. The site owner had questions about layout and the graphics. Language agnostic stuff. Even in this case, most reviewers commented that understanding the site would have helped them give better feedback.
7. I'm a company lawyer and noticed a request for my client's site on Mechanical Turk!
Great! I love company lawyers. Please accept the possibility that a web design or user experience firm your client contracted with made the request. Unfortunately, I can't tell you who made the request because it would violate the privacy of my customers.
Feedback Army is not claiming an affiliation with your company and Feedback Army is not attempting to infringe on your intellectual property.
If you choose to contact me, be respectful. I will respond to such queries in kind.
8. How do you quality control responses?
As an experiment, I'm putting that power into your hands. Feedback Army responses are auto-approved after 48 hours. Until that time you will see a "Reject this response" link on your results page. You can click this link to reject the response and request a new one in its place.
Please be nice when using this feature. If the worker said something honest (but negative) about your site, this isn't a good reason to reject their response. You should reject responses that make no sense, look fake, or fail to follow your instructions.
9. I'd like a refund!
Sure, no problem. Simply visit the refund request page and fill out the form. I'll process it ASAP. I'm sincere about bringing value to your process. If you're not happy with your results or if Feedback Army fell short of your expectations then let me take care of it.
About the Reviewers
10. Who are the reviewers?
The reviewers are really cool people. The reviewers come from a service called Mechanical Turk. Mechanical Turk gives Feedback Army a fresh pool of workers ready to handle your request day and night.
The Mechanical Turker Worker demographics are: 70% of the reviewers are from the United States. Over half are female and over half have college degrees.
1 May 11: I was recently asked about these demographics again. These numbers are from a 2008 sample. From my own data, I see that 30% of the reviewers who work with me (and you) are from the United States. The rest come from other parts of the world.
5 Oct 11: Feedback Army now uses geolocation to limit reviews to reviewers from the United States, Canada, Australia, and the United Kingdom. The location is now reported with each review too.
11. How can I become a reviewer?
Visit www.mturk.com and sign up for an account. Here are some interesting articles on the life of a turker:
- "I make $1.45 a week and I love it" (Salon.com)
- Amazon's Mechanical Turk lets you make $$$, sort of (cnet.com)
12. Can I skip you and post to Mechanical Turk directly?
Absolutely! Here is what you need to do:
- Create an Amazon Web Services account
- Figure out how much you want to spend (and don't forget to calculate Amazon's share)
- Prepay the exact amount or more
- Create a Human Intelligence Task template
- Publish the Human Intelligence Task
- Login to Amazon's portal regularly and navigate several pages to find your results. (Have fun clicking refresh.)
- Review the results and promptly pay your workers (or they may ignore future requests)
That said, Feedback Army does add value to the Mechanical Turk process. Here is what you get:
- Results are made available as an RSS feed
- Feedback Army gives workers access to built-in spell, style, and grammar check.
- Feedback Army is much easier to use (saves you from those steps above).
- I'm told Feedback Army has excellent customer service. You'll have to find out for yourself :)