Home/Quality & Methodology
The methodology, not a metrics page
We're not going to show you a client logo wall or a contributor counter — we don't have verified numbers yet. What we can show you is exactly how a project is run and reviewed, so you can judge the approach on its own terms.
Contributor vetting
Every contributor goes through identity verification, a screening task specific to the dialect, profession, or skill a project requires, and a paid trial batch that's reviewed before they're qualified for live work. Performance on live tasks is tracked per contributor, and it affects whether they continue to be routed work.
Gold-standard seeding
Batches include a proportion of tasks with a known correct answer, seeded in without contributors being able to identify them. This measures reviewer reliability directly, rather than inferring it only from agreement between reviewers.
Inter-annotator agreement
A sample of tasks in every batch is reviewed by more than one contributor. Where they disagree, the task is escalated to a senior reviewer rather than resolved by simple majority vote. Agreement rates are calculated per batch and reported to you — not smoothed over.
Review tiers
- Tier 1
- First-pass contributor review against the task's written rubric.
- Tier 2
- Escalation review for flagged items, low-agreement items, and gold-standard misses.
- Tier 3
- Final QA sampling across the full delivered batch before it ships to you.
What's in a QA report
Task counts by category, review-pass rates, inter-annotator agreement scores where multiple review was used, a summary of contributor pool composition relevant to the task (dialect/region mix, not personally identifying detail), and known limitations of the batch — we report what we found, including the parts that weren't perfect.
Acceptance and rework
You review a sample of any delivered batch against the agreed spec. Items that don't meet it go back through rework, not a refund negotiation — the goal is a dataset you can actually use, and the terms are built around getting there.
Sample deliverable
A downloadable sample package (schema, annotation guidelines, and a redacted QA report) is planned but not yet published — we'd rather ship a real one than a mocked-up placeholder. If you want to see this ahead of general availability, ask on the project form and we'll walk you through an example directly.