Lesson plan

Expert review (method)

Expert review or expert evaluation is a method to evaluate survey questions from the perspective of one or more experts. An expert review has two primary goals: # Identify potential problems related to data quality and data collection so they can be mitigated. # Group survey items by how likely they are to result in measurement errors, such as to classify interviewer effects. An individual expert or a panel of experts such as survey methodologists, subject matter experts (such as sociologists) or other people familiar with questionnaire design review the survey, and they may be asked to suggest revisions. Because variation is found in the problems identified by each expert, a panel rather than one single expert is recommended. The experts document their assessment and typically report the results in open-ended comments. They can also conduct a systematic review using a coding tool such as the Questionnaire Appraisal System that is supported by an item taxonomy of the cognitive demands of a question and a detailed set of potential problem codes document the features that may lead to response error. When applied to translated surveys, the expert panel also includes translators and they usually produce individual reports and then convene a resolution meeting that generates an improved version of the survey translation.…

Back to article

Article summary

Expert review or expert evaluation is a method to evaluate survey questions from the perspective of one or more experts. An expert review has two primary goals: # Identify potential problems related to data quality and data collection so they can be mitigated. # Group survey items by how likely they are to result in measurement errors, such as to classify interviewer effects. An individual expert or a panel of experts such as survey methodologists, subject matter experts (such as sociologists) or other people familiar with questionnaire design review the survey, and they may be asked to suggest revisions. Because variation is found in the problems identified by each expert, a panel rather than one single expert is recommended. The experts document their assessment and typically report the results in open-ended comments. They can also conduct a systematic review using a coding tool such as the Questionnaire Appraisal System that is supported by an item taxonomy of the cognitive demands of a question and a detailed set of potential problem codes document the features that may lead to response error. When applied to translated surveys, the expert panel also includes translators and they usually produce individual reports and then convene a resolution meeting that generates an improved version of the survey translation.…

Key claims

No claims recorded.

Discussion questions

  1. Summarise the article in your own words. Which single claim would you verify first, and why?
  2. Which claim in this article is most likely to be revised as the sources update? How would you check its current status?
  3. If two sources disagreed about one of these facts, what would you need in order to trust one over the other?

Bibliography

No sources recorded.

Key claims and bibliography are generated from the same tracked claims and sources as the article page.