Quantifying Quality: Using Quantitative Methods To Evaluation Observation Quality And Its Impact On Incident Reduction

Behavior-based safety (BBS) management systems are effective occupational safety programs that rely on behavioral data collection, intervention, and evaluation to assess intervention effectiveness. This data is collected on observation checklists via peer-to-peer behavioral observations. While there has been research confirming the effectiveness of observation reports on incident prevention, there has been limited research on how checklist quality moderates observation checklist's impact on incident reduction. This may be important, as the BBS iterative process may yield inconsistent results if the data collected is inaccurate.The current study investigates checklist design and response, operationalizing quality checklist design as having more discrimination items, more safety-confirmative items, and more free-response questions. It operationalizes response quality as fixed-response variation and free-response length. All operationalizations of checklist design moderated the effectiveness of observation checklists on interventions such that observation checklists with greater quality were more effective at reducing incident likelihood. Both operationalizations of response quality had insignificant effects on observation effectiveness.