Variability of a job search indicator induced by operationalization decisions when using digital traces from a meter

Abstract

Digital traces —particularly metered data—offer researchers a valuable alternative to surveys for studying online behavior. However, because the concepts being measured are not directly observable in the data, their operationalization requires multiple decisions—for example, which events (e.g., visited websites) represent the concepts and which metrics (e.g., visit counts or time spent) capture their intensity. Using metered data from 600 Netquest panelists in Spain, this study investigates how operationalization choices affect the measurement of job search intensity. By varying metrics—combinations of measurement targets (such as sessions on job platforms or job offer pages) and measurement types (such as visit counts or time spent)—along with other factors (e.g., methods for separating search activity into spells or handling outliers), the study explores 10,080 operationalizations. Results reveal significant variability, with correlations between measurement pairs ranging from 0.14 to 0.91. Metrics sharing the same measurement target (e.g., sessions, job offer pages) demonstrate stronger convergent validity than those sharing only the same measurement type (e.g., time or visits). Other operationalization factors, such as session segmentation methods, also influence results, though less than metric choice. Importantly, operationalization decisions can affect substantive findings.