Research Methods
Strengthening the Foundations of Scholarship
Faculty and Ph.D. students in the Department of Management and Entrepreneurship are deeply engaged in advancing research methods—the tools and practices that underpin reliable, trustworthy science. By examining how studies are designed, conducted, analyzed and reported, our work contributes to improving the accuracy and impact of organizational research.
Our research spans two major areas:
- Meta-Science: Investigating research practices themselves, including issues such as publication bias, detection of irregularities and the curation of scientific knowledge through initiatives like metaBUS.
- Statistical Techniques: Examining common errors and developing solutions to ensure studies use rigorous methods that produce accurate, robust and replicable results.
Together, these research streams help shape more credible and transparent scholarship, ensuring that management research stands on a strong and reliable foundation.
Research Methods: Meta-science
The general term meta-science refers to research about research practices. The purpose of this type of research is to compare current research practices with best practices and to suggest strategies for bridging the gap.
Meta-science: Publication bias and the trustworthiness of our cumulative knowledge
Publication bias refers to a situation in which the publicly available literature on a particular relation of interest is not representative of all studies on that relation. Unfortunately, research has shown that our journals tend to publish articles with mostly statistically significant results, leading to a cumulative knowledge base that may not be trustworthy. Our research seeks to understand the extent to which our published results are robust and the associated conclusions trustworthy.
- Keener, S. K., Kepes, S., & Torka, A.-K. (2023). The trustworthiness of the cumulative knowledge in industrial/organizational psychology: The current state of affairs and a path forward. Acta Psychologica, 239, 104005. doi: 10.1016/j.actpsy.2023.104005
- Field, J. G., Bosco, F. A., & Kepes, S. (2021). How robust is our cumulative knowledge on turnover? Journal of Business and Psychology, 36, 349-365. doi: 10.1007/s10869-020-09687-3
- Kepes, S., Banks, G. C., Keener, S. K. (2020). The TOP factor: An indicator of quality to complement journal impact factor. Industrial and Organizational Psychology: Perspectives on Science and Practice, 13, 328-333. doi:10.1017/iop.2020.56
- Kepes, S., & McDaniel, M. A. (2013). How trustworthy is the scientific literature in industrial and organizational psychology? Industrial and Organizational Psychology: Perspectives on Science and Practice, 6, 252-268. doi: 10.1111/iops.12045
- Kepes, S., Banks, G. C., McDaniel, M. A., & Whetzel, D. L. (2012). Publication bias in the organizational sciences. Organizational Research Methods, 15, 624-662. doi: 10.1177/1094428112452760
Meta-science: Detection of irregularities and inaccuracies in existing research
To address the potential adverse effects of publication bias and related biases to the accuracy of our cumulative knowledge, we seek to understand their causes. They include hypothesizing after the results are known (HARKing), the disconnect between models that researchers claim to test and the models that they actually test, and the sheer misuse of statistical techniques. We have published extensively on these and related causes.
- Kepes, S., Keener, S. K., McDaniel, M. A., & Hartman, N. S. (2022). Questionable research practices among researchers in the most research-productive management programs. Journal of Organizational Behavior, 43, 1190-1208. doi: 10.1002/job.2623
- Cortina, J. M., Markell, H. M., Green, J. P., & Chang, Y. (2019). How do we test interactions in latent variable models? Surging forward or fighting shy? Organizational Research Methods.
- Cortina, J., Koehler, T., Keeler, K. R., Nielsen, B. B. (2019). Restricted variance interactions: What they are and why they are your friends. Journal of Management.
- Keeler, K.R., Kong, W., Dalal, R., Cortina, J.M. (2019). Situational strength interactions: Are variance patterns consistent with the theory? Journal of Applied Psychology.
- Cortina, J. M., Green, J. P., Keeler, K. R., & Vandenberg, R. J. (2017). Degrees of freedom in SEM: Are we testing the models that we claim to test? Organizational Research Methods, 20, 350-378. doi: 10.1177/1094428116676345
- Bosco, F. A., Aguinis, H., Field, J. G., Pierce, C. A., & Dalton, D. R. (2016). HARKing’s threat to organizational research: Evidence from primary and meta-analytic sources. Personnel Psychology, 69, 709-750. doi: 10.1111/peps.12111
- Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349, 943-951. doi: 10.1126/science.aac4716
Meta-science: Research curation
To attain a more accurate picture of our cumulative knowledge, one has to summarize the results of many scientific findings, which can be arduous and takes years. To facilitate this endeavor, we have started an effort to collect and curate scientific results in organizational behavior and human resource management and build a platform that allows for a rapid search, summary and interaction with the data (see metaBUS.org).
- Bosco, F. A., Field, J. G., Larsen, K., Chang, Y., & Uggerslev, K. L. (2019). Advancing meta-analysis with knowledge management platforms: Using metaBUS in psychology. Advances in Methods and Practices in Psychological Science.
- Bosco, F. A., Uggerslev, K. L., & Steel, P. (2017). metaBUS as a vehicle for facilitating meta-analysis. Human Resource Management Review, 27, 237-254. doi: 10.1016/j.hrmr.2016.09.013
- Bosco, F. A., Steel, P., Oswald, F. L., Uggerslev, K. L., & Field, J. G. (2015). Cloud-based meta-analysis to bridge science and practice: Welcome to metaBUS. Personnel Assessment and Decisions, 1, 3-17. doi: 10.25035/pad.2015.002
Research Methods: Statistical techniques
To obtain more accurate and robust results, the use of sound scientific processes and rigorous statistical techniques is paramount. Unfortunately, many published studies use less than optimal processes and statistical techniques. Therefore, in this research stream, we are examining commonly made mistakes and provide solutions in an effort to ensure that published results are accurate and robust, a requirement for a cumulative knowledge base that is trustworthy.
- Kepes, S., Wang, W., & Cortina, J. M. (2024). Heterogeneity in meta-analytic effect sizes: An assessment of the current state of the literature. Organizational Research Methods, 27, 369-413. doi: 10.1177/10944281231169942
- Kepes, S., Wenhao, W., & Cortina, J. M. (2023). Assessing publication bias: A 7-step user’s guide with best-practice recommendations. Journal of Business and Psychology, 38, 957-982. doi: 10.1007/s10869-022-09840-0
- Cortina, J. M., Koehler, T., Keeler, K. R., & Nielsen, B. B. (2018). Restricted variance interaction effects: What they are and why they are your friends. Journal of Management. doi: 10.1177/0149206318770735
- Kepes, S., & Bushman, B. J., & Anderson, C. A. (2017). Violent video game effects remain a societal concern: Reply to Hilgard, Engelhardt, and Rouder (2017). Psychological Bulletin, 143, 775-782. doi: 10.1037/bul0000112
- Kepes, S., McDaniel, M. A., Brannick, M. T., & Banks, G. C. (2013). Meta-analytic reviews in the organizational sciences: Two meta-analytic schools on the way to MARS (the Meta-analytic Reporting Standards). Journal of Business and Psychology, 28, 123-143. doi: 10.1007/s10869-013-9300-2
- Cortina, J. M. (2003). Apples and oranges (and pears, oh my!): The search for moderators in meta-analysis. Organizational Research Methods, 6, 415-439. doi: 10.1177/1094428103257358