Experimentology: An Open Science Approach to Experimental Psychology Methods

“How do we create generalizable theories of human behavior? Experiments provide us a tool for measuring causal effects, which provide the basis for building theories. If we design our experiments appropriately, we can even begin to estimate generalizable relationships between different psychological constructs. But how do you do an experiment?

This book provides an introduction to the workflow of the experimental researcher in the psychological sciences. The organization is sequential, from the planning stages of the research process through design, data collection, analysis, and reporting. We introduce these concepts via narrative examples from a range of sub-disciplines, including cognitive, developmental, and social psychology. Throughout, we also illustrate the pitfalls that led to the “replication crisis” in psychology. To help researchers avoid these pitfalls, we advocate for an open-science based approach, providing readers with guidance for preregistration, project management, data sharing, and reproducible writing….”

Ten Strategies to Foster Open Science in Psychology and Beyond | Collabra: Psychology | University of California Press

Abstract:  The scientific community has long recognized the benefits of open science. Today, governments and research agencies worldwide are increasingly promoting and mandating open practices for scientific research. However, for open science to become the by-default model for scientific research, researchers must perceive open practices as accessible and achievable. A significant obstacle is the lack of resources providing a clear direction on how researchers can integrate open science practices in their day-to-day workflows. This article outlines and discusses ten concrete strategies that can help researchers use and disseminate open science. The first five strategies address basic ways of getting started in open science that researchers can put into practice today. The last five strategies are for researchers who are more advanced in open practices to advocate for open science. Our paper will help researchers navigate the transition to open science practices and support others in shifting toward openness, thus contributing to building a better science.

 

NIMH » NIMH Creates Publicly Accessible Resource With Data From Healthy Volunteers

“Studying healthy people can help researchers understand how the brain works in states of health and illness. Although many mental health studies include healthy participants as a comparison group, these studies typically focus on selected measures relevant to a certain functional domain or specific mental illness. The Healthy Research Volunteer Study at the National Institute of Mental Health aims to build a comprehensive, publicly accessible resource with a range of brain and behavioral data from healthy volunteers.

This resource aims to shed light on basic questions about brain function and translational questions about the relationship between brain and behavior. Although the study focuses on healthy volunteers, the data also have relevance to clinical questions about neurobiological, cognitive, and emotional processes associated with certain mental illnesses.

The NIMH Healthy Research Volunteer Study is unique in the breadth and depth of its measures. All data collected as part of the study are anonymized and shared with the research community via the OpenNeuro repository….”

About Meta-Psychology

“Meta-Psychology publishes theoretical and empirical contributions that advance psychology as a science through critical discourse related to individual articles, research lines, research areas, or psychological science as a field. Important contributions include systematic reviews, meta-analyses, replicability reports, and replication studies. We encourage pre-registered studies and registered reports (i.e., peer-review on the basis of theory, methods, and planned data-analysis, before data has been collected). Manuscripts introducing novel methods are welcome, but also tutorials on established methods that are still poorly understood by psychology researchers. We further welcome papers introducing statistical packages or other software useful for psychology researchers….”

 

Open Science, Mental Health, and Sustainable Development: A Proposed Model for a Low-Resource Setting

“Mental health is an important concern in low and middle income countries and must be addressed for sustainable development. Open science is a movement which can contribute significantly towards addressing mental health challenges. Mental health in India and other low and middle income countries faces many challenges, such as lack of resources and low investment. This policy brief proposes an intervention model using the core principles of open science to transform the mental health programmes run by local self-government institutions in India. The model can co-opt key stakeholders involved in the data collection, programme implementation, and monitoring for standardisation. Kerala’s participatory development experience is employed as a case to describe the model. By empowering frontline health workers, accredited volunteers, and officials of the childcare system, and implementing open science principles, this model could help address mental health challenges with minimal resource allocation through the streamlining of the data management process. It could also encourage increased participation in open science through the citizen science model, opening scientific research to non-specialists. Open science principles such as collective benefit, equity, participation, sustainability, and inclusiveness can also be promoted.”

 

Further action toward valid science in Law and Human Behavior: Requiring open data, analytic code, and research materials.

“Beginning on March 1, 2023, Law and Human Behavior will raise its standard for data reporting and expand its focus to include analytic code and research materials. Adopting the recommended language from the TOP Guidelines (Nosek et al., 2015b), the journal will publish articles only if the data, analytic code, and research materials are clearly and precisely documented and are fully available to any researcher who wishes to reproduce the results or replicate the procedure.

Accordingly, authors using original data who seek to publish their research in the journal must make the following items publicly available: …

 

Authors reusing data from public repositories who pursue publication in Law and Human Behavior must provide program code, scripts for statistical packages, and other documentation sufficient to allow an informed researcher to precisely reproduce all published results….”

 

Responsible Research Assessment I: Implementing DORA for hiring and promotion in psychology | PsychArchives

Abstract:  The use of journal impact factors and other metric indicators of research productivity, such as the h-index, has been heavily criticized for being invalid for the assessment of individual researchers and for fueling a detrimental “publish or perish” culture. Multiple initiatives call for developing alternatives to existing metrics that better reflect quality (instead of quantity) in research assessment. This report, written by a task force established by the German Psychological Society, proposes how responsible research assessment could be done in the field of psychology. We present four principles of responsible research assessment in hiring and promotion and suggest a two-step assessment procedure that combines the objectivity and efficiency of indicators with a qualitative, discursive assessment of shortlisted candidates. The main aspects of our proposal are (a) to broaden the range of relevant research contributions to include published data sets and research software, along with research papers, and (b) to place greater emphasis on quality and rigor in research evaluation.

 

Open Science in Developmental Science | Annual Review of Developmental Psychology

Abstract:  Open science policies have proliferated in the social and behavioral sciences in recent years, including practices around sharing study designs, protocols, and data and preregistering hypotheses. Developmental research has moved more slowly than some other disciplines in adopting open science practices, in part because developmental science is often descriptive and does not always strictly adhere to a confirmatory approach. We assess the state of open science practices in developmental science and offer a broader definition of open science that includes replication, reproducibility, data reuse, and global reach.

 

Open Access information specialist 1.0 FTE, permanent | Leibniz-Institut für Psychologie (ZPID)

English version via deepl.com:

We are looking for an Open Access Information Specialist (m/f/d), (pay grade TV-L E13, working hours 100%, permanent), start date February 01, 2023 or later

Tasks and Functions:

You are part of the department “Archiving and Publishing Services” at ZPID, which provides innovative products and services for psychology and related disciplines that follow the Open Science idea. One of the lead products of this infrastructure area is PsychOpen GOLD, ZPID’s open access publication platform for first publications in psychology and related disciplines. In close cooperation with renowned scholars, professional societies and international editorial boards, 15 journals are currently produced and published on PsychOpen GOLD.

Together with the PsychOpen GOLD team, you will provide operational support for all tasks arising in the context of the PsychOpen GOLD service catalog and, in coordination with other stakeholders (e.g., external service providers and scientific partners), contribute significantly to the optimization of the associated workflows. For detailed information on the PsychOpen GOLD service catalog, please see https://doi.org/10.23668/psycharchives.4632. In addition, you will provide scientific support for the development of the platform and participate in the acquisition of third-party funding.

Your profile:

Master’s degree (or equivalent/higher) in information science or another relevant discipline, e.g., data science, library science, social and behavioral sciences.
Advanced experience in scientific work and publishing (e.g. as author, reviewer, etc.)
Expertise in publication-relevant application programs (e.g. in the area of office applications, graphics processing, PDF authoring)
IT skills preferably in the areas of markup languages (e.g. XML, HTML, LaTeX), transformation and stylesheet languages (e.g. XSL, CSS), scripting and programming languages (e.g. PHP, Javascript, Python), database systems and languages (e.g. MySQL).
Project management skills
Very good command of the German and English language, both written and spoken
Proactive, structured, result-oriented and independent way of working
Distinct ability to work in a team and social competence

Desirable:

Knowledge of scientific publication infrastructures (e.g., subject databases, DOI registration agencies), standards (e.g., metadata standards, Open Science Standards), and/or software systems (e.g., OJS, Editorial Manager)
Psychological expertise and methodological knowledge
Experience in the area of third-party funding acquisition
Basic knowledge of legal frameworks (licenses, copyright, data and privacy rights)

German original:

 

 

Wir suchen zum 01. Februar 2023 oder später eine*n:

Open Access Informationsspezialist*in (m/f/d)

(Entgeltgruppe TV-L E13, Arbeitszeit 100%, unbefristet)

 

Aufgaben und Funktionen:

Sie sind Teil der Abteilung “Archivierungs- und Veröffentlichungsdienste” am ZPID, die innovative und dem Open Science-Gedanken folgende Produkte und Dienstleistungen für die Psychologie und verwandte Disziplinen zur Verfügung stellt. Eines der Leitprodukte dieses Infrastrukturbereichs ist PsychOpen GOLD, die Open Access Publikationsplattform des ZPID für Erstveröffentlichungen aus der Psychologie und verwandten Fächern. In enger Zusammenarbeit mit renommierten Fachwissenschaftler*innen, Fachgesellschaften und international besetzten Editorial Boards werden aktuell 15 Zeitschriften auf PsychOpen GOLD produziert und veröffentlicht.

Gemeinsam mit dem PsychOpen GOLD Team werden Sie  sämtliche im Rahmen des PsychOpen GOLD Servicekatalogs anfallenden Aufgaben operativ unterstützen und in Abstimmung mit weiteren Stakeholdern (z.B. externe Dienstleister und Wissenschaftspartner) maßgeblich zur Optimierung der zugehörigen Workflows beitragen. Für detaillierte Informationen zum Servicekatalog von PsychOpen GOLD siehe https://doi.org/10.23668/psycharchives.4632. Darüber hinaus werden Sie die Entwicklung der Plattform wissenschaftlich begleiten und sich bei der Akquise von Drittmitteln beteiligen.

Ihr Profil:

Masterabschluss (oder gleichwertig/höher) in den Informationswissenschaften oder einer anderen einschlägigen Disziplin, z.B. Data Science, Bibliothekswissenschaften, Sozial- und Verhaltenswissenschaften
Erweiterte Erfahrungen im wissenschaftlichen Arbeiten und Publizieren (z. B. als Autor*in, Gutachter*in, etc.)
Expertise bei publikationsrelevanten Anwendungsprogrammen (z. B. im Bereich Office-Anwendungen, Grafikbearbeitung, PDF-Authoring)
IT-Kompete

How to protect privacy in open data | Nature Human Behaviour

“When sharing research data for verification and reuse, behavioural researchers should protect participants’ privacy, particularly when studying sensitive topics. Because personally identifying data remain present in many open psychology datasets, we urge researchers to mend privacy via checks of re-identification risk before sharing data. We offer guidance for sharing responsibly….”

Full article: Establishing open science research priorities in health psychology: a research prioritisation Delphi exercise

Abstract:  Objective

Research on Open Science practices in Health Psychology is lacking. This meta-research study aimed to identify research question priorities and obtain consensus on the Top 5 prioritised research questions for Open Science in Health Psychology.

Methods and measures

An international Delphi consensus study was conducted. Twenty-three experts in Open Science and Health Psychology within the European Health Psychology Society (EHPS) suggested research question priorities to create a ‘long-list’ of items (Phase 1). Forty-three EHPS members rated the importance of these items, ranked their top five and suggested their own additional items (Phase 2). Twenty-four EHPS members received feedback on Phase 2 responses and then re-rated and re-ranked their top five research questions (Phase 3).

Results

The top five ranked research question priorities were: 1. ‘To what extent are Open Science behaviours currently practised in Health Psychology?’, 2. ‘How can we maximise the usefulness of Open Data and Open Code resources?’, 3. ‘How can Open Data be increased within Health Psychology?’, 4. ‘What interventions are effective for increasing the adoption of Open Science in Health Psychology?’ and 5. ‘How can we increase free Open Access publishing in Health Psychology?’.

Conclusion

Funding and resources should prioritise the research questions identified here.

Full article: Establishing open science research priorities in health psychology: a research prioritisation Delphi exercise

Abstract:  Objective

Research on Open Science practices in Health Psychology is lacking. This meta-research study aimed to identify research question priorities and obtain consensus on the Top 5 prioritised research questions for Open Science in Health Psychology.

Methods and measures

An international Delphi consensus study was conducted. Twenty-three experts in Open Science and Health Psychology within the European Health Psychology Society (EHPS) suggested research question priorities to create a ‘long-list’ of items (Phase 1). Forty-three EHPS members rated the importance of these items, ranked their top five and suggested their own additional items (Phase 2). Twenty-four EHPS members received feedback on Phase 2 responses and then re-rated and re-ranked their top five research questions (Phase 3).

Results

The top five ranked research question priorities were: 1. ‘To what extent are Open Science behaviours currently practised in Health Psychology?’, 2. ‘How can we maximise the usefulness of Open Data and Open Code resources?’, 3. ‘How can Open Data be increased within Health Psychology?’, 4. ‘What interventions are effective for increasing the adoption of Open Science in Health Psychology?’ and 5. ‘How can we increase free Open Access publishing in Health Psychology?’.

Conclusion

Funding and resources should prioritise the research questions identified here.

Watch the Supporting Open Science in the Promotion & Tenure Process: Lessons from the University of Maryland Webinar

“The academic promotion and tenure process establishes the incentive structure for institutions of higher education. Open science champions have long advocated for the process to better reflect important open science scholarship that is often under-valued and neglected in academia.

COS hosted a webinar on September 27, 2022, highlighting the five-year effort in the Psychology Department at the University of Maryland to adopt new guidelines that explicitly codify open science as a core criteria in tenure and promotion review. According to Dr. Michael Doughetry, Department Chair, the new policy was necessary to ensure incentives for advancement reflect the values of scientists and their institutions….”

Open science and data sharing in trauma research: Developing a trauma-informed protocol for archiving sensitive qualitative data. – PsycNET

Abstract:  Objective: The open science movement seeks to make research more transparent, and to that end, researchers are increasingly expected or required to archive their data in national repositories. In qualitative trauma research, data sharing could compromise participants’ safety, privacy, and confidentiality because narrative data can be more difficult to de-identify fully. There is little guidance in the traumatology literature regarding how to discuss data-sharing requirements with participants during the informed consent process. Within a larger research project in which we interviewed assault survivors, we developed and evaluated a protocol for informed consent for qualitative data sharing and engaging participants in data de-identification. Method: We conducted qualitative interviews with N = 32 adult sexual assault survivors regarding (a) how to conduct informed consent for data sharing, (b) whether participants should have input on sharing their data, and (c) whether they wanted to redact information from their transcripts prior to archiving. Results: No potential participants declined participation after learning about the archiving mandate. Survivors indicated that they wanted input on archiving because the interview is their story of trauma and abuse and it would be disempowering not to have control over how this information was shared and disseminated. Survivors also wanted input on this process to help guard their privacy, confidentiality, and safety. None of the participants elected to redact substantive data prior to archiving. Conclusions: Engaging participants in the archiving process is a feasible practice that is important and empowering for trauma survivors. (PsycInfo Database Record (c) 2022 APA, all rights reserved)

When open data closes the door: A critical examination of the past, present and the potential future for open data guidelines in journals – Prosser – British Journal of Social Psychology – Wiley Online Library

Abstract:  Opening data promises to improve research rigour and democratize knowledge production. But it also presents practical, theoretical, and ethical considerations for qualitative researchers in particular. Discussion about open data in qualitative social psychology predates the replication crisis. However, the nuances of this ongoing discussion have not been translated into current journal guidelines on open data. In this article, we summarize ongoing debates about open data from qualitative perspectives, and through a content analysis of 261 journals we establish the state of current journal policies for open data in the domain of social psychology. We critically discuss how current common expectations for open data may not be adequate for establishing qualitative rigour, can introduce ethical challenges, and may place those who wish to use qualitative approaches at a disadvantage in peer review and publication processes. We advise that future open data guidelines should aim to reflect the nuance of arguments surrounding data sharing in qualitative research, and move away from a universal “one-size-fits-all” approach to data sharing. This article outlines the past, present, and the potential future of open data guidelines in social-psychological journals. We conclude by offering recommendations for how journals might more inclusively consider the use of open data in qualitative methods, whilst recognizing and allowing space for the diverse perspectives, needs, and contexts of all forms of social-psychological research.