From November 6th to 17th, the campus will host a special recurring event: ‘van Torentje naar Torentje’. This is the third edition on campus of a unique series of interviews where Tubantia and the University of Twente engage in discussions with the leading candidates in the lead-up to the Dutch parliamentary elections.
There is no doubt that social psychology and its applied fields like behavioral economics and consumer psychology have a credibility problem. Many of the findings cannot be replicated because they were obtained with questionable research practices or p-hacking. QRPs are statistical tricks that help researchers to obtain p-values below the necessary threshold to claim a discovery (p < .05). To be clear, although lay people and undergraduate students consider these practices to be deceptive, fraudulent, and unscientific, they are not considered fraudulent by researchers, professional organizations, funding agencies, or universities. Demonstrating that a researchers used QRPs to obtain significant results is easy-peasy, undermines the credibility of their work, but they can keep their jobs because it is not (yet) illegal to use these practices.
The Gino-Harvard scandal is different because the DataColada team claimed that they found “four studies for which we had accumulated the strongest evidence of fraud” and that they “believe that many more Gino-authored papers contain fake data.” To lay people, it can be hard to understand the difference between allowed QRPs and forbidden fraud or data manipulation. An example of QRPs, could be selectively removing extreme values so that the difference between two groups becomes larger (e.g., removing extremely low depression scores from a control group to show a bigger treatment effect). Outright data manipulation would be switching participants with low scores from the control group to the treatment group and vice versa.
DataColada used features of the excel spreadsheet that contained the data to claim that the data were manually manipulated.
The focus is on six rows that have a strong influence on the results for all three dependent variables that were reported in the article, namely cheated or not, overreporting of performance, and deductions.
Based on the datasheet, participants in the sign-at-the-top condition (1) in rows 67, 68, and 69, did not cheat and therewith also did not overreport performance, and had very low deductions an independent measure of cheating. In contrast, participants in rows 70, 71, and 72 all cheated, had moderate amounts of overreporting, and very high deductions.
Yadi, yadi, yada, yesterday Gino posted a blog post that responded to these accusations. Personally, the most interesting rebuttal was the claim that there was no need to switch rows because the study results hold even without the flagged rows.
“Finally, recall the lack of motive for the supposed manipulation: If you re-run the entire study excluding all of the red observations (the ones that should be considered “suspicious” using Data Colada’s lens), the findings of the study still hold. Why would I manipulate data, if not to change the results of a study?“
This argument makes sense to me because fraud appears to be the last resort for researchers who are eager to present a statistically significant results. After all, nobody claims that there was no data collection as in some cases by Diederik Stapel, who committed blatant fraud around the time this article in question was published and the use of questionable research practices was rampant. When researchers conduct an actual study, they probably hope to get the desired result without QRPs or fraud. As significance requires luck, they may just hope to get lucky. When this does not work, they can use a few QRPs. When this does not work, they can just shelf the study and try again. All of this would be perfectly legal by current standards of research ethics. However, if the results are close and it is not easy to collect more data to hope for better results), it may be tempting to change a few labels of conditions to reach p < .05. And the accusation here (there are other studies) is that only 6 (or a couple more) rows were switched to get significance. However, Gino claims that the results were already significant and I agree that it makes no sense for somebody to temper with data, if the p-value is already below .05.
However, Gino did not present evidence that the results hold without the contested cases. So, I downloaded the data and took a look.
First, I was able to reproduce the published result of an ANOVA with the three conditions as categorical predictor variable and deductions as outcome variable.
In addition, the original article reported that the differences between the experimental “signature-on-top” and each of the two control conditions (“signature-on-bottom”, “no signature”) were significant. I also confirmed these results.
Now I repeated the analysis without rows 67 to 72. Without the six contested cases, the results are no longer statistically significant, F(2, 92) = 2.96, p = .057.
Interestingly, the comparisons of the experimental group with the two control groups were statistically significant.
Combining the two control groups and comparing it to the experimental group and presenting the results as a planned contrast would also have produced a significant result.
However, these results do not support Gino’s implication that the same analysis that was reported in the article would have produced a statistically significant result, p < .05, without the six contested cases. Moreover, the accusation is that she switched rows with low values to the experimental condition and rows with high values to the control condition. To simulate this scenario, I recoded the contested rows 67-69 as signature-at-the-bottom and 70-72 as signature-at-the-top and repeated the analysis. In this case, there was no evidence that the group means differed from each other, F(2,98) = 0.45, p = .637.
Experimental social psychology has a credibility crisis because researchers were (and still are) allowed to use many statistical tricks to get significant results or to hide studies that didn’t produce the desired results. The Gino scandal is only remarkable because outright manipulation of data is the only ethics violations that has personal consequences for researchers when it can be proven. Lack of evidence that fraud was committed or lack of fraud do not imply that results are credible. For example, the results in Study 2 are meaningless even without fraud because the null-hypothesis was rejected with a confidence interval that had a value close to zero as a plausible value. While the article claims to show evidence of mediation, the published data alone show that there is no empirical evidence for this claim even if p < .05 was obtained without p-hacking or fraud. Misleading claims based on weak data, however, do not violate any ethics guidelines and are a common, if not essential, part of a game called social psychology.
This blog post only examined one minor question. Gino claimed that she did not have to manipulate data because the results were already significant.
“Finally, recall the lack of motive for the supposed manipulation: If you re-run the entire study excluding all of the red observations (the ones that should be considered “suspicious” using Data Colada’s lens), the findings of the study still hold. Why would I manipulate data, if not to change the results of a study?“
My results suggest that this claim lacks empirical support. A key result was only significant with the rows of data that have been contested. Of course, this finding does not warrant the conclusion that the data were tempered with to get statistical significance. We have to wait to get the answer to this 25 million dollar question.
Abstract: Background: Research and researchers are heavily evaluated, and over the past decade it has become apparent that the consequences of evaluating the research enterprise and particularly individual researchers are considerable. This has resulted in the publishing of several guidelines and principles to support moving towards more responsible research assessment (RRA). To ensure that research evaluation is meaningful, responsible, and effective the International Network of Research Management Societies (INORMS) Research Evaluation Group created the SCOPE framework enabling evaluators to deliver on existing principles of RRA. SCOPE bridges the gap between principles and their implementation by providing a structured five-stage framework by which evaluations can be designed and implemented, as well as evaluated.
Methods: SCOPE is a step-by-step process designed to help plan, design, and conduct research evaluations as well as check effectiveness of existing evaluations. In this article, four case studies are presented to show how SCOPE has been used in practice to provide value-based research evaluation.
Results: This article situates SCOPE within the international work towards more meaningful and robust research evaluation practices and shows through the four case studies how it can be used by different organisations to develop evaluations at different levels of granularity and in different settings.
Conclusions: The article demonstrates that the SCOPE framework is rooted firmly in the existing literature. In addition, it is argued that it does not simply translate existing principles of RRA into practice, but provides additional considerations not always addressed in existing RRA principles and practices thus playing a specific role in the delivery of RRA. Furthermore, the use cases show the value of SCOPE across a range of settings, including different institutional types, sizes, and missions.
“The intersection of scholarly communication librarianship and open education offers a unique opportunity to expand knowledge of scholarly communication topics in both education and practice. Open resources can address the gap in teaching timely and critical scholarly communication topics—copyright in teaching and research environments, academic publishing, emerging modes of scholarship, impact measurement—while increasing access to resources and equitable participation in education and scholarly communication.
Scholarly Communication Librarianship and Open Knowledge is an open textbook and practitioner’s guide that collects theory, practice, and case studies from nearly 80 experts in scholarly communication and open education. Divided into three parts:
What is Scholarly Communication?
Scholarly Communication and Open Culture
Voices from the Field: Perspectives, Intersections, and Case Studies
The book delves into the economic, social, policy, and legal aspects of scholarly communication as well as open access, open data, open education, and open science and infrastructure. Practitioners provide insight into the relationship between university presses and academic libraries, defining collection development as operational scholarly communication, and promotion and tenure and the challenge for open access.
Scholarly Communication Librarianship and Open Knowledge is a thorough guide meant to increase instruction on scholarly communication and open education issues and practices so library workers can continue to meet the changing needs of students and faculty. It is also a political statement about the future to which we aspire and a challenge to the industrial, commercial, capitalistic tendencies encroaching on higher education. Students, readers, educators, and adaptors of this resource can find and embrace these themes throughout the text and embody them in their work.
This book is also available as an open access edition at https://bit.ly/SCLAOK ”
“A number of federal agencies are celebrating 2023 as a Year of Open Science. This initiative is focused on sparking change and inspiring open-science engagement through events and activities that will advance adoption of open, equitable, and secure science. The Association of Research Libraries (ARL) applauds the ongoing commitment of the US National Science Foundation (NSF) to public access through the Year of Open Science.
In 2023, through their public-access program, the NSF has funded 10 awards totaling over $1.8 million. These awards support strategic investments in critical national infrastructure for public access, such as machine-actionable data management and sharing plans (DMSPs) and a national summit for US-based research data organizations; culture change, such as the transformative work happening at MIT Library to create a fellows program for open and equitable scholarship; and a number of open-publishing initiatives….”
Today we publish “Insights into developments in European Open Education institutional policymaking” report. Drawing on insights from leading policymakers, participants of the 2021 and 2022 SPARC Europe Open Education surveys, and […]
The post HOT OFF THE PRESS: “Insights into developments in European Open Education institutional policymaking appeared first on SPARC Europe.
“Researchers worldwide grapple with the “publish or perish” dilemma, leading some to fall into predatory journal traps. The issue is especially severe in the developing world. But we have some good news to share. Under the umbrella of the InterAcademy Partnership (IAP) INYAS has joined hands with other three international science academies i.e. Bangladesh, Benin, and the Czech Republicto form a consortium. The consortium has received fund support of 50,000 USD from InterAcademy Partnership. Our mission is to create awareness and unite against predatory publishing practices and push for change at the highest levels to safeguard academic quality and integrity”
“Subscribe to Open (S2O) is an increasingly popular approach for transitioning journals from subscription to open access—without article processing charges (APCs). Under the model, a publisher commits to opening a journal’s current content contingent on the journal’s annual revenue target being met. To better scale S2O, it is possible to link S2O offers in ways that benefit both subscribing institutions (and their authors) as well as publishers. On October 10th at 12pm ET / 9am PT, the next OpenCon Library Community Call will feature Raym Crow, who developed S2O, to discuss how the model can scale to support the transition of more journals to open access.”
Language barriers and Open Access
“We are living through a time of accelerating change and transformation, where the landscape of scholarly publishing is undergoing tectonic shifts in how we operate, how we communicate, and where we add value to the research and learning lifecycles. Like all organizations dedicated to the business of information and data, content and service providers in our industry are experiencing challenges brought on by open access mandates, ongoing institutional budget crises, technological revolutions, and more.
The 2023 New Directions Seminar will focus on how those working in scholarly communications are managing commercial and cultural disruptions. Where are the sands shifting most dramatically? How are content and service providers responding to these disruptions? What are the priorities and what is being left behind? What tools and methods do we need to successfully weather these disruptive changes?…”
“FAIR-IMPACT launched our second open call for support on 30 August 2023. Applications are invited to join one of three support programmes which are targeted to our three key stakeholder groups:
Research Performing Organisations
Repositories & Data service Providers
National Level initiatives…”
“For those just getting started on their FAIR-enabling journeys, FAIR-IMPACT provides dedicated guidance and one-to-one support to successful applicants from our three key stakeholder groups to self-assess their current capabilities and to develop FAIR implementation action plans: National Levels Initiatives, Research Performing Organisations, Repositories and Data Service Providers.
Successful applicants from each of the three stakeholder groups will work over a period of nine months with FAIR-IMPACT mentors and external experts to learn how to support the uptake of FAIR-enabling practices. This will include participation in a series of virtual workshops based on the FAIR Implementation Framework, assessed homework and one-to-one support. This call is suitable for those who are just getting started with supporting the FAIR Principles or are less advances on their FAIR-enabling journey….”
Posted Sep 29, 2023 by Emmy Tsang & Kaitlin Thaney
Invest in Open Infrastructure (IOI) works to increase the investment in and adoption of open infrastructure to further equitable access to and participation in research. We do this by providing actionable, evidence-based tools to institutions and funders of open infrastructure, piloting funding mechanisms to catalyse investment and diversify funding sources for open infrastructure, and offering strategic support in partnership with open infrastructure service providers and funders.
We are thrilled to welcome five new members to our Steering Committee. The IOI Steering Committee represents community leaders, practitioners, and experts who have demonstrated success in influencing and bringing systemic change to communities. This group exists to bring new ideas, strategies, and learnings to the work of IOI from adjacent communities and areas of infrastructure.
Each of the following new members brings a wealth of experience in the open infrastructure and funding spaces, diverse perspectives, and a strong commitment to our mission to improve investment in and adoption of open infrastructure.
Joe Deville, Open Book Collective; Mattering Press; Lancaster University
Robert Karanja, Co-Develop
Louise Marston, Resolution Foundation
Eunice Mercado-Lara, Open & Equitable Program, Open Research Funders Group
Jeff Ubois, Lever for Change
[…]
Libraries are an important part of the open access publishing landscape. To achieve the open access future we’d like to see, we need to be acting now.
International Open Access Week is an opportunity for the global research community to learn about and share the benefits of Open Access, and to inspire wider participation in helping to make Open Access a new norm in scholarship and research.
This year’s theme is ‘Community over Commercialisation’ and with this in mind, Jisc and the Open Institutional Publishing Association have joined forces to discuss how libraries can come together as a community and support open access publishing initiatives.
Jisc set up the Open Access Community Framework (OACF) in response to community calls to make it simpler for libraries to support open access publishing – and other similar schemes are beginning to emerge too.
In this webinar, we will ask ‘what’s stopping us?’ and we will consider the levers that are at our disposal, as a strong and active community, including library and publisher perspectives.
As a Walled Culture explained back in 2021, open access (OA) to published academic research comes in two main varieties. “Gold” open access papers are freely available to the public because the researchers’ institutions pay “article-processing charges” to a publisher. “Green” OA papers are available because the authors self-archive their work on a personal Web site or institutional repository that is publicly accessible.
The self-archived copies are generally the accepted manuscripts, rather than the final published version, largely because academics foolishly assign copyright to the publishers. This gives the latter the power to refuse to allow members of the public to read published research they have paid for with their taxes, unless they pay again with a subscription to the journal, or on a per article basis.
You might think that is unfair and inconvenient, but easy to circumvent, because the public will be able to download copies of the peer-reviewed manuscripts that the researchers self-archive as green OA. But many publishers have a problem with the idea that people can access for free the papers in any form, and demand that public access to the green OA versions should be embargoed, typically for 12 months. There is no reason for academics to agree to this other than habit and a certain deference on their part. It’s also partly the fault of the funding agencies. The open access expert and campaigner, Peter Suber, explained in 2005 why they are to blame:
Researchers sign funding contracts with the research councils long before they sign copyright transfer agreements with publishers. Funders have a right to dictate terms, such as mandated open access, precisely because they are upstream from publishers. If one condition of the funding contract is that the grantee will deposit the peer-reviewed version of any resulting publication in an open-access repository [immediately], then publishers have no right to intervene.
Accepting embargoes on green OA at all was perhaps the biggest blunder made by the open access movement and their funders. Even today, nearly 20 years after Suber pointed out the folly of letting publishers tell academics what they can do with their own manuscripts, many publishers still demand – and get – embargoes. Against this background, ACS Publications, the publishing wing of the American Chemical Society, has come up with what it calls “Zero-Embargo Green Open Access” (pointed out by Richard Poynder):
A number of funders and institutions require authors to retain the right to post their accepted manuscripts immediately upon acceptance for publication in a journal, sometimes referred to as zero-embargo green open access (OA). More than 90% of ACS authors under these mandates have a simple and funded pathway to publish gold OA in ACS journals.
For those not covered by an institutional read and publish agreement or through other types of funding, ACS offers the option to post their accepted manuscripts with a CC BY license in open access repositories immediately upon acceptance. This option expands this small subset of authors’ choices beyond the existing option to wait 12 months to post at no cost.
Great news? Well, no, because a hefty new fee must be paid:
The article development charge (ADC) is a flat fee of $2,500 USD and is payable once the manuscript is sent for peer review. The ADC covers the cost of ACS’ pre-acceptance publishing services, from initial submission through to the final editorial decision.
That is, if academics publish a paper with the ACS, their institution must pay $2,500 for the privilege of being allowed to post immediately the accepted manuscript version on their own institutional server – something that should have been a matter of course, but was weakly given up in the early days of open access, as Suber pointed out. There is a feeble attempt to justify the cost, on the basis that the $2,500 is for “pre-acceptance publishing service”. But this apparently refers to things like peer review, which is generally conducted by fellow academics for free, and decisions by journal editors, who are often unpaid too. In general, the costs involved in “pre-acceptance publishing” are negligible.
“Zero-Embargo Green Open Access” sounds so promising. But it turns out to be yet another example of the copyright industry’s limitless sense of entitlement. Publishing is constantly finding new ways to extract money from hard-pressed academic institutions – money that could be used for more research or simply paying underfunded researchers better.
This is a personal issue for me. In 2013, I spoke at a conference celebrating the tenth anniversary of the Berlin declaration on open access. More formally, the “Berlin Declaration on Open Access to Knowledge in the Sciences and Humanities” is one of three seminal formulations of the open access idea: the other two are the Bethesda Statement (2003) and the original Budapest Open Access Initiative (2002) (all discussed in Walled Culture the book, free digital versions available). I entitled my speech “Half a Revolution”, and the slides I used can be freely downloaded from SlideShare, along with many more of my presentations.
My Berlin talk concluded with a call to action under the slogan “Zero Embargo Now” (ZEN). Back then, I looked forward to a world where all academic papers would routinely be available under green OA immediately, without any embargo. I’m still waiting.
Follow me @glynmoody on Mastodon. This post originally appeared on Walled Culture.