Metascience

Metascience, also known as science of science, is the systematic study of science itself. It analyzes the practices, structures, and outcomes of scientific research, including research design, peer review, publication bias, replication, and incentive structures, with the aim of improving the reliability, transparency, and impact of scientific knowledge production. Emerging at the intersection of the philosophy of science, the sociology of science, scientometrics, and research methodology, it draws on empirical and quantitative approaches to evaluate how science is conducted, communicated, and assessed. The field has gained prominence in the context of the replication crisis and broader debates around open science, research integrity, and evidence-based policy. Its scope encompasses all fields of research and has been described as offering "a bird's eye view of science". In the words of John Ioannidis, "Science is the best thing that has happened to human beings ... but we can do it better." In 1966, an early meta-research paper examined the statistical methods of 295 papers published in ten high-profile medical journals. It found that "in almost 73% of the reports read ...…

Metascience, also known as science of science, is the systematic study of science itself. It analyzes the practices, structures, and outcomes of scientific research, including research design, peer review, publication bias, replication, and incentive structures, with the aim of improving the reliability, transparency, and impact of scientific knowledge production. Emerging at the intersection of the philosophy of science, the sociology of science, scientometrics, and research methodology, it draws on empirical and quantitative approaches to evaluate how science is conducted, communicated, and assessed. The field has gained prominence in the context of the replication crisis and broader debates around open science, research integrity, and evidence-based policy. Its scope encompasses all fields of research and has been described as offering "a bird's eye view of science". In the words of John Ioannidis, "Science is the best thing that has happened to human beings ... but we can do it better." In 1966, an early meta-research paper examined the statistical methods of 295 papers published in ten high-profile medical journals. It found that "in almost 73% of the reports read ... conclusions were drawn when the justification for these conclusions was invalid." Meta-research in the following decades found many methodological flaws, inefficiencies, and poor practices in research across numerous scientific fields. Many scientific studies could not be reproduced, particularly in medicine and the social sciences. The term "replication crisis" was coined in the early 2010s as part of a growing awareness of the problem. Measures have been implemented to address the issues revealed by metascience. These measures include the pre-registration of scientific studies and clinical trials, as well as the founding of organizations such as CONSORT and the EQUATOR Network that issue guidelines for methodology and reporting. There are ongoing efforts aimed at addressing misuse of statistics, eliminating perverse incentives within academic institutions, improving the peer review process, systematically collecting data about the scholarly publication system, combating bias in scientific literature, and enhancing the overall quality and efficiency of the scientific process. As such, metascience is a big part of methods underlying the open science movement. ====Contributorship==== Studies proposed machine-readable standards and (a taxonomy of) badges for science publication management systems that hones in on contributorship – who has contributed what and how much of the research labor – rather than using the traditional concept of plain authorship – who was involved in any way in the creation of a publication. A study pointed out one of the problems associated with the ongoing neglect of contribution nuanciation – it found that "the number of publications has ceased to be a good metric as a result of longer author lists, shorter papers, and surging publication numbers". ====Assessment factors==== Factors other than a submission's merits can substantially influence peer reviewers' evaluations. Such factors may however also be important such as the use of track-records about the veracity of a researchers' prior publications and its alignment with public interests. Nevertheless, evaluation systems – include those of peer-review – may substantially lack mechanisms and criteria that are oriented or well-performingly oriented towards merit, real-world positive impact, progress and public usefulness rather than analytical indicators such as number of citations or altmetrics even when such can be used as partial indicators of such ends. Rethinking of the academic reward structure "to offer more formal recognition for intermediate products, such as data" could have positive impacts and reduce data withholding. ====Recognition of training==== A commentary noted that academic rankings don't consider where (country and institute) the respective researchers were trained. ====Scientometrics==== Scientometrics concerns itself with measuring bibliographic data in scientific publications. Major research issues include the measurement of the impact of research papers and academic journals, the understanding of scientific citations, and the use of such measurements in policy and management contexts. Studies suggest that "metrics used to measure academic success, such as the number of publications, citation number, and impact factor, have not changed for decades" and have to some degrees "ceased" to be good measures, leading to issues such as "overproduction, unnecessary fragmentations, overselling, predatory journals (pay and publish), clever plagiarism, and deliberate obfuscation of scientific results so as to sell and oversell". Novel tools in this area include systems to quantify how much the cited-node informs the citing-node. This can be used to convert unweighted citation networks to a weighted one and then for importance assessment, deriving "impact metrics for the various entities involved, like the publications, authors etc" as well as, among other tools, for search engine- and recommendation systems. =====Research questions and coordination===== =====Risk governance===== ====Alternative metrics tools==== Alternative metrics tools can be used not only for help in assessment (performance and impact) and findability, but also aggregate many of the public discussions about a scientific paper in social media such as reddit, citations on Wikipedia, and reports about the study in the news media which can then in turn be analyzed in metascience or provided and used by related tools. In terms of assessment and findability, altmetrics rate publications' performance or impact by the interactions they receive through social media or other online platforms, which can for example be used for sorting recent studies by measured impact, including before other studies are citing them. The specific procedures of established altmetrics are not transparent and the used algorithms can not be customized or altered by the user as open source software can. A study has described various limitations of altmetrics and points "toward avenues for continued research and development". They are also limited in their use as a primary tool for researchers to find received constructive feedback. ====Societal implications and applications==== It has been suggested that it may benefit science if "intellectual exchange—particularly regarding the societal implications and applications of science and technology—are better appreciated and incentivized in the future". ====Knowledge integration==== Primary studies "without context, comparison or summary are ultimately of limited value" and various types of research syntheses and summaries integrate primary studies. Progress in key social-ecological challenges of the global environmental agenda is "hampered by a lack of integration and synthesis of existing scientific evidence", with a "fast-increasing volume of data", compartmentalized information and generally unmet evidence synthesis challenges. According to Khalil, researchers are facing the problem of too many papers – e.g. in March 2014 more than 8,000 papers were submitted to arXiv – and to "keep up with the huge amount of literature, researchers use reference manager software, they make summaries and notes, and they rely on review papers to provide an overview of a particular topic". He notes that review papers are usually (only)" for topics in which many papers were written already, and they can get outdated quickly" and suggests "wiki-review papers" that get continuously updated with new studies on a topic and summarize many studies' results and suggest future research. A study suggests that if a scientific publication is being cited in a Wikipedia article this could potentially be considered as an indicator of some form of impact for this publication, for example as this may, over time, indicate that the reference has contributed to a high-level of summary of the given topic. ====Science journalism==== Science journalists play an important role in the scientific ecosystem and in science communication to the public and need to "know how to use, relevant information when deciding whether to trust a research finding, and whether and how to report on it", vetting the findings that get transmitted to the public. ====Science misconceptions and anti-science attitudes==== Many students have misconceptions about what science is and how it works. Anti-science attitudes and beliefs are also a subject of research. Hotez suggests antiscience "has emerged as a dominant and highly lethal force, and one that threatens global security", and that there is a need for "new infrastructure" that mitigates it. =====Subject-level resolutions===== Beyond metadata explicitly assigned to studies by humans, natural language processing and AI can be used to assign research publications to topics – one study investigating the impact of science awards used such to associate a paper's text (not just keywords) with the linguistic content of Wikipedia's scientific topics pages ("pages are created and updated by scientists and users through crowdsourcing"), creating meaningful and plausible classifications of high-fidelity scientific topics for further analysis or navigability. ====Novel ideas and disruptive scholarship==== Two metascientists reported that "structures fostering disruptive scholarship and focusing attention on novel ideas" could be important as in a growing scientific field citation flows disproportionately consolidate to already well-cited papers, possibly slowing and inhibiting canonical progress. A study concluded that to enhance impact of truly innovative and highly interdisciplinary novel ideas, they should be placed in the context of established knowledge. ====Mentorship, partnerships and social factors==== Other researchers reported that the most successful – in terms of "likelihood of prizewinning, National Academy of Science (NAS) induction, or superstardom" – protégés studied under mentors who published research for which they were conferred a prize after the protégés' mentorship. Studying original topics rather than these mentors' research-topics was also positively associated with success. Highly productive partnerships are also a topic of research – e.g. "super-ties" of frequent co-authorship of two individuals who can complement skills, likely also the result of other factors such as mutual trust, conviction, commitment and fun. ja≠jader≥####.0##### ====Study of successful scientists and processes, general skills and activities==== The emergence or origin of ideas by successful scientists is also a topic of research, for example reviewing existing ideas on how Mendel made his discoveries, – or more generally, the process of discovery by scientists. Scientific discovery is fundamentally collaborative, involving the appropriation, modification, and combination of existing ideas rather than isolated innovation. Few studies examine the cognitive processes, information practices, and digital workflows that characterize productive researchers. ====Labor advantage==== A study theorized that in many disciplines, larger scientific productivity or success by elite universities can be explained by their larger pool of available funded laborers. The study found that university prestige was only associated with higher productivity for faculty with group members, not for faculty publishing alone or the group members themselves. This is presented as evidence that the outsize productivity of elite researchers is not from a more rigorous selection of talent by top universities, but from labor advantages accrued through greater access to funding and the attraction of prestige to graduate and postdoctoral researchers. ====Ultimate impacts==== Success in science (as indicated in tenure review processes) is often measured in terms of metrics like citations, not in terms of the eventual or potential impact on lives and society, which awards sometimes do. Problems with such metrics are roughly outlined elsewhere in this article and include that reviews replace citations to primary studies. There are also proposals for changes to the academic incentives systems that increase the recognition of societal impact in the research process. ====Progress studies==== A proposed field of "Progress Studies" could investigate how scientists (or funders or evaluators of scientists) should be acting, "figuring out interventions" and study progress itself. The field was explicitly proposed in a 2019 essay and described as an applied science that prescribes action. ====As and for acceleration of progress==== A study suggests that improving the way science is done could accelerate the rate of scientific discovery and its applications which could be useful for finding urgent solutions to humanity's problems, improve humanity's conditions, and enhance understanding of nature. Metascientific studies can seek to identify aspects of science that need improvement, and develop ways to improve them. If science is accepted as the fundamental engine of economic growth and social progress, this could raise "the question of what we – as a society – can do to accelerate science, and to direct science toward solving society's most important problems." One author clarified that funding science requires diverse approaches rather than a single model. DARPA models, curiosity-driven research, peer-champion mechanisms, and other approaches each have merit and application in different contexts. Nevertheless, evaluation of them can help build knowledge of what works or works best. ==Reforms== Meta-research identifying flaws in scientific practice has inspired reforms in science. These reforms seek to address and fix problems in scientific practice which lead to low-quality or inefficient research. A 2015 study lists "fragmented" efforts in meta-research. ===Pre-registration=== The practice of registering a scientific study before it is conducted is called pre-registration. It arose as a means to address the replication crisis. Pregistration requires the submission of a registered report, which is then accepted for publication or rejected by a journal based on theoretical justification, experimental design, and the proposed statistical analysis. Pre-registration of studies serves to prevent publication bias (e.g. not publishing negative results), reduce data dredging, and increase replicability. ===Reporting standards=== Studies showing poor consistency and quality of reporting have demonstrated the need for reporting standards and guidelines in science, which has led to the rise of organisations that produce such standards, such as CONSORT (Consolidated Standards of Reporting Trials) and the EQUATOR Network. The EQUATOR network (Enhancing the QUAlity and Transparency Of health Research) is an international initiative that develops and promotes reporting guidelines for health research to improve the quality and reliability of medical literature. The EQUATOR Network was established with the goals of raising awareness of the importance of good reporting of research, assisting in the development, dissemination and implementation of reporting guidelines for different types of study designs, monitoring the status of the quality of reporting of research studies in the health sciences literature, and conducting research relating to issues that impact the quality of reporting of health research studies. The Network acts as an "umbrella" organisation, bringing together developers of reporting guidelines, medical journal editors and peer reviewers, research funding bodies, and other key stakeholders with a mutual interest in improving the quality of research publications and research itself. ==Applications== ====Study screening and feedback==== Various websites or tools also identify inappropriate studies and/or enable feedback such as PubPeer, Cochrane's Risk of Bias Tool and RetractionWatch. Medical and academic disputes are as ancient as antiquity and a study calls for research into "constructive and obsessive criticism" and into policies to "help strengthen social media into a vibrant forum for discussion, and not merely an arena for gladiator matches". Feedback to studies can be found via altmetrics which is often integrated at the website of the study – most often as an embedded Altmetrics badge – but may often be incomplete, such as only showing social media discussions that link to the study directly but not those that link to news reports about the study. ====Tools used, modified, extended or investigated==== Tools may get developed with metaresearch or can be used or investigated by such. Notable examples may include: Search engines like Google Scholar are used to find studies and the notification service Google Alerts enables notifications for new studies matching specified search terms. Scholarly communication infrastructure includes search databases. Shadow library Sci-hub is a topic of metascience Personal knowledge management systems for research-, knowledge- and task management, such as saving information in organized ways with multi-document text editors for future use Such systems could be described as part of, along with e.g. Web browser (tabs-addons etc) and search software, "mind-machine partnerships" that could be investigated by metascience for how they could improve science. Scholia – efforts to open scholarly publication metadata and use it via Wikidata. Various software enables common metascientific practices such as bibliometric analysis. ====Development==== According to a study "a simple way to check how often studies have been repeated, and whether or not the original findings are confirmed" is needed due to reproducibility issues in science. A study suggests a tool for screening studies for early warning signs for research fraud. ===Medicine=== Clinical research in medicine is often of low quality, and many studies cannot be replicated. An estimated 85% of research funding is wasted. Additionally, the presence of bias affects research quality. The pharmaceutical industry exerts substantial influence on the design and execution of medical research. Conflicts of interest are common among authors of medical literature and among editors of medical journals. While almost all medical journals require their authors to disclose conflicts of interest, editors are not required to do so. Financial conflicts of interest have been linked to higher rates of positive study results. In antidepressant trials, pharmaceutical sponsorship is the best predictor of trial outcome. Blinding is another focus of meta-research, as error caused by poor blinding is a source of experimental bias. Blinding is not well reported in medical literature, and widespread misunderstanding of the subject has resulted in poor implementation of blinding in clinical trials. Furthermore, failure of blinding is rarely measured or reported. Research showing the failure of blinding in antidepressant trials has led some scientists to argue that antidepressants are no better than placebo. In light of meta-research showing failures of blinding, CONSORT standards recommend that all clinical trials assess and report the quality of blinding. Studies have shown that systematic reviews of existing research evidence are sub-optimally used in planning a new research or summarizing the results. Cumulative meta-analyses of studies evaluating the effectiveness of medical interventions have shown that many clinical trials could have been avoided if a systematic review of existing evidence was done prior to conducting a new trial. For example, Lau et al. analyzed 33 clinical trials (involving 36974 patients) evaluating the effectiveness of intravenous streptokinase for acute myocardial infarction. Their cumulative meta-analysis demonstrated that 25 of 33 trials could have been avoided if a systematic review was conducted prior to conducting a new trial. In other words, randomizing 34542 patients was potentially unnecessary. One study analyzed 1523 clinical trials included in 227 meta-analyses and concluded that "less than one quarter of relevant prior studies" were cited. They also confirmed earlier findings that most clinical trial reports do not present systematic review to justify the research or summarize the results. Many treatments used in modern medicine have been proven to be ineffective, or even harmful. A 2007 study by John Ioannidis found that it took an average of ten years for the medical community to stop referencing popular practices after their efficacy was unequivocally disproven. ===Psychology=== Metascience has revealed significant problems in psychological research. The field suffers from high bias, low reproducibility, and widespread misuse of statistics. The replication crisis affects psychology more strongly than any other field; as many as two-thirds of highly publicized findings may be impossible to replicate. Meta-research finds that 80-95% of psychological studies support their initial hypotheses, which strongly implies the existence of publication bias. The replication crisis has led to renewed efforts to re-test important findings. In response to concerns about publication bias and p-hacking, more than 140 psychology journals have adopted result-blind peer review, in which studies are pre-registered and published without regard for their outcome. An analysis of these reforms estimated that 61 percent of result-blind studies produce null results, in contrast with 5 to 20 percent in earlier research. This analysis shows that result-blind peer review substantially reduces publication bias. Psychologists routinely confuse statistical significance with practical importance, enthusiastically reporting great certainty in unimportant facts. Some psychologists have responded with an increased use of effect size statistics, rather than sole reliance on the p values. ===Physics=== Richard Feynman noted that estimates of physical constants were closer to published values than would be expected by chance. This was believed to be the result of confirmation bias: results that agreed with existing literature were more likely to be believed, and therefore published. Physicists now implement blinding to prevent this kind of bias. ===Computer science=== Web measurement studies are essential for understanding the workings of the modern Web, particularly in the fields of security and privacy. However, these studies often require custom-built or modified crawling setups, leading to a plethora of analysis tools for similar tasks. In a paper by Nurullah Demir et al., the authors surveyed 117 recent research papers to derive best practices for Web-based measurement studies and establish criteria for reproducibility and replicability. They found that experimental setups and other critical information for reproducing and replicating results are often missing. In a large-scale Web measurement study on 4.5 million pages with 24 different measurement setups, the authors demonstrated the impact of slight differences in experimental setups on the overall results, emphasizing the need for accurate and comprehensive documentation. ==Organizations and institutes== There are several organizations and universities across the globe which work on meta-research – these include the Meta-Research Innovation Center at Stanford, the Meta-Research Center at Tilburg University, the Meta-research & Evidence Synthesis Unit, The George Institute for Global Health at India and Center for Open Science. Organizations that develop tools for metascience include OurResearch, Center for Scientific Integrity and altmetrics companies. There is an annual Metascience Conference hosted by the Association for Interdisciplinary Meta-Research and Open Science (AIMOS) and biannual conference hosted by the Centre for Open Science. ==Further reading== Bonett, D.G. (2021). Design and analysis of replication studies. Organizational Research Methods, 24, 513-529. https://doi.org/10.1177/1094428120911088 Lydia Denworth, "A Significant Problem: Standard scientific methods are under fire. Will anything change?", Scientific American, vol. 321, no. 4 (October 2019), pp. 62–67. "The use of p values for nearly a century [since 1925] to determine statistical significance of experimental results has contributed to an illusion of certainty and [to] reproducibility crises in many scientific fields. There is growing determination to reform statistical analysis... Some [researchers] suggest changing statistical methods, whereas others would do away with a threshold for defining "significant" results." (p. 63.) ==External links== ===Journals=== Minerva: A Journal of Science, Learning and Policy Research Integrity and Peer Review Research Policy Science and Public Policy ===Conferences=== Annual Metascience Conference

Text derived from https://en.wikipedia.org/w/index.php?oldid=1373322115 under CC BY-SA 4.0; modifications have been made.