Eye-catching advances in some AI fields are not real

Curated on Posted on by Stefaan Verhulst

Matthew Hutson at Science: “Artificial intelligence (AI) just seems to get smarter and smarter. Each iPhone learns your face, voice, and habits better than the last, and the threats AI poses to privacy and jobs continue to grow. The surge reflects faster chips, more data, and better algorithms. But some of the improvement comes from tweaks rather than the core innovations their inventors claim—and some of the gains may not exist at all, says Davis Blalock, a computer science graduate student at the Massachusetts Institute of Technology (MIT). Blalock and his colleagues compared dozens of approaches to improving neural networks—software architectures that loosely mimic the brain. “Fifty papers in,” he says, “it became clear that it wasn’t obvious what the state of the art even was.”

The researchers evaluated 81 pruning algorithms, programs that make neural networks more efficient by trimming unneeded connections. All claimed superiority in slightly different ways. But they were rarely compared properly—and when the researchers tried to evaluate them side by side, there was no clear evidence of performance improvements over a 10-year period. The result, presented in March at the Machine Learning and Systems conference, surprised Blalock’s Ph.D. adviser, MIT computer scientist John Guttag, who says the uneven comparisons themselves may explain the stagnation. “It’s the old saw, right?” Guttag said. “If you can’t measure something, it’s hard to make it better.”

Researchers are waking up to the signs of shaky progress across many subfields of AI. A 2019 meta-analysis of information retrieval algorithms used in search engines concluded the “high-water mark … was actually set in 2009.” Another study in 2019 reproduced seven neural network recommendation systems, of the kind used by media streaming services. It found that six failed to outperform much simpler, nonneural algorithms developed years before, when the earlier techniques were fine-tuned, revealing “phantom progress” in the field. In another paper posted on arXiv in March, Kevin Musgrave, a computer scientist at Cornell University, took a look at loss functions, the part of an algorithm that mathematically specifies its objective. Musgrave compared a dozen of them on equal footing, in a task involving image retrieval, and found that, contrary to their developers’ claims, accuracy had not improved since 2006. “There’s always been these waves of hype,” Musgrave says….(More)”.

Why open science is critical to combatting COVID-19

Curated on Posted on by Stefaan Verhulst

Article by the OECD: “…In January 2020, 117 organisations – including journals, funding bodies, and centres for disease prevention – signed a statement titled “Sharing research data and findings relevant to the novel coronavirus outbreak”, committing to provide immediate open access for peer-reviewed publications at least for the duration of the outbreak, to make research findings available via preprint servers, and to share results immediately with the World Health Organization (WHO). This was followed in March by the Public Health Emergency COVID-19 Initiative, launched by 12 countries1 at the level of chief science advisors or equivalent, calling for open access to publications and machine-readable access to data related to COVID-19, which resulted in an even stronger commitment by publishers.

The Open COVID Pledge was launched in April 2020 by an international coalition of scientists, lawyers, and technology companies, and calls on authors to make all intellectual property (IP) under their control available, free of charge, and without encumbrances to help end the COVID-19 pandemic, and reduce the impact of the disease….

Remaining challenges

While clinical, epidemiological and laboratory data about COVID-19 is widely available, including genomic sequencing of the pathogen, a number of challenges remain:

  • All data is not sufficiently findable, accessible, interoperable and reusable (FAIR), or not yet FAIR data.
  • Sources of data tend to be dispersed, even though many pooling initiatives are under way, curation needs to be operated “on the fly”.
  • Providing access to personal health record sharing needs to be readily accessible, pending the patient’s consent. Legislation aimed at fostering interoperability and avoiding information blocking are yet to be passed in many OECD countries. Access across borders is even more difficult under current data protection frameworks in most OECD countries.
  • In order to achieve the dual objectives of respecting privacy while ensuring access to machine readable, interoperable and reusable clinical data, the Virus Outbreak Data Network (VODAN) proposes to create FAIR data repositories which could be used by incoming algorithms (virtual machines) to ask specific research questions.
  • In addition, many issues arise around the interpretation of data – this can be illustrated by the widely followed epidemiological statistics. Typically, the statistics concern “confirmed cases”, “deaths” and “recoveries”. Each of these items seem to be treated differently in different countries, and are sometimes subject to methodological changes within the same country.
  • Specific standards for COVID-19 data therefore need to be established, and this is one of the priorities of the UK COVID-19 Strategy. A working group within Research Data Alliance has been set up to propose such standards at an international level.
  • In some cases it could be inferred that the transparency of the statistics may have guided governments to restrict testing in order to limit the number of “confirmed cases” and avoid the rapid rise of numbers. Lower testing rates can in turn reduce the efficiency of quarantine measures, lowering the overall efficiency of combating the disease….(More)”.

Research methods to consider in a pandemic

Curated on Posted on by Stefaan Verhulst

Blog by Helen Kara: “Since lockdown began, researchers have been discussing how best to change our methods. Of the ‘big three’ – questionnaires, interviews, and focus groups – only questionnaires are still being used in much the same way. There are no face-to-face interviews or focus groups, though interviews can still be held by telephone and both can be done online. However, doing research online comes with new ethical problems. Some organisations are forbidding the use of Zoom because it has had serious security problems, others are promoting the use of Jitsi because it is open source.

I’ve been thinking about appropriate methods and I have come up with three options I think are particularly worth considering at this time: documentary research, autoethnography, and digital methods. These are all comparatively new approaches and each offers scope for considerable creativity. Documentary research seems to be the oldest; I understand that its first textbook, A Matter of Record by UK academic John Scott, was published in 1990. Autoethnography was devised by US academic Carolyn Ellis in the 1990s, and digital methods have developed as technological devices have become more available to more people through the 21st century….

Doing research in a pandemic also requires considerable thought about ethics. I have long argued that ethical considerations should start at the research question, and I believe that is even more crucial at present. Does this research need doing – or does it need doing now, in the middle of a global collective trauma? If not, then don’t do that research, or postpone it until life is easier. Alternatively, you may be doing urgent research to help combat COVID19, or important research that will go towards a qualification, or have some other good reason. In which case, fine, and the next ethical question is: how can my research be done in a way that places the least burden on others? The methods introduced above all offer scope for conducting empirical research without requiring much input from other people. Right now, everyone is upset; many are worried about their health, income, housing, and/or loved ones; increasing numbers are recently bereaved. Therefore everyone is vulnerable, and so needs more care and kindness than usual. This includes potential participants and it also includes researchers. We need to choose our methods with great care for us all….(More)”.

Open science: after the COVID-19 pandemic there can be no return to closed working

Curated on Posted on by Stefaan Verhulst

Article by Virginia Barbour and Martin Borchert: “In the few months since the first case of COVID-19 was identified, the underlying cause has been isolated, its symptoms agreed on, its genome sequenced, diagnostic tests developed, and potential treatments and vaccines are on the horizon. The astonishingly short time frame of these discoveries has only happened through a global open science effort.

The principles and practices underpinning open science are what underpin good research—research that is reliable, reproducible, and has the broadest impact possible. It specifically requires the application of principles and practices that make research FAIR (Findable, Accessible, Interoperable, Reusable); researchers are making their data and preliminary publications openly accessible, and then publishers are making the peer-reviewed research immediately and freely available to all. The rapid dissemination of research—through preprints in particular as well as journal articles—stands in contrast to what happened in the 2003 SARS outbreak when the majority of research on the disease was published well after the outbreak had ended.

Many outside observers might reasonably assume, given the digital world we all now inhabit, that science usually works like this. Yet this is very far from the norm for most research. Science is not something that just happens in response to emergencies or specific events—it is an ongoing, largely publicly funded, national and international enterprise….

Sharing of the underlying data that journal articles are based on is not yet a universal requirement for publication, nor are researchers usually recognised for data sharing.

There are many benefits associated with an open science model. Image adapted from: Gaelen Pinnock/UCT; CC-BY-SA 4.0 .

Once published, even access to research is not seamless. The majority of academic journals still require a subscription to access. Subscriptions are expensive; Australian universities alone currently spend more than $300 million per year on subscriptions to academic journals. Access to academic journals also varies between universities with varying library budgets. The main markets for subscriptions to the commercial journal literature are higher education and health, with some access to government and commercial….(More)”.

Are there laws of history?

Curated on Posted on by Stefaan Verhulst

Amanda Rees at AEON: “…If big data could enable us to turn big history into mathematics rather than narratives, would that make it easier to operationalise our past? Some scientists certainly think so.

In February 2010, Peter Turchin, an ecologist from the University of Connecticut, predicted that 2020 would see a sharp increase in political volatility for Western democracies. Turchin was responding critically to the optimistic speculations of scientific progress in the journal Nature: the United States, he said, was coming to the peak of another instability spike (regularly occurring every 50 years or so), while the world economy was reaching the point of a ‘Kondratiev wave’ dip, that is, a steep downturn in a growth-driven supercycle. Along with a number of ‘seemingly disparate’ social pointers, all indications were that serious problems were looming. In the decade since that prediction, the entrenched, often vicious, social, economic and political divisions that have increasingly characterised North American and European society, have made Turchin’s ‘quantitative historical analysis’ seem remarkably prophetic.

A couple of years earlier, in July 2008, Turchin had made a series of trenchant claims about the nature and future of history. Totting up in excess of ‘200 explanations’ proposed to account for the fall of the Roman empire, he was appalled that historians were unable to agree ‘which explanations are plausible and which should be rejected’. The situation, he maintained, was ‘as risible as if, in physics, phlogiston theory and thermodynamics coexisted on equal terms’. Why, Turchin wanted to know, were the efforts in medicine and environmental science to produce healthy bodies and ecologies not mirrored by interventions to create stable societies? Surely it was time ‘for history to become an analytical, and even a predictive, science’. Knowing that historians were themselves unlikely to adopt such analytical approaches to the past, he proposed a new discipline: ‘theoretical historical social science’ or ‘cliodynamics’ – the science of history.

Like C P Snow 60 years before him, Turchin wanted to challenge the boundary between the sciences and humanities – even as periodic attempts to apply the theories of natural science to human behaviour (sociobiology, for example) or to subject natural sciences to the methodological scrutiny of the social sciences (science wars, anyone?) have frequently resulted in hostile turf wars. So what are the prospects for Turchin’s efforts to create a more desirable future society by developing a science of history?…

In 2010, Cliodynamics, the flagship journal for this new discipline, appeared, with its very first article (by the American sociologist Randall Collins) focusing on modelling victory and defeat in battle in relation to material resources and organisational morale. In a move that paralleled Comte’s earlier argument regarding the successive stages of scientific complexity (from physics, through chemistry and biology, to sociology), Turchin passionately rejected the idea that complexity made human societies unsuitable for quantitative analysis, arguing that it was precisely that complexity which made mathematics essential. Weather predictions were once considered unreliable because of the sheer complexity of managing the necessary data. But improvements in technology (satellites, computers) mean that it’s now possible to describe mathematically, and therefore to model, interactions between the system’s various parts – and therefore to know when it’s wise to carry an umbrella. With equal force, Turchin insisted that the cliodynamic approach was not deterministic. It would not predict the future, but instead lay out for governments and political leaders the likely consequences of competing policy choices.

Crucially, and again on the back of the abundantly available and cheap computer power, cliodynamics benefited from the surge in interest in the digital humanities. Existing archives were being digitised, uploaded and made searchable: every day, it seemed, more data were being presented in a format that encouraged quantification and enabled mathematical analysis – including the Old Bailey’s online database, of which Wolf had fallen foul. At the same time, cliodynamicists were repositioning themselves. Four years after its initial launch, the subtitle of their flagship journal was renamed, from The Journal of Theoretical and Mathematical History to The Journal of Quantitative History and Cultural Evolution. As Turchin’s editorial stated, this move was intended to position cliodynamics within a broader evolutionary analysis; paraphrasing the Russian-American geneticist Theodosius Dobzhansky, he claimed that ‘nothing in human history makes sense except in the light of cultural evolution’. Given Turchin’s ecological background, this evolutionary approach to history is unsurprising. But given the historical outcomes of making politics biological, it is potentially worrying….

Mathematical, data-driven, quantitative models of human experience that aim at detachment, objectivity and the capacity to develop and test hypotheses need to be balanced by explicitly fictional, qualitative and imaginary efforts to create and project a lived future that enable their audiences to empathically ground themselves in the hopes and fears of what might be to come. Both, after all, are unequivocally doing the same thing: using history and historical experience to anticipate the global future so that we might – should we so wish – avoid civilisation’s collapse. That said, the question of who ‘we’ are does, always, remain open….(More)”.

National Academies, National Science Foundation Create Network to Connect Decision-Makers with Social Scientists on Pressing COVID-19 Questions

Curated on Posted on by Stefaan Verhulst

Press Release: “The National Academies of Sciences, Engineering, and Medicine and the National Science Foundation announced today the formation of a Societal Experts Action Network (SEAN) to connect social and behavioral science researchers with decision-makers who are leading the response to COVID-19. SEAN will respond to the most pressing social, behavioral, and economic questions that are being asked by federal, state, and local officials by working with appropriate experts to quickly provide actionable answers.

The new network’s activities will be overseen by an executive committee in coordination with the National Academies’ Standing Committee on Emerging Infectious Diseases and 21st Century Health Threats, established earlier this year to provide rapid expert input on urgent questions facing the federal government on the COVID-19 pandemic. Standing committee members Robert Groves, executive vice president and provost at Georgetown University, and Mary T. Bassett, director of the François-Xavier Bagnoud Center for Health and Human Rights at Harvard University, will co-chair the executive committee to manage SEAN’s solicitation of questions and expert responses, anticipate leaders’ research needs, and guide the dissemination of network findings.

SEAN will include individual researchers from a broad range of disciplines as well as leading national social and behavioral science institutions. Responses to decision-maker requests may range from individual phone calls and presentations to written committee documents such as Rapid Expert Consultations.

“This pandemic has broadly impacted all aspects of life — not just our health, but our work, families, education, supply chains, and even the global environment,” said Marcia McNutt, president of the National Academy of Sciences. “Therefore, to address the myriad questions that are being raised by mayors, governors, local representatives, and other leaders, we must recruit the full range of scientific expertise from across the social, natural, and biomedical sciences.”   

“Our communities and our society at large are facing a range of complex issues on multiple fronts due to COVID-19,” said Arthur Lupia, head of the Directorate for Social, Behavioral, and Economic Sciences at the National Science Foundation. “These are human-centered issues affecting our daily lives — the education and well-being of our children, the strength of our economy, the health of our loved ones, neighbors, and so many more. Through SEAN, social and behavioral scientists will provide actionable, evidence-driven guidance to our leaders across the U.S. who are working to support our communities and speed their recovery.”…(More)”.

A data sharing method in the open web environment: Data sharing in hydrology

Curated on Posted on by Stefaan Verhulst

Paper by Jin Wang et al: “Data sharing plays a fundamental role in providing data resources for geographic modeling and simulation. Although there are many successful cases of data sharing through the web, current practices for sharing data mostly focus on data publication using metadata at the file level, which requires identifying, restructuring and synthesizing raw data files for further usage. In hydrology, because the same hydrological information is often stored in data files with different formats, modelers should identify the required information from multisource data sets and then customize data requirements for their applications. However, these data customization tasks are difficult to repeat, which leads to repetitive labor. This paper presents a data sharing method that provides a solution for data manipulation based on a structured data description model rather than raw data files. With the structured data description model, multisource hydrological data can be accessed and processed in a unified way and published as data services using a designed data server. This study also proposes a data configuration manager to customize data requirements through an interactive programming tool, which can help in using the data services. In addition, a component-based data viewer is developed for the visualization of multisource data in a sharable visualization scheme. A case study that involves sharing and applying hydrological data is designed to examine the applicability and feasibility of the proposed data sharing method….(More)”.

Epistemic Humility—Knowing Your Limits in a Pandemic

Curated on Posted on by Stefaan Verhulst

Essay by Erik Angner: “Ignorance,” wrote Charles Darwin in 1871, “more frequently begets confidence than does knowledge.”

Darwin’s insight is worth keeping in mind when dealing with the current coronavirus crisis. That includes those of us who are behavioral scientists. Overconfidence—and a lack of epistemic humility more broadly—can cause real harm.

In the middle of a pandemic, knowledge is in short supply. We don’t know how many people are infected, or how many people will be. We have much to learn about how to treat the people who are sick—and how to help prevent infection in those who aren’t. There’s reasonable disagreement on the best policies to pursue, whether about health care, economics, or supply distribution. Although scientists worldwide are working hard and in concert to address these questions, final answers are some ways away.

Another thing that’s in short supply is the realization of how little we know. Even a quick glance at social or traditional media will reveal many people who express themselves with way more confidence than they should…

Frequent expressions of supreme confidence might seem odd in light of our obvious and inevitable ignorance about a new threat. The thing about overconfidence, though, is that it afflicts most of us much of the time. That’s according to cognitive psychologists, who’ve studied the phenomenon systematically for half a century. Overconfidence has been called “the mother of all psychological biases.” The research has led to findings that are at the same time hilarious and depressing. In one classic study, for example, 93 percent of U.S. drivers claimed to be more skillful than the median—which is not possible.

“But surely,” you might object, “overconfidence is only for amateurs—experts would not behave like this.” Sadly, being an expert in some domain does not protect against overconfidence. Some research suggests that the more knowledgeable are more prone to overconfidence. In a famous study of clinical psychologists and psychology students, researchers asked a series of questions about a real person described in psychological literature. As the participants received more and more information about the case, their confidence in their judgment grew—but the quality of their judgment did not. And psychologists with a Ph.D. did no better than the students….(More)”.

We Have the Power to Destroy Ourselves Without the Wisdom to Ensure That We Don’t

Curated on Posted on by Stefaan Verhulst

EdgeCast by Toby Ord: “Lately, I’ve been asking myself questions about the future of humanity, not just about the next five years or even the next hundred years, but about everything humanity might be able to achieve in the time to come.

The past of humanity is about 200,000 years. That’s how long Homo sapiens have been around according to our current best guess (it might be a little bit longer). Maybe we should even include some of our other hominid ancestors and think about humanity somewhat more broadly. If we play our cards right, we could live hundreds of thousands of years more. In fact, there’s not much stopping us living millions of years. The typical species lives about a million years. Our 200,000 years so far would put us about in our adolescence, just old enough to be getting ourselves in trouble, but not wise enough to have thought through how we should act.

But a million years isn’t an upper bound for how long we could live. The horseshoe crab, for example, has lived for 450 million years so far. The Earth should remain habitable for at least that long. So, if we can survive as long as the horseshoe crab, we could have a future stretching millions of centuries from now. That’s millions of centuries of human progress, human achievement, and human flourishing. And if we could learn over that time how to reach out a little bit further into the cosmos to get to the planets around other stars, then we could have longer yet. If we went seven light-years at a time just making jumps of that distance, we could reach almost every star in the galaxy by continually spreading out from the new location. There are already plans in progress to send spacecraft these types of distances. If we could do that, the whole galaxy would open up to us….

Humanity is not a typical species. One of the things that most worries me is the way in which our technology might put us at risk. If we look back at the history of humanity these 2000 centuries, we see this initially gradual accumulation of knowledge and power. If you think back to the earliest humans, they weren’t that remarkable compared to the other species around them. An individual human is not that remarkable on the Savanna compared to a cheetah, or lion, or gazelle, but what set us apart was our ability to work together, to cooperate with other humans to form something greater than ourselves. It was teamwork, the ability to work together with those of us in the same tribe that let us expand to dozens of humans working together in cooperation. But much more important than that was our ability to cooperate across time, across the generations. By making small innovations and passing them on to our children, we were able to set a chain in motion wherein generations of people worked across time, slowly building up these innovations and technologies and accumulating power….(More)”.

Covid-19 Changed How the World Does Science, Together

Curated on Posted on by Stefaan Verhulst

Matt Apuzzo and David D. Kirkpatrick at The New York Times: “…Normal imperatives like academic credit have been set aside. Online repositories make studies available months ahead of journals. Researchers have identified and shared hundreds of viral genome sequences. More than 200 clinical trials have been launched, bringing together hospitals and laboratories around the globe.

“I never hear scientists — true scientists, good quality scientists — speak in terms of nationality,” said Dr. Francesco Perrone, who is leading a coronavirus clinical trial in Italy. “My nation, your nation. My language, your language. My geographic location, your geographic location. This is something that is really distant from true top-level scientists.”

On a recent morning, for example, scientists at the University of Pittsburgh discovered that a ferret exposed to Covid-19 particles had developed a high fever — a potential advance toward animal vaccine testing. Under ordinary circumstances, they would have started work on an academic journal article.

“But you know what? There is going to be plenty of time to get papers published,” said Paul Duprex, a virologist leading the university’s vaccine research. Within two hours, he said, he had shared the findings with scientists around the world on a World Health Organization conference call. “It is pretty cool, right? You cut the crap, for lack of a better word, and you get to be part of a global enterprise.”…

Several scientists said the closest comparison to this moment might be the height of the AIDS epidemic in the 1990s, when scientists and doctors locked arms to combat the disease. But today’s technology and the pace of information-sharing dwarfs what was possible three decades ago.

As a practical matter, medical scientists today have little choice but to study the coronavirus if they want to work at all. Most other laboratory research has been put on hold because of social distancing, lockdowns or work-from-home restrictions.

The pandemic is also eroding the secrecy that pervades academic medical research, said Dr. Ryan Carroll, a Harvard Medical professor who is involved in the coronavirus trial there. Big, exclusive research can lead to grants, promotions and tenure, so scientists often work in secret, suspiciously hoarding data from potential competitors, he said.

“The ability to work collaboratively, setting aside your personal academic progress, is occurring right now because it’s a matter of survival,” he said….(More)”.