Stefaan Verhulst

Why our peer review system is a toothless watchdog

Curated on April 16, 2016August 15, 2018 by Stefaan Verhulst

Ivan Oransky and Adam Marcus at StatNews: “While some — namely, journal editors and publishers — would like us to consider it the opposable thumb of scientific publishing, the key to differentiating rigor from rubbish, some of those very same people seem to think it’s good for nothing. Here is a partial list of the things that editors, publishers, and others have told the world peer review is not designed to do:

1. Detect irresponsible practices

Don’t expect peer reviewers to figure out if authors are “using public data as if it were the author’s own, submitting papers with the same content to different journals, or submitting an article that has already been published in another language without reference to the original,” said the InterAcademy Partnership, a consortium of national scientific academies.

2. Detect fraud

“Journal editors will tell you that peer review is not designed to detect fraud — clever misinformation will sail right through no matter how scrupulous the reviews,” Dan Engber wrote in Slate in 2005.

3. Pick up plagiarism

Peer review “is not designed to pick up fraud or plagiarism, so unless those are really egregious it usually doesn’t,” according to the Rett Syndrome Research Trust.

4. Spot ethics issues

“It is not the role of the reviewer to spot ethics issues in papers,” said Jaap van Harten, executive publisher of Elsevier (the world’s largest academic imprint)in a recent interview. “It is the responsibility of the author to abide by the publishing ethics rules. Let’s look at it in a different way: If a person steals a pair of shoes from a shop, is this the fault of the shop for not protecting their goods or the shoplifter for stealing them? Of course the fault lies with the shoplifter who carried out the crime in the first place.”

5. Spot statistical flaccidity

“Peer reviewers do not check all the datasets, rerun calculations of p-values, and so forth, except in the cases where statistical reviewers are involved — and even in these cases, statistical reviewers often check the methodologies used, sample some data, and move on.” So wrote Kent Anderson, who has served as a publishing exec at several top journals, including Science and the New England Journal of Medicine, in a recent blog post.

6. Prevent really bad research from seeing the light of day

Again, Kent Anderson: “Even the most rigorous peer review at a journal cannot stop a study from being published somewhere. Peer reviewers can’t stop an author from self-promoting a published work later.”

But …

Even when you lower expectations for peer review, it appears to come up short. Richard Smith, former editor of the BMJ, reviewed research showing that the system may be worse than no review at all, at least in biomedicine. “Peer review is supposed to be the quality assurance system for science, weeding out the scientifically unreliable and reassuring readers of journals that they can trust what they are reading,” Smith wrote. “In reality, however, it is ineffective, largely a lottery, anti-innovatory, slow, expensive, wasteful of scientific time, inefficient, easily abused, prone to bias, unable to detect fraud and irrelevant.”

So … what’s left? And are whatever scraps that remain worth the veneration peer review receives? Don’t write about anything that isn’t peer-reviewed, editors frequently admonish us journalists, even creating rules that make researchers afraid to talk to reporters before they’ve published. There’s a good chance it will turn out to be wrong. Oh? Greater than 50 percent? Because that’s the risk of preclinical research in biomedicine being wrong after it’s been peer-reviewed.

With friends like these, who needs peer review? In fact, we do need it, but not just only in the black box that happens before publication. We need continual scrutiny of findings, at sites such as PubMed Commons and PubPeer, in what is known as post-publication peer review. That’s where the action is, and where the scientific record actually gets corrected….(More)”

citizenscience.gov

Curated on April 14, 2016May 29, 2019 by Stefaan Verhulst

Simplexity

Curated on April 13, 2016August 3, 2018 by Stefaan Verhulst

Paper by Joshua D. Blank and Leigh Osofsky: “In recent years, federal government agencies have increasingly attempted to use plain language in written communications with the public. The Plain Writing Act of 2010, for instance, requires agencies to incorporate “clear and simple” explanations of rules and regulations into their official publications. In the tax context, as part of its “customer service” mission, the Internal Revenue Service bears a “duty to explain” the tax law to hundreds of millions of taxpayers who file tax returns each year. Proponents of the plain language movement have heralded this form of communication as leading to simplicity in tax compliance, more equitable access to federal programs and increased open government.

This Article casts plain language efforts in a different light. As we argue, rather than achieving simplicity, which would involve reform of the underlying law, the use of plain language to describe complex legal rules and regulations often yields “simplexity.” As we define it, simplexity occurs when the government presents clear and simple explanations of the law without highlighting its underlying complexity or reducing this complexity through formal legal changes. We show that in its numerous taxpayer publications, the IRS frequently uses plain language to transform complex, often ambiguous tax law into seemingly simple statements that (1) present contested tax law as clear tax rules, (2) add administrative gloss to the tax law and (3) fail to fully explain the tax law, including possible exceptions. Sometimes these plain language explanations benefit the government; at other times, they benefit taxpayers.

While simplexity offers a number of potential tax administration benefits, such as making the tax law understandable and even bolstering the IRS’s ability to collect tax revenue, it can also threaten vital values of transparency and democratic governance and can result in inequitable treatment of different taxpayers. We offer approaches for preserving some of the benefits of simplexity while also responding to some of its drawbacks. We also forecast the likely emergence of simplexity in potential future tax compliance measures, such as government-prepared tax returns, interactive tax return filing and increased third-party reporting….(More)”.

How to See Gentrification Coming

Curated on April 13, 2016October 9, 2018 by Stefaan Verhulst

Nathan Collins at Pacific Standard: “Depending on whom you ask, gentrification is either damaging, not so bad, or maybe even good for the low-income people who live in what we euphemistically call up-and-coming neighborhoods. Either way, it’d be nice for everybody to know which neighborhoods are going to get revitalized/eviscerated next. Now, computer scientists think they’ve found a way to do exactly that: Using Twitter and Foursquare, map the places visited by the most socially diverse crowds. Those, it turns out, are the most likely to gentrify.

Led by University of Cambridge graduate student Desislava Hristova, the researchers began their study by mapping out the social network of 37,722 Londoners who posted Foursquare check-ins via Twitter. Two people were presumed to be friends—connected on the social network—if they followed each other’s Twitter feeds. Next, Hristova and her colleagues built a geographical network of 42,080 restaurants, clubs, shops, apartments, and so on. Quaint though it may seem, the researchers treated two places as neighbors in the geographical network if they were, in fact, physically near each other. The team then linked the social and geographical networks using 549,797 Foursquare check-ins, each of which ties a person in the social network to a place in the geographical one.

Gentrification doesn’t start when outsiders move in; it starts when outsiders come to visit.

Using the network data, the team next constructed several measures of the social diversity of places, each of which helps distinguish between places that bring together friends versus strangers, and to distinguish between spots that attract socially diverse crowds versus a steady group of regulars. Among other things, those measures showed that places in the outer boroughs of London brought together more socially homogenous groups of people—in terms of their Foursquare check-ins, at least—compared with boroughs closer to the core.

But the real question is what social diversity has to do with gentrification. To measure that, the team used the United Kingdom’s Index of Multiple Deprivation, which takes into account income, education, environmental factors such as air quality, and more to quantify the socioeconomic state of affairs in localities across the U.K., including each of London’s 32 boroughs.

The rough pattern, according to the analysis: The most socially diverse places in London were also the most deprived. This is about the opposite of what you’d expect, based on social networks studied in isolation from geography, which indicates that, generally, the people with the most diverse social networks are the most prosperous….(More)”

Fairness in Machine Learning

Curated on April 13, 2016August 3, 2018 by Stefaan Verhulst

Presentation by Delip Rao: “…The models you create have power to get people arrested or vindicated, get loans approved or rejected, determine what interest rate should be charged for such loans, who should be shown to you in your long list of pursuits on your Tinder, what news do you read, who gets called for a job phone screen or even a college admission… the list goes on.

So what can you do about it?…

I have detailed notes for some of these slides. If you would like to follow those, try going directly to Google Slides.

Data protection laws around the world

Curated on April 13, 2016August 3, 2018 by Stefaan Verhulst

Fifth edition Handbook by DLA Piper’s Data Protection and Privacy practice: “More than ever it is crucial that organisations manage and safeguard personal information and address their risks and legal responsibilities in relation to processing personal data, to address the growing thicket of applicable data protection legislation.

A well‑constructed and comprehensive compliance program can solve these competing interests and is an important risk‑management tool.

This handbook sets out an overview of the key privacy and data protection laws and regulations across nearly 100 different jurisdictions and offers a primer to businesses as they consider this complex and increasingly important area of compliance….(More)”

Infomediaries and accountability

Curated on April 13, 2016August 3, 2018 by Stefaan Verhulst

Paper by Becky Carter: “A synthesis of what the existing evidence says (and where there are gaps) on:

1) What role might ‘infomediaries’, and specifically the media have in helping translate transparency into greater government accountability? In generating that accountability? In empowering citizens?

2) In what contexts or types of contexts do ‘infomediaries’ and media play such a facilitating role, and why?

3) What enabling factors contributed to success?

4) What role, if any, have donors had in supporting these sectors in this capacity?

5) What risks exist in this space?…(More)”

A ‘design-thinking’ approach to governing the future

Curated on April 12, 2016August 3, 2018 by Stefaan Verhulst

Bronwyn van der Merwe at The Hill: “…Government organizations are starting to realize the benefits of digital transformation to reinvent the citizen experience in the form of digital services tailored to individual needs. However, public service leaders are finding that as they move further into the digital age, they need to re-orient their internal organizations around this paradigm shift, or their investments in digital are likely to fail. This is where Design Thinking comes into play.

Design Thinking has become a proven approach to reimagining complex service or organizational issues in the private sector. This approach of user research, rapid prototyping, constant feedback and experimentation is starting to take hold in leading business, like Citrix Systems, Ebay and Google, and is slowly spilling over into government bodies.

Challenges to Adopting a Design-Led Approach

Success in implementing Design Thinking depends on disrupting embedded organizational beliefs and practices, including cultural shifts, changing attitudes toward risk and failure, and encouraging openness and collaboration. Specifically, government bodies need to consider:

Top to bottom support – any change as wide-ranging as the shift to Design Thinking requires support from the top. Those at the top of design-led organizations need to be experimenters, improvisers and networkers who lead by example and set the tone for change on the ground.
Design skills gap – talent to execute innovation is in short supply and few governments are in a financial position to outbid private sector firms on pay. But the public sector does have something to offer that private companies most often do not: the ability to do meaningful work for the public good. Public sector bodies also need to upskill their current employees – at times partnering with outside design experts.
No risk, no reward – for government agencies, it can be challenging to embrace a culture of trial and error. But Design Thinking is useless without Design Doing. Agencies need to recognize the benefits of agile prototyping, iterating and optimizing processes, and that failings early on can save millions while costing little.

What Can Government Bodies Do to Change?

Digital has paved the way for governments and the private sector to occasionally partner to solve thorny challenges. For instance, the White House brought together the U.N. Refugee Agency and crowdfunding platform Kickstarter to raise money for the Syrian relief effort. The weeklong partnership raised nearly $1.8 million for more than 7,000 people in need.

But to effectively communicate with today’s digitally-enabled citizens, there are several key principals government bodies must follow:

Plain and simple – use simple language focused on content, structure, navigation, grouping and completion. Strip away the bureaucratic, government-speak and be transparent.
Take an outside-in design approach – by considering the entire ecosystem, and using research to uncover insights, service design reveals an outside-in view of the people in the entire ecosystem.
Be sensitive – too many government services, tools and processes are opaque and cumbersome when dealing with sensitive issues, such as immigration, making a tax submission, or adopting a child. Fjord recently took a human-centered design framework to the State of Michigan by designing a system that allowed caseworkers to convey the fairness of a child support order, while delivering excellent customer service and increasing transparency and accuracy to families in the midst of an emotionally-charged separation.
Work to digitize processes and services across departments – Governments should look to organize their digital services around the needs of the people – whether they are starting a business, retiring or having a child – rather than around their own departmental structures.
Address privacy concerns – The assurance of privacy and security is a critical step to encourage adoption of digital channels….(More)”

E-Government Strategy, ICT and Innovation for Citizen Engagement

Curated on April 12, 2016August 3, 2018 by Stefaan Verhulst

Brief by Dennis Anderson, Robert Wu, Dr. June-Suh Cho, and Katja Schroeder: “This book discusses three levels of e-government and national strategies to reach a citizen-centric participatory e-government, and examines how disruptive technologies help shape the future of e-government. The authors examine how e-government can facilitate a symbiotic relationship between the government and its citizens. ICTs aid this relationship and promote transparencies so that citizens can place greater trust in the activities of their government. If a government can manage resources more effectively by better understanding the needs of its citizens, it can create a sustainable environment for citizens. Having a national strategy on ICT in government and e-government can significantly reduce government waste, corruption, and inefficiency. Businesses, CIOs and CTOs in the public sector interested in meeting sustainability requirements will find this book useful. …(More)”