education

The Metric Tide

Curated on February 6, 2016August 3, 2018 by Stefaan Verhulst

Book by James Wilsdon: Metrics evoke a mixed reaction from the research community. A commitment to using data and evidence to inform decisions makes many of us sympathetic, even enthusiastic, about the prospect of granular, real-time analysis of our own activities. Yet we only have to look around us, at the blunt use of metrics to be reminded of the pitfalls. Metrics hold real power: they are constitutive of values, identities and livelihoods.

How to exercise that power to positive ends is the focus of this book. Using extensive evidence-gathering, analysis and consultation, the authors take a thorough look at potential uses and limitations of research metrics and indicators. They explore the use of metrics across different disciplines, assess their potential contribution to the development of research excellence and impact and consider the changing ways in which universities are using quantitative indicators in their management systems. Finally, they consider the negative or unintended effects of metrics on various aspects of research culture.

Including an updated introduction from James Wilsdon, the book proposes a framework for responsible metrics and makes a series of targeted recommendations to show how responsible metrics can be applied in research management, by funders, and in the next cycle of the Research Excellence Framework.

The metric tide is certainly rising. Unlike King Canute, we have the agency and opportunity – and in this book, a serious body of evidence – to influence how it washes through higher education and research….(More)”.

Direct democracy may be key to a happier American democracy

Curated on January 30, 2016August 15, 2018 by Stefaan Verhulst

Benjamin Radcliff and Michael Krassa in the Conversation: “Is American democracy still “by the people, for the people?” According to recent research, it may not be. Martin Gilens at Princeton University confirms that the wishes of the American working and middle class play essentially no role in our nation’s policy making. A BBC story rightly summarized this with the headline: US Is an Oligarchy, Not a Democracy.

However new research by Benjamin Radcliff and Gregory Shufeldt suggests a ray of hope.

Ballot initiatives, they argue, may better serve the interests of ordinary Americans than laws passed by elected officials….

Today, 24 states allow citizens to directly vote on policy matters.

This year, more than 42 initiatives already are approved for the ballot in 18 states.

Voters in California will decide diverse questions including banning plastic bags, voter approval of state expenses greater than US$2 billion dollars, improving school funding, and the future of bilingual education.

The people of Colorado will vote on replacing their current medical insurance programs with a single payer system, and in Massachusetts people may consider legalizing recreational marijuana….

However, many have pointed to problems with direct democracy in the form of ballot initiatives.

Maxwell Sterns at the University of Maryland, for example, writes that legislatures are better because initiatives are the tools of special interests and minorities. In the end, initiatives are voted upon by an unrepresentative subset of the population, Sterns concludes.

Others like Richard Ellis of Willamette University argue that the time-consuming process of gathering signatures introduces a bias toward moneyed interests. Some suggest this has damaged direct democracy in California, where professional petition writers andpaid signature gatherers dominate the process. Moneyed interests also enjoy a natural advantage in having the resources that ordinary people lack to mount media campaigns to support their narrow interests.

To curb this kind of problem, bans on paying people per signature are proposed in many states, but have not yet passed any legislature. However, because Californians like direct democracy in principle, they have recently amended the process to allow for a review and revision, and they require mandatory disclosures about the funding and origins of ballot initiatives.

Finally, some say initiatives can be confusing for voters, like the two recent Ohio propositions concerning marijuana, where one ballot proposition essentially canceled out the other. Similarly, Mississippi’s Initiative 42 required marking the ballot in two places for approval but only one for disapproval, resulting in numerous nullified “yes” votes.

Routes to happiness

Despite these flaws, our research shows that direct democracy might improve happiness in two ways.

One is through its psychological effect on voters, making them feel they have a direct impact on policy outcomes. This holds even if they may not like, and thus vote against, a particular proposition. The second is that it may indeed produce policies more consistent with human well being.

The psychological benefits are obvious. By allowing people literally to be the government, just as in ancient Athens, people develop higher levels of political efficacy. In short, they may feel they have some control over their lives. Direct democracy can give people political capital because it offers a means by which citizens may place issues on the ballot for popular vote, giving them an opportunity both to set the agenda and to vote on the outcome.

We think this is important today given America’s declining faith in government. Overall today only 19 percent believe the government is run for all citizens. The same percentage trusts government to mostly do what is right. The poor and working classes are even more alienated….(More)”

The Crusade Against Multiple Regression Analysis

Curated on January 26, 2016August 3, 2018 by Stefaan Verhulst

Richard Nisbett at the Edge: (VIDEO) “…The thing I’m most interested in right now has become a kind of crusade against correlational statistical analysis—in particular, what’s called multiple regression analysis. Say you want to find out whether taking Vitamin E is associated with lower prostate cancer risk. You look at the correlational evidence and indeed it turns out that men who take Vitamin E have lower risk for prostate cancer. Then someone says, “Well, let’s see if we do the actual experiment, what happens.” And what happens when you do the experiment is that Vitamin E contributes to the likelihood of prostate cancer. How could there be differences? These happen a lot. The correlational—the observational—evidence tells you one thing, the experimental evidence tells you something completely different.

In the case of health data, the big problem is something that’s come to be called the healthy user bias, because the guy who’s taking Vitamin E is also doing everything else right. A doctor or an article has told him to take Vitamin E, so he does that, but he’s also the guy who’s watching his weight and his cholesterol, gets plenty of exercise, drinks alcohol in moderation, doesn’t smoke, has a high level of education, and a high income. All of these things are likely to make you live longer, to make you less subject to morbidity and mortality risks of all kinds. You pull one thing out of that correlate and it’s going to look like Vitamin E is terrific because it’s dragging all these other good things along with it.

This is not, by any means, limited to health issues. A while back, I read a government report in The New York Times on the safety of automobiles. The measure that they used was the deaths per million drivers of each of these autos. It turns out that, for example, there are enormously more deaths per million drivers who drive Ford F150 pickups than for people who drive Volvo station wagons. Most people’s reaction, and certainly my initial reaction to it was, “Well, it sort of figures—everybody knows that Volvos are safe.”

Let’s describe two people and you tell me who you think is more likely to be driving the Volvo and who is more likely to be driving the pickup: a suburban matron in the New York area and a twenty-five-year-old cowboy in Oklahoma. It’s obvious that people are not assigned their cars. We don’t say, “Billy, you’ll be driving a powder blue Volvo station wagon.” Because of this self-selection problem, you simply can’t interpret data like that. You know virtually nothing about the relative safety of cars based on that study.

I saw in The New York Times recently an article by a respected writer reporting that people who have elaborate weddings tend to have marriages that last longer. How would that be? Maybe it’s just all the darned expense and bother—you don’t want to get divorced. It’s a cognitive dissonance thing.

Let’s think about who makes elaborate plans for expensive weddings: people who are better off financially, which is by itself a good prognosis for marriage; people who are more educated, also a better prognosis; people who are richer; people who are older—the later you get married, the more likelihood that the marriage will last, and so on.

The truth is you’ve learned nothing. It’s like saying men who are a somebody III or IV have longer-lasting marriages. Is it because of the suffix there? No, it’s because those people are the types who have a good prognosis for a lengthy marriage.

A huge range of science projects are done with multiple regression analysis. The results are often somewhere between meaningless and quite damaging….(More)”

How Measurement Fails Doctors and Teachers

Curated on January 20, 2016August 3, 2018 by Stefaan Verhulst

Robert M. Wachter at the New York Times: “Two of our most vital industries, health care and education, have become increasingly subjected to metrics and measurements. Of course, we need to hold professionals accountable. But the focus on numbers has gone too far. We’re hitting the targets, but missing the point.

Through the 20th century, we adopted a hands-off approach, assuming that the pros knew best. Most experts believed that the ideal “products” — healthy patients and well-educated kids — were too strongly influenced by uncontrollable variables (the sickness of the patient, the intellectual capacity of the student) and were too complex to be judged by the measures we use for other industries.

By the early 2000s, as evidence mounted that both fields were producing mediocre outcomes at unsustainable costs, the pressure for measurement became irresistible. In health care, we saw hundreds of thousands of deaths from medical errors, poor coordination of care and backbreaking costs. In education, it became clear that our schools were lagging behind those in other countries.

So in came the consultants and out came the yardsticks. In health care, we applied metrics to outcomes and processes. Did the doctor document that she gave the patient a flu shot? That she counseled the patient about smoking? In education, of course, the preoccupation became student test scores.

All of this began innocently enough. But the measurement fad has spun out of control. There are so many different hospital ratings that more than 1,600 medical centers can now lay claim to being included on a “top 100,” “honor roll,” grade “A” or “best” hospitals list. Burnout rates for doctors top 50 percent, far higher than other professions. A 2013 study found that the electronic health record was a dominant culprit. Another 2013 study found that emergency room doctors clicked a mouse 4,000 times during a 10-hour shift. The computer systems have become the dark force behind quality measures.

Education is experiencing its own version of measurement fatigue. Educators complain that the focus on student test performance comes at the expense of learning. Art, music and physical education have withered, because, really,why bother if they’re not on the test?…

Thoughtful and limited assessment can be effective in motivating improvements and innovations, and in weeding out the rare but disproportionately destructive bad apples.

But in creating a measurement and accountability system, we need to tone down the fervor and think harder about the unanticipated consequences….(More)”

State of the Commons

Curated on December 9, 2015August 3, 2018 by Stefaan Verhulst

Creative Commons: “Creative Commoners have known all along that collaboration, sharing, and cooperation are a driving force for human evolution. And so for many it will come as no surprise that in 2015 we achieved a tremendous milestone: over 1.1 billion CC licensed photos, videos, audio tracks, educational materials, research articles, and more have now been contributed to the shared global commons…..

Whether it’s open education, open data, science, research, music, video, photography, or public policy, we are putting sharing and collaboration at the heart of the Web. In doing so, we are much closer to realizing our vision: unlocking the full potential of the Internet to drive a new era of development, growth, and productivity.

I am proud to share with you our 2015 State of the Commons report, our best effort to measure the immeasurable scope of the commons by looking at the CC licensed content, along with content marked as public domain, that comprise the slice of the commons powered by CC tools. We are proud to be a leader in the commons movement, and we hope you will join us as we celebrate all we have accomplished together this year. ….Report at https://stateof.creativecommons.org/2015/”

ClearGov aims to bring town finances into the 21st century

Curated on December 8, 2015July 19, 2019 by Stefaan Verhulst

Dan Adams at the Boston Globe: “Earlier this year, Hopkinton resident Chris Bullock was deciding how to vote on a tax increase that would fund a new school. He wanted to know how much the town spent on education, and how that compared to other nearby towns — reasonable questions that any engaged voter might ask.

But the information was surprisingly hard to find. Charts buried in the town’s 230-page annual report were inscrutable, the money scattered across various funds labeled with jargon. Even after piecing together a few figures, Bullock had no way to make sense of them. Was his town spending a lot or only a little on education?
That frustrating exercise was the genesis of ClearGov , a startup founded by Bullock that takes towns’ raw financial data and turns them into visually appealing online infographics, along with comparisons to similar towns nearby.

For residents, Bullock said, ClearGov aims to make local governments more approachable, transparent, and accountable.

And for officials, the software should help them parse voluminous budget spreadsheets to better compare their finances to those of nearby towns.

The site also encourages officials to answer questions posted on the site by residents and annotate their numbers with plain-English explanations of the policies behindthem….

Easton is one of five Massachusetts municipalities — along with Athol, Northfield, Oxford, and Warwick — to sign up for new paid service, inking a $1,500 deal with ClearGov in September that runs through June 2016, according to town officials.

The town’s ClearGov page gives a snapshot of its population and median home values and incomes, plus graphs of Easton’s debt load and rainy day reserve fund. There’s also a detailed, per-capita breakdown of where Easton’s revenues come from…(More).

New frontiers in social innovation research

Curated on December 3, 2015August 3, 2018 by Stefaan Verhulst

Geoff Mulgan: “Nesta has published a new book with Palgrave which contains an introduction by me and many important chapters from leading academics around the world. I hope that many people will read it, and think about it, because it challenges, in a highly constructive way, many of the rather tired assumptions of the London media/political elite of both left and right.

The essay is by Roberto Mangabeira Unger, perhaps the world’s most creative and important contemporary intellectual. He is Professor of Law at Harvard (where he taught Obama); a philosopher and political theorist; author of one of the most interesting recent books on religion; co-author of an equally ground-breaking recent book on theoretical physics; and serves as strategy minister in the Brazilian government.

His argument is that a radically different way of thinking about politics, government and social change is emerging, which has either not been noticed by many political leaders, or misinterpreted. The essence of the argument is that practice is moving faster than theory; that systematic experimentation is a faster way to solve problems than clever authorship of pamphlets, white papers and plans; and that societies have the potential to be far more active agents of their own future than we assume.

The argument has implications for many fields. One is think-tanks. Twenty years ago I set up a think-tank, Demos. At that time the dominant model for policy making was to bring together some clever people in a capital city to write pamphlets, white papers and then laws. In the 1950s to 1970s a primary role was played by professors in universities, or royal commissions. Then it shifted to think-tanks. Sometimes teams within governments played a similar role – and I oversaw several of these, including the Strategy Unit in government. All saw policy as an essentially paper-based process, involving a linear transmission from abstract theories and analyses to practical implementation.

There’s still an important role to be played by think-tanks. But an opposite approach has now become common, and is promoted by Unger. In this approach, practice precedes theory. Experiment in the real world drives the development of new ideas – in business, civil society, and on the edges of the public sector. Learning by doing complements, and often leads analysis. The role of the academics and think-tanks shifts from inventing ideas to making sense of what’s emerging, and generalising it. Policies don’t try to specify every detail but rather set out broad directions and then enable a process of experiment and discovery.

As Unger shows, this approach has profound philosophical roots (reaching back to the 19th century pragmatists and beyond), and profound political implications (it’s almost opposite to the classic Marxist view, later adopted by the neoliberal right, in which intellectuals define solutions in theory which are then translated into practice). It also has profound implications for civil society – which he argues should adopt a maximalist rather than a minimalist view of social innovation.

The Unger approach doesn’t work for everything – for example, constitutional reform. But it is a superior method for improving most of the fields where governments have power – from welfare and health, to education and economic policy, and it has worked well for Nesta – evolving new models of healthcare, working with dozens of governments to redesign business policy, testing out new approaches to education.

The several hundred public sector labs and innovation teams around the world – from Chile to China, south Africa to Denmark – share this ethos too, as do many political leaders. Michael Bloomberg has been an exemplar, confident enough to innovate and experiment constantly in his time as New York Mayor. Won Soon Park in Korea is another…..

Unger’s chapter should be required reading for anyone aspiring to play a role in 21st century politics. You don’t have to agree with what he says. But you do need to work out where you disagree and why….(New Frontiers in Social Innovation Research)

The Problem-Solving Process That Prevents Groupthink

Curated on December 3, 2015August 3, 2018 by Stefaan Verhulst

Art Markman at Harvard Business Review: “There are two reasons most of us aren’t very good at creative problem solving. First, few people get training in how to be creative in their education. Second, few people understand group dynamics well enough to harness their power to help groups maximize their creativity.

Resolving the first issue requires getting your employees to learn more about the way they think… a tall order for managers. The second issue, though, is well within your ability to change.

A key element of creativity is bringing existing knowledge to bear on a new problem or goal. The more people who can engage with that problem or goal, the more knowledge that is available to work on it. Unfortunately, quite a bit of research demonstrates that the traditional brainstorming methods first described by Alex Osborn in the 1950’s fail. When groups simply get together and start throwing out ideas, they actually come up with fewer ideas overall and fewer novel, actionable ideas than the individuals in that group would have come up with had they worked alone.

To fix this problem, it is important to think about the two phases of group problem-solving: divergence and convergence.

Divergence happens when the group considers as many different potential solutions as possible. For example, a common test of creativity is the “alternative uses” test. People are asked questions like, “How many different uses can you find for a brick?” This test requires strategies for considering as many distinct solutions as possible.

Convergence happens when the variety of proposed solutions are evaluated. In this phase, a large number of ideas are whittled to a smaller set of candidate solutions to the current problem.

The core principle of group creativity is that individuals working alone diverge, while group members working together converge. In group settings, as soon as one person states a potential solution to everyone else, that influences the memory of every person in the group in ways that make everyone think about the problem more similarly. That is why groups working together diverge less than individuals working alone.

To fix group idea generation, then, be aware of when you are trying to diverge and when you are trying to converge. For example, early in the process of problem-solving, think carefully about the problem itself. Have your group members work alone to craft statements describing the problem. Then, get them back together to discuss their descriptions. The individuals are likely to come up with a variety of distinct problem statements. The group discussion will lead everyone to accept one or a small number of variants of these statements to work on – this is healthy convergence.

When you start to generate solutions, you again want divergence. Again, have people work alone to start. Then collect people’s initial ideas and send them around to other group members and allow the divergence to continue as group members individually build on the ideas of their colleagues. Because people are still working alone, the way they build on other people’s ideas is still going to be different from how other group members are building on those ideas.

After this process, you can give the resulting ideas to everyone and then let the group get together to discuss them. This discussion will gradually lead the group to converge on a small number of candidate solutions….(More)”

The ‘data revolution’ will be open

Curated on December 3, 2015August 3, 2018 by Stefaan Verhulst

Martin Tisne at Devex: “There is a huge amount of talk about a “data revolution.” The phrase emerged in the years preceding this September’s announcement of the Sustainable Development Goals, and has recently been strongly reaffirmed by the launch of a Global Partnership on Sustainable Development Data.

The importance of data in measuring, assessing and verifying the new SDGs has been powerfully made and usually includes a mention of the data needing to be “open.” However, the role of “open” has not been clearly articulated. Fundamentally, the discussion focuses on the role of data (statistics, for example) in decision-making, and not on the benefits of that data being open to the public. Until this case is made, difficult decisions to make data open will go by the wayside.

Much of the debate justly focuses on why data matters for decision-making. Knowing how many boys and girls are in primary and secondary schools, how good their education is, and the number of teachers in their schools, are examples of relevant data used in shaping education delivery, and perhaps policy. Likewise, new satellite and cellphone data can help us prevent and understand the causes of death by HIV and AIDS, tuberculosis, and malaria.

Proponents of the data revolution make powerful points, such as that 1 in 3 births go unregistered. If you are uncounted, you will be ignored. If you don’t have an identity, you do not exist.

Yet as important as this information is, I still can’t help but think: Do we change the course of history with the mere existence of more data or because people access it, mobilize and press for change?

We need an equally eloquent narrative for why open data matters and what it means.

To my thinking, we need the data to be open because we need to hold governments accountable for their promises under the SDGs, in order to incentivize action. The data needs to be available, accessible and comparable to enable journalists and civil society to prod, push and test the validity of these promises. After all, what good are the goals if governments do not deliver, beginning with the funding to implement? We will need to know what financial resources, both public and private, will be put to work and what budget allocations governments will make in their draft budgets. We need to have those debates in the open, not in smoke-filled rooms.

Second, the data needs to be open in order to be verified, quality-checked and improved. …(More)”