Big data’s big role in humanitarian aid

Curated on Posted on by Stefaan Verhulst

Mary K. Pratt at Computerworld: “Hundreds of thousands of refugees streamed into Europe in 2015 from Syria and other Middle Eastern countries. Some estimates put the number at nearly a million.

The sheer volume of people overwhelmed European officials, who not only had to handle the volatile politics stemming from the crisis, but also had to find food, shelter and other necessities for the migrants.

Sweden, like many of its European Union counterparts, was taking in refugees. The Swedish Migration Board, which usually sees 2,500 asylum seekers in an average month, was accepting 10,000 per week.

“As you can imagine, with that number, it requires a lot of buses, food, registration capabilities to start processing all the cases and to accommodate all of those people,” says Andres Delgado, head of operational control, coordination and analysis at the Swedish Migration Board.

Despite the dramatic spike in refugees coming into the country, the migration agency managed the intake — hiring extra staff, starting the process of procuring housing early, getting supplies ready. Delgado credits a good part of that success to his agency’s use of big data and analytics that let him predict, with a high degree of accuracy, what was heading his way.

“Without having that capability, or looking at the tool every day, to assess every need, this would have crushed us. We wouldn’t have survived this,” Delgado says. “It would have been chaos, actually — nothing short of that.”

The Swedish Migration Board has been using big data and analytics for several years, as it seeks to gain visibility into immigration trends and what those trends will mean for the country…./…

“Can big data give us peace? I think the short answer is we’re starting to explore that. We’re at the very early stages, where there are shining examples of little things here and there. But we’re on that road,” says Kalev H. Leetaru, creator of the GDELT Project, or the Global Database of Events, Language and Tone, which describes itself as a comprehensive “database of human society.”

The topic is gaining traction. A 2013 report, “New Technology and the Prevention of Violence and Conflict,” from the International Peace Institute, highlights uses of telecommunications technology, including data, in several crisis situations around the world. The report emphasizes the potential these technologies hold in helping to ease tensions and address problems.

The report’s conclusion offers this idea: “Big data can be used to identify patterns and signatures associated with conflict — and those associated with peace — presenting huge opportunities for better-informed efforts to prevent violence and conflict.”

That’s welcome news to Noel Dickover. He’s the director of PeaceTech Data Networks at the PeaceTech Lab, which was created by the U.S. Institute of Peace (USIP) to advance USIP’s work on how technology, media and data help reduce violent conflict around the world.

Such work is still in the nascent stages, Dickover says, but people are excited about its potential. “We have unprecedented amounts of data on human sentiment, and we know there’s value there,” he says. “The question is how to connect it.”

Dickover is working on ways to do just that. One example is the Open Situation Room Exchange (OSRx) project, which aims to “empower greater collective impact in preventing or mitigating serious violent conflicts in particular arenas through collaboration and data-sharing.”…(More)

Improving government effectiveness: lessons from Germany

Curated on Posted on by Stefaan Verhulst

Tom Gash at Global Government Forum: “All countries face their own unique challenges but advanced democracies also have much in common: the global economic downturn, aging populations, increasingly expensive health and pension spending, and citizens who remain as hard to please as ever.

At an event last week in Bavaria, attended by representatives of Bavaria’s governing party, the Christian Social Union (CSU) and their guests, it also became clear that there is a growing consensus that governments face another common problem. They have relied for too long on traditional legislation and regulation to drive change. The consensus was that simply prescribing in law what citizens and companies can and can’t do will not solve the complex problems governments are facing, that governments cannot legislate their way to improved citizen health, wealth and wellbeing….

…a number of developments …from which both UK and international policymakers and practitioners can learn to improve government effectiveness.

  1. Behavioural economics: The Behavioural Insights Team (BIT), which span out of government in 2013 and is the subject of a new book by one of its founders and former IfG Director of Research, David Halpern, is being watched carefully by many countries abroad. Some are using its services, while others – including the New South Wales Government in Australia –are building their own skills in this area. BIT and others using similar principles have shown that using insights from social psychology – alongside an experimental approach – can help save money and improve outcomes. Well known successes include increasing the tax take through changing wording of reminder letters (work led by another IfG alumni Mike Hallsworth) and increasing pension take-up through auto-enrolment.
  2. Market design: There is an emerging field of study which is examining how algorithms can be used to match people better with services they need – particularly in cases where it is unfair or morally repugnant to let allow a free market to operate. Alvin Roth, the Harvard Professor and Nobel prize winner, writes about these ‘matching markets’ in his book Who Gets What and Why – in which he also explains how the approach can ensure that more kidneys reach compatible donors, and children find the right education.
  3. Big data: Large datasets can now be mined far more effectively, whether it is to analyse crime patterns to spot where police patrols might be useful or to understand crowd flows on public transport. The use of real-time information allows far more sophisticated deployment of public sector resources, better targeted at demand and need, and better tailored to individual preferences.
  4. Transparency: Transparency has the potential to enhance both the accountability and effectiveness of governments across the world – as shown in our latest Whitehall Monitor Annual Report. The UK government is considered a world-leader for its transparency – but there are still areas where progress has stalled, including in transparency over the costs and performance of privately provided public services.
  5. New management models: There is a growing realisation that new methods are best harnessed when supported by effective management. The Institute’s work on civil service reform highlights a range of success factors from past reforms in the UK – and the benefits of clear mechanisms for setting priorities and sticking to them, as is being attempted by governments new(ish) Implementation Taskforces and the Departmental Implementation Units currently cropping up across Whitehall. I looked overseas for a different model that clearly aligns government activities behind citizens’ concerns – in this case the example of the single non-emergency number system operating in New York City and elsewhere. This system supports a powerful, highly responsive, data-driven performance management regime. But like many performance management regimes it can risk a narrow and excessively short-term focus – so such tools must be combined with the mind-set of system stewardship that the Institute has long championed in its policymaking work.
  6. Investment in new capability: It is striking that all of these developments are supported by technological change and research insights developed outside government. But to embed new approaches in government, there appear to be benefits to incubating new capacity, either in specialist departmental teams or at the centre of government….(More)”

Crowdsourcing City Government: Using Tournaments to Improve Inspection Accuracy

Curated on Posted on by Stefaan Verhulst

Edward Glaeser, Andrew Hillis, Scott Kominers and Michael Luca in American Economic Review: Papers and Proceedings: “The proliferation of big data makes it possible to better target city services like hygiene inspections, but city governments rarely have the in-house talent needed for developing prediction algorithms. Cities could hire consultants, but a cheaper alternative is to crowdsource competence by making data public and offering a reward for the best algorithm. A simple model suggests that open tournaments dominate consulting contracts when cities can tolerate risk and when there is enough labor with low opportunity costs. We also report on an inexpensive Boston-based restaurant tournament, which yielded algorithms that proved reasonably accurate when tested “out-of-sample” on hygiene inspections….(More)”

 

Innovating and changing the policy-cycle: Policy-makers be prepared!

Curated on Posted on by Stefaan Verhulst

Marijn Janssen and Natalie Helbig in Government Information Quarterly: “Many policy-makers are struggling to understand participatory governance in the midst of technological changes. Advances in information and communication technologies (ICTs) continue to have an impact on the ways that policy-makers and citizens engage with each other throughout the policy-making process. A set of developments in the areas of opening government data, advanced analytics, visualization, simulation, and gaming, and ubiquitous citizen access using mobile and personalized applications is shaping the interactions between policy-makers and citizens. Yet the impact of these developments on the policy-makers is unclear. The changing roles and need for new capabilities required from the government are analyzed in this paper using two case studies. Salient new roles for policy-makers are outlined focused on orchestrating the policy-making process. Research directions are identified including understand the behavior of users, aggregating and analyzing content from scattered resources, and the effective use of the new tools. Understanding new policy-makers roles will help to bridge the gap between the potential of tools and technologies and the organizational realities and political contexts. We argue that many examples are available that enable learning from others, in both directions, developed countries experiences are useful for developing countries and experiences from the latter are valuable for the former countries…(More)”

Big-data analytics: the power of prediction

Curated on Posted on by Stefaan Verhulst

Rachel Willcox in Public Finance: “The ability to anticipate demands will improve planning and financial efficiency, and collecting and analysing data will enable the public sector to look ahead…

Hospitals around the country are well accustomed to huge annual rises in patient numbers as winter demand hits accident and emergency departments. But Wrightington, Wigan and Leigh NHS Foundation Trust (WWL) had to rethink service planning after unprecedented A&E demand during a sunny July 2014, which saw ambulances queuing outside the hospital. The trust now employs computer analysis to help predict and prepare for peaks in demand.

As public sector organisations grapple with ever-tighter savings targets, analysis of a broad range of historical data – big data analytics – offers an opportunity to pre-empt service requirements and so help the public sector manage demand more effectively and target scarce resources better. However, working with data to gain insight and save money is not without its challenges.

At WWL, a partnership with business support provider NHS Shared Business Services – a 50:50 joint venture between the Department of Health and technology firm Sopra Steria – resulted in a project that uses an analysis of historical data and complex algorithms to predict the most likely scenarios. In September, the partners launched HealthIntell, a suite of data reporting tools for A&E, procurement and finance.

The suite includes an application designed to help hospitals better cope with A&E pressures and meet waiting time targets. HealthIntell presents real-time data on attendances at A&E departments to doctors and other decision makers. It can predict demand on a daily and hourly basis, and allows trusts to use their own data to identify peaks and troughs – for example, the likely rise in attendances due to bad weather or major sporting events – to help deploy the right people with the right expertise at the right time….

Rikke Duus, a senior teaching fellow at University College London’s School of Management, agrees strongly that an evidence-based approach to providing services is key to efficiency gains, using data that is already available. Although the use of big data across the public sector is trailing well behind that in the private sector, pressure is mounting for it to catch up. Consumers’ experiences with private sector organisations – in particular the growing personalisation of services – is raising expectations about the sort of public services people expect to receive.

Transparency, openness and integration can benefit consumers, Duus says. “It’s about reinventing the business model to cut costs and improve efficiency. We have to use data to predict and prevent. The public-sector mindset is getting there and the huge repositories of data held across the public sector offer a great starting point, but often they don’t know how to get into it and skills are an issue,” Duus says.

Burgeoning demand for analytics expertise in retail, banking and finance has created a severe skills shortage that is allowing big-data professionals to command an average salary of £55,000 – 31% higher than the average IT position, according to a report published in November 2014 by the Tech Partnership employers’ network and business analytics company SAS. More than three quarters of posts were considered “fairly” or “very” difficult to fill, and the situation is unlikely to have eased in the interim.

Professor Robert Fildes, director of the Lancaster Centre for Forecasting, part of Lancaster University Management School, warns that public sector organisations are at a distinct disadvantage when it comes to competing for such sought-after skills.

The centre has worked on a number of public sector forecasting projects, including a Department of Health initiative to predict pay drift for its non-medical workforce and a scheme commissioned by NHS Blackpool to forecast patient activity.

“The other constraint is data,” Fildes observes. “People talk about data as if it is a uniform value. But the Department of Health doesn’t have any real data on the demand for, say, hip operations. They only have data on the operations they’ve done. The data required for analysis isn’t good enough,” he says….

Despite the challenges, projects are reaping rewards across a variety of public sector organisations. Since 2008, the London Fire Brigade (LFB) has been using software from SAS to prioritise the allocation of fire prevention resources, even pinpointing specific households most at risk of fire. The software brings together around 60 data inputs including demographic information, geographical locations, historical data, land use and deprivation levels to create lifestyle profiles for London households.

Deaths caused by fire in the capital fell by almost 50% between 2010 and 2015, according to the LFB. It attributes much of the reduction to better targeting of around 90,000 home visits the brigade carries out each year, to advise on fire safety….(More)”

 

Met Office warns of big data floods on the horizon

Curated on Posted on by Stefaan Verhulst

 at V3: “The amount of data being collected by departments and agencies mean government services will not be able to implement truly open data strategies, according to Met Office CIO Charles Ewen.

Ewen said the rapidly increasing amount of data being stored by companies and government departments mean it will not be technologically possible able to share all their data in the near future.

During a talk at the Cloud World Forum on Wednesday, he said: “The future will be bigger and bigger data. Right now we’re talking about petabytes, in the near future it will be tens of petabytes, then soon after it’ll be hundreds of petabytes and then we’ll be off into imaginary figure titles.

“We see a future where data has gotten so big the notion of open data and the idea ‘lets share our data with everybody and anybody’ just won’t work. We’re struggling to make it work already and by 2020 the national infrastructure will not exist to shift this stuff [data] around in the way anybody could access and make use of it.”

Ewen added that to deal with the shift he expects many departments and agencies will adapt their processes to become digital curators that are more selective about the data they share, to try and ensure it is useful.

“This isn’t us wrapping our arms around our data and saying you can’t see it. We just don’t see how we can share all this big data in the way you would want it,” he said.

“We see a future where a select number of high-capacity nodes become information brokers and are used to curate and manage data. These curators will be where people bring their problems. That’s the future we see.”

Ewan added that the current expectations around open data are based on misguided views about the capabilities of cloud technology to host and provide access to huge amounts of data.

“The trendy stuff out there claims to be great at everything, but don’t get carried away. We don’t see cloud as anything but capability. We’ve been using appropriate IT and what’s available to deliver our mission services for over 50 to 60 years, and cloud is playing an increasing part of that, but purely for increased capability,” he said.

“It’s just another tool. The important thing is having the skill and knowledge to not just believe vendors but to look and identify the problem and say ‘we have to solve this’.”

The Met Office CIO’s comments follow reports from other government service providers that people’s desire for open data is growing exponentially….(More)”

Algorithmic Life: Calculative Devices in the Age of Big Data

Curated on Posted on by Stefaan Verhulst

Book edited by Louise Amoore and Volha Piotukh: “This book critically explores forms and techniques of calculation that emerge with digital computation, and their implications. The contributors demonstrate that digital calculative devices matter beyond their specific functions as they progressively shape, transform and govern all areas of our life. In particular, it addresses such questions as:

  • How does the drive to make sense of, and productively use, large amounts of diverse data, inform the development of new calculative devices, logics and techniques?
  • How do these devices, logics and techniques affect our capacity to decide and to act?
  • How do mundane elements of our physical and virtual existence become data to be analysed and rearranged in complex ensembles of people and things?
  • In what ways are conventional notions of public and private, individual and population, certainty and probability, rule and exception transformed and what are the consequences?
  • How does the search for ‘hidden’ connections and patterns change our understanding of social relations and associative life?
  • Do contemporary modes of calculation produce new thresholds of calculability and computability, allowing for the improbable or the merely possible to be embraced and acted upon?
  • As contemporary approaches to governing uncertain futures seek to anticipate future events, how are calculation and decision engaged anew?

Drawing together different strands of cutting-edge research that is both theoretically sophisticated and empirically rich, this book makes an important contribution to several areas of scholarship, including the emerging social science field of software studies, and will be a vital resource for students and scholars alike….(More)”

Big Data in U.S. Agriculture

Curated on Posted on by Stefaan Verhulst

Megan Stubbs at the Congressional Research Service: “Recent media and industry reports have employed the term big data as a key to the future of increased food production and sustainable agriculture. A recent hearing on the private elements of big data in agriculture suggests that Congress too is interested in potential opportunities and challenges big data may hold. While there appears to be great interest, the subject of big data is complex and often misunderstood, especially within the context of agriculture.

There is no commonly accepted definition of the term big data. It is often used to describe a modern trend in which the combination of technology and advanced analytics creates a new way of processing information that is more useful and timely. In other words, big data is just as much about new methods for processing data as about the data themselves. It is dynamic, and when analyzed can provide a useful tool in a decisionmaking process. Most see big data in agriculture at the end use point, where farmers use precision tools to potentially create positive results like increased yields, reduced inputs, or greater sustainability. While this is certainly the more intriguing part of the discussion, it is but one aspect and does not necessarily represent a complete picture.

Both private and public big data play a key role in the use of technology and analytics that drive a producer’s evidence-based decisions. Public-level big data represent records collected, maintained, and analyzed through publicly funded sources, specifically by federal agencies (e.g., farm program participant records and weather data). Private big data represent records generated at the production level and originate with the farmer or rancher (e.g., yield, soil analysis, irrigation levels, livestock movement, and grazing rates). While discussed separately in this report, public and private big data are typically combined to create a more complete picture of an agricultural operation and therefore better decisionmaking tools.

Big data may significantly affect many aspects of the agricultural industry, although the full extent and nature of its eventual impacts remain uncertain. Many observers predict that the growth of big data will bring positive benefits through enhanced production, resource efficiency, and improved adaptation to climate change. While lauded for its potentially revolutionary applications, big data is not without issues. From a policy perspective, issues related to big data involve nearly every stage of its existence, including its collection (how it is captured), management (how it is stored and managed), and use (how it is analyzed and used). It is still unclear how big data will progress within agriculture due to technical and policy challenges, such as privacy and security, for producers and policymakers. As Congress follows the issue a number of questions may arise, including a principal one—what is the federal role?…(More)”

Predictive Analytics

Curated on Posted on by Stefaan Verhulst

Revised book by Eric Siegel: “Prediction is powered by the world’s most potent, flourishing unnatural resource: data. Accumulated in large part as the by-product of routine tasks, data is the unsalted, flavorless residue deposited en masse as organizations churn away. Surprise! This heap of refuse is a gold mine. Big data embodies an extraordinary wealth of experience from which to learn.

Predictive analytics unleashes the power of data. With this technology, the computer literally learns from data how to predict the future behavior of individuals. Perfect prediction is not possible, but putting odds on the future drives millions of decisions more effectively, determining whom to call, mail, investigate, incarcerate, set up on a date, or medicate.

In this lucid, captivating introduction — now in its Revised and Updated edition — former Columbia University professor and Predictive Analytics World founder Eric Siegel reveals the power and perils of prediction:

    • What type of mortgage risk Chase Bank predicted before the recession.
    • Predicting which people will drop out of school, cancel a subscription, or get divorced before they even know it themselves.
    • Why early retirement predicts a shorter life expectancy and vegetarians miss fewer flights.
    • Five reasons why organizations predict death — including one health insurance company.
    • How U.S. Bank and Obama for America calculated — and Hillary for America 2016 plans to calculate — the way to most strongly persuade each individual.
    • Why the NSA wants all your data: machine learning supercomputers to fight terrorism.
    • How IBM’s Watson computer used predictive modeling to answer questions and beat the human champs on TV’s Jeopardy!
    • How companies ascertain untold, private truths — how Target figures out you’re pregnant and Hewlett-Packard deduces you’re about to quit your job.
    • How judges and parole boards rely on crime-predicting computers to decide how long convicts remain in prison.
    • 183 examples from Airbnb, the BBC, Citibank, ConEd, Facebook, Ford, Google, the IRS, LinkedIn, Match.com, MTV, Netflix, PayPal, Pfizer, Spotify, Uber, UPS, Wikipedia, and more….(More)”

 

Big Data: A Tool for Inclusion or Exclusion? Understanding the Issues

Curated on Posted on by Stefaan Verhulst

Press Release: “A new report from the Federal Trade Commission outlines a number of questions for businesses to consider to help ensure that their use of big data analytics, while producing many benefits for consumers, avoids outcomes that may be exclusionary or discriminatory.

“Big data’s role is growing in nearly every area of business, affecting millions of consumers in concrete ways,” said FTC Chairwoman Edith Ramirez. “The potential benefits to consumers are significant, but businesses must ensure that their big data use does not lead to harmful exclusion or discrimination.”

The report, Big Data: A Tool for Inclusion or Exclusion? Understanding the Issues, looks specifically at big data at the end of its lifecycle – how it is used after being collected and analyzed, and draws on information from the FTC’s 2014 workshop, “Big Data: A Tool for Inclusion or Exclusion?,” as well as the Commission’s seminar on Alternative Scoring Products. The Commission also considered extensive public comments and additional public research in compiling the report.

The report highlights a number of innovative uses of big data that are providing benefits to underserved populations, including increased educational attainment, access to credit through non-traditional methods, specialized health care for underserved communities, and better access to employment.

In addition, the report looks at possible risks that could result from biases or inaccuracies about certain groups, including more individuals mistakenly denied opportunities based on the actions of others, exposing sensitive information, creating or reinforcing existing disparities, assisting in the targeting of vulnerable consumers for fraud, creating higher prices for goods and services in lower-income communities and weakening the effectiveness of consumer choice.

The report outlines some of the various laws that apply to the use of big data, especially in regards to possible issues of discrimination or exclusion, including the Fair Credit Reporting Act, FTC Act and equal opportunity laws. It also provides a range of questions for businesses to consider when they examine whether their big data programs comply with these laws.

The report also proposes four key policy questions that are drawn from research into the ways big data can both present and prevent harms. The policy questions are designed to help companies determine how best to maximize the benefit of their use of big data while limiting possible harms, by examining both practical questions of accuracy and built-in bias as well as whether the company’s use of big data raises ethical or fairness concerns….(More)”