Discover: I was on a call with Teresa Murphy-Skorzova, Community Growth Manager for OpenSignal, an app that uses crowd-sourcing to aggregate cell phone signals and WiFi strength data throughout the world. …She explains that while cell phone networks like Verizon and AT&T measure the percent of the population that usually has coverage, OpenSignal is “measuring the experience of the user,” mapping signals from the devices themselves in real time. Individuals record their connection as they go about their day. The app recognizes that people and their cell phone devices are, well… mobile.

In reception to Teresa’s curiosity about my connection, I opened the app and pressed the start button, trying a “Speedtest”. A number begins to fluctuate on my screen. Download speed: 14.9 mbps. A new number begins to fluctuate, testing upload speed. 5.3 mbps. I felt like I had just played slots, already anticipating my next results. I tried again, and saw that my download speed was up to 17.5 mbps. I wondered what my speeds were at the coffee shops I frequent. What about in the woods where I took a hike last weekend, or in the subway tunnel where my texts rarely send?

…While individuals learn where to find their own best signals, they contribute to a much larger voice about network quality, Teresa explained. “When a user discovers an area that hasn’t been measured or when they discover an area with poor signal, they’re eager to contribute.” While users are interested in their personal signals, OpenSignal is interesting in tracking the aggregated signal of all devices of a particular location and network. Individual device data is therefore kept anonymous.

Some surprising research projects have used OpenSignal’s data to discover implications about health, the economy, and weather. In one of these projects a team at the Royal Netherlands Meteorological Institute (RNMI) collaborated with OpenSignal to expand the rain radar program. Rainfall gradually weakens reception between cell phone towers creating a space-time map of rainfall, or rain radar map, with cellular link data. RNMI looked at OpenSignal data from unlikely rain radar locations. Some areas were remote or impoverished while others had fairly arid climates. They can now determine whether rain radar is feasible on a larger scale….(More)”

Hachem, Sara et al in Research and Technologies for Society and Industry Leveraging a better tomorrow (RTSI): “While the design of smart city ICT systems of today is still largely focused on (and therefore limited to) passive sensing, the emergence of mobile crowd-sensing calls for more active citizen engagement in not only understanding but also shaping of our societies. The Urban Civics Internet of Things (IoT) middleware enables such involvement while effectively closing several feedback loops by including citizens in the decision-making process thus leading to smarter and healthier societies. We present our initial design and planned experimental evaluation of city-scale architecture components where data assimilation, actuation and citizen engagement are key enablers toward democratization of urban data, longer-term transparency, and accountability of urban development policies. All of these are building blocks of smart cities and societies….(More)”

Tony Hirst at IODC: “Whilst promoting the publication of open data is a key, indeed necessary, ingredient in driving the global open data agenda, promoting initiatives that support the use of open data is perhaps an even more pressing need….

This, then, is the first issue we need to address: improving basic levels of literacy in interpreting  – and manipulating (for example, sorting and grouping) – simple tables and charts. Sensemaking, in other words: what does the chart you’ve just produced actually say? What story does it tell? And there’s an added benefit that arises from learning to read and critique charts better – it makes you better at creating your own.

Associated with reading stories from data comes the reason for telling the story and putting the data to work. How does “data” help you make a decision, or track the impact of a particular intervention? (Your original question should also have informed the data you searched for in the first place). Here we have a need to develop basic skills in how to actually use data, from finding anomalies to hold publishers to account, to using the data as part of a positive advocacy campaign.

After a quick read, on site, of some of the stories the data might have to tell, there may be a need to do further analysis, or more elaborate visualization work. At this point, a range of technical craft skills often come into play, as well as statistical knowledge.

Many openly published datasets just aren’t that good – they’re “dirty”, full of misspellings, missing data, things in the wrong place or wrong format, even if the data they do contain is true. A significant amount of time that should be spent analyzing the data gets spent trying to clean the data set and get it into a form where it can be worked with. I would argue here that a data technician, with a wealth of craft knowledge about how to repair what is essentially a broken dataset, can play an important timesaving role here getting data into a state where an analyst can actually start to do their job analyzing the data.

But at the same time, there are a range of tools and techniques that can help the everyday user improve the quality of their data. Many of these tools require an element of programming knowledge, but less than you might at first think. In the Open University/FutureLean MOOC “Learn to Code for Data Analysis” we use an interactive notebook style of computing to show how you can use code literally one line at a time to perform powerful data cleaning, analysis, and visualization operations on a range of open datasets, including data from the World Bank and Comtrade.

Here, then, is yet another area where skills development may be required: statistical literacy. At its heart, statistics simply provide us with a range of tools for comparing sets of numbers. But knowing what comparisons to make, or the basis on which particular comparisons can be made, knowing what can be said about those comparisons or how they might be interpreted, in short, understanding what story the stats appear to be telling, can quickly become bewildering. Just as we need to improve sensemaking skills associated with reading charts, so to we need to develop skills in making sense of statistics, even if not actually producing those statistics ourselves.

As more data gets published, there are more opportunities for more people to make use of that data. In many cases, what’s likely to hold back that final data use is a skills gap: primary among these are the skills required to interpret simple datasets and the statistics associated with them associated with developing knowledge about how to make decisions or track progress based on that interpretation. However, the path to producing the statistics or visualizations used by the end-users from the originally published open data dataset may also be a windy one, requiring skills not only in analyzing data and uncovering – and then telling – the stories it contains, but also in more mundane technical operational concerns such as actually accessing, and cleaning, dirty datasets….(More)”

Book by Arthur Lupia: “Research polls, media interviews, and everyday conversations reveal an unsettling truth: citizens, while well-meaning and even passionate about current affairs, appear to know very little about politics. Hundreds of surveys document vast numbers of citizens answering even basic questions about government incorrectly. Given this unfortunate state of affairs, it is not surprising that more knowledgeable people often deride the public for its ignorance. Some experts even think that less informed citizens should stay out of politics altogether.

As Arthur Lupia shows in Uninformed, this is not constructive. At root, critics of public ignorance fundamentally misunderstand the problem. Many experts believe that simply providing people with more facts will make them more competent voters. However, these experts fail to understand how most people learn, and hence don’t really know what types of information are even relevant to voters. Feeding them information they don’t find relevant does not address the problem. In other words, before educating the public, we need to educate the educators.

Lupia offers not just a critique, though; he also has solutions. Drawing from a variety of areas of research on topics like attention span and political psychology, he shows how we can actually increase issue competence among voters in areas ranging from gun regulation to climate change. To attack the problem, he develops an arsenal of techniques to effectively convey to people information they actually care about.

Citizens sometimes lack the knowledge that they need to make competent political choices, and it is undeniable that greater knowledge can improve decision making. But we need to understand that voters either don’t care about or pay attention to much of the information that experts think is important. Uninformed provides the keys to improving political knowledge and civic competence: understanding what information is important to others and knowing how to best convey it to them….(More)”

Pew Research Center: “A year ahead of the presidential election, the American public is deeply cynical about government, politics and the nation’s elected leaders in a way that has become quite familiar.

Currently, just 19% say they can trust the government always or most of the time,among the lowest levels in the past half-century. Only 20% would describe government programs as being well-run. And elected officials are held in such low regard that 55% of the public says “ordinary Americans” would do a better job of solving national problems.

Yet at the same time, most Americans have a lengthy to-do list for this object of their frustration: Majorities want the federal government to have a major role in addressing issues ranging from terrorism and disaster response to education and the environment.

And most Americans like the way the federal government handles many of these same issues, though they are broadly critical of its handling of others – especially poverty and immigration.

A new national survey by Pew Research Center, based on more than 6,000 interviews conducted between August 27 and October 4, 2015, finds that public attitudes about government and politics defy easy categorization. The study builds upon previous reports about the government’s role and performance in 2010 and 1998. This report was made possible by The Pew Charitable Trusts, which received support for the survey from The William and Flora Hewlett Foundation.

The partisan divide over the size and scope of government remains as wide as ever: Support for smaller government endures as a Republican touchstone. Fully 80% of Republicans and Republican-leaning independents say they prefer a smaller government with fewer services, compared with just 31% of Democrats and Democratic leaners.

Yet both Republicans and Democrats favor significant government involvement on an array of specific issues. Among the public overall, majorities say the federal government should have a major role in dealing with 12 of 13 issues included in the survey, all except advancing space exploration.

There is bipartisan agreement that the federal government should play a major role in dealing with terrorism, natural disasters, food and medicine safety, and roads and infrastructure. And while the presidential campaign has exposed sharp partisan divisions over immigration policy, large majorities of both Republicans (85%) and Democrats (80%) say the government should have a major role in managing the immigration system.

But the partisan differences over government’s appropriate role are revealing – with the widest gaps on several issues relating to the social safety net….(More)

O’Leary, Daniel E. at Intelligent Systems, IEEE : “The goals of big data and privacy are fundamentally opposed to each other. Big data and knowledge discovery are aimed reducing information asymmetries between organizations and the data sources, whereas privacy is aimed at maintaining information asymmetries of data sources. A number of different definitions of privacy are used to investigate some of the tensions between different characteristics of big data and potential privacy concerns. Specifically, the author examines the consequences of unevenness in big data, digital data going from local controlled settings to uncontrolled global settings, privacy effects of reputation monitoring systems, and inferring knowledge from social media. In addition, the author briefly analyzes two other emerging sources of big data: police cameras and stingray for location information….(More)”

Matthew Woollard at LSE Impact Blog: “…Data made available for reuse, such as those in the UK Data Service collection have huge potential. They can unlock new discoveries in research, provide evidence for policy decisions and help promote core data skills in the next generation of researchers. By being part of a single infrastructure, data owners and data creators can work together with the UK Data Service – rather than duplicating efforts – to engage with the people who can drive the impact of their research further to provide real benefit to society. As a service we are also identifying new ways to understand and promote our impact, and our Impact Fellow and Director of Impact and Communications, Victoria Moody, is focusing on raising the visibility of the UK Data Service holdings and developing and promoting the use and impact of the data and resources in policy-relevant research, especially to new audiences such as policymakers, government sectors, charities, the private sector and the media…..

We are improving how we demonstrate the impact of both the Service and the data which we hold, by focusing on generating more and more authentic user corroboration. Our emphasis is on drawing together evidence about the reach and significance of the impact of our data and resources, and of the Service as a whole through our infrastructure and expertise. Headline impact indicators through which we will better understand our impact cover a range of areas (outlined above) where the Service brings efficiency to data access and re-use, benefit to its users and a financial and social return on investment.

We are working to understand more about how Service data contributes to impact by tracking the use of Service data in a range of initiatives focused on developing impact from research and by developing our insight into usage of our data by our users. Data in the collection have featured in a range of impact case studies in the Research Excellence Framework 2014. We are also developing a focus on understanding the specific beneficial effect, rather than simply that data were used in an output, that is – as it appears in policy, debate or the evidential process (although important). Early thoughts in developing this process are where (ideally) cited data can be tracked through the specific beneficial outcome and on to an evidenced effect, corroborated by the end user.

Our impact case studies demonstrate how the data have supported research which has led to policy change in a range of areas including; the development of mathematical models for Practice based Commissioning budgets for adult mental health in the UK and informing public policy on obesity; both using the Health Survey for England. Service data have also informed the development of impact around understanding public attitudes towards the police and other legal institutions using the Crime Survey for England and Wales and research to support the development of the national minimum wage using the Labour Force Survey. The cutting-edge new Demos Integration Hub maps the changing face of Britain’s diversity, revealing a mixed picture in the integration and upward mobility of ethnic minority communities and uses 2011 Census aggregate data (England and Wales) and Understanding Society….(More)”

Todd W. Schneider: “The New York City Taxi & Limousine Commission has released a staggeringly detailed historical dataset covering over 1.1 billion individual taxi trips in the city from January 2009 through June 2015. Taken as a whole, the detailed trip-level data is more than just a vast list of taxi pickup and drop off coordinates: it’s a story of New York. How bad is the rush hour traffic from Midtown to JFK? Where does the Bridge and Tunnel crowd hang out on Saturday nights? What time do investment bankers get to work? How has Uber changed the landscape for taxis? And could Bruce Willis and Samuel L. Jackson have made it from 72nd and Broadway to Wall Street in less than 30 minutes? The dataset addresses all of these questions and many more.

I mapped the coordinates of every trip to local census tracts and neighborhoods, then set about in an attempt to extract stories and meaning from the data. This post covers a lot, but for those who want to pursue more analysis on their own: everything in this post—the data, software, and code—is freely available. Full instructions to download and analyze the data for yourself are available on GitHub.

Table of Contents

1. Maps
2. The Data
3. Borough Trends, and the Rise of Uber
4. Airport Traffic
5. On the Realism of Die Hard 3
6. How Does Weather Affect Taxi and Uber Ridership?
7. NYC Late Night Taxi Index
8. The Bridge and Tunnel Crowd
9. Northside Williamsburg
10. Privacy Concerns
11. Investment Bankers
12. Parting Thoughts…(More)