big data

Infoglut: How Too Much Information Is Changing the Way We Think and Know

Curated on July 11, 2013December 6, 2018 by Stefaan Verhulst

New book by Mark Andrejevic: “Today, more mediated information is available to more people than at any other time in human history. New and revitalized sense-making strategies multiply in response to the challenges of “cutting through the clutter” of competing narratives and taming the avalanche of information. Data miners, “sentiment analysts,” and decision markets offer to help bodies of data “speak for themselves”—making sense of their own patterns so we don’t have to. Neuromarketers and body language experts promise to peer behind people’s words to see what their brains are really thinking and feeling. New forms of information processing promise to displace the need for expertise and even comprehension—at least for those with access to the data.
Infoglut explores the connections between these wide-ranging sense-making strategies for an era of information overload and “big data,” and the new forms of control they enable. Andrejevic critiques the popular embrace of deconstructive debunkery, calling into question the post-truth, post-narrative, and post-comprehension politics it underwrites, and tracing a way beyond them.”

Infographics: Winds of change

Curated on July 10, 2013August 3, 2018 by Stefaan Verhulst

Book Review in the Economist:

Data Points: Visualisation That Means Something. By Nathan Yau. Wiley; 300 pages; $32 and £26.99.
Facts are Sacred. By Simon Rogers. Faber and Faber; 311 pages; £20.
The Infographic History of the World. By James Ball and Valentina D’Efilippo. Collins; 224 pages; £20.

“IN THE late 1700s William Playfair, a Scottish engineer, created the bar chart, pie chart and line graph. These amounted to visual breakthroughs, innovations that allowed people to see patterns in data that they would otherwise have missed if they just stared at long tables of numbers.
Big data, the idea that the world is replete with more information than ever, is now all the rage. And the search for fresh and enlightened ways to help people absorb it is causing a revolution. A new generation of statisticians and designers—often the same person—are working on computer technologies and visual techniques that will depict data at scales and in forms previously unimaginable. The simple line graph and pie chart are being supplemented by things like colourful, animated bubble charts, which can present more variables. Three-dimensional network diagrams show ratios and relationships that were impossible to depict before.

The Real-Time City? Big Data and Smart Urbanism

Curated on July 9, 2013December 6, 2018 by Stefaan Verhulst

New paper by Rob Kitchin from the National University of Ireland, Maynooth (NUI Maynooth) – NIRSA: “‘Smart cities’ is a term that has gained traction in academia, business and government to describe cities that, on the one hand, are increasingly composed of and monitored by pervasive and ubiquitous computing and, on the other, whose economy and governance is being driven by innovation, creativity and entrepreneurship, enacted by smart people. This paper focuses on the former and how cities are being instrumented with digital devices and infrastructure that produce ‘big data’ which enable real-time analysis of city life, new modes of technocratic urban governance, and a re-imagining of cities. The paper details a number of projects that seek to produce a real-time analysis of the city and provides a critical reflection on the implications of big data and smart urbanism”

City Data: Big, Open and Linked

Curated on July 8, 2013October 9, 2018 by Stefaan Verhulst

Working Paper by Mark S. Fox (University of Toronto): “Cities are moving towards policymaking based on data. They are publishing data using Open Data standards, linking data from disparate sources, allowing the crowd to update their data with Smart Phone Apps that use Open APIs, and applying “Big Data” Techniques to discover relationships that lead to greater efficiencies.
One Big City Data example is from New York City (Schönberger & Cukier, 2013). Building owners were illegally converting their buildings into rooming houses that contained 10 times the number people they were designed for. These buildings posed a number of problems, including fire hazards, drugs, crime, disease and pest infestations. There are over 900,000 properties in New York City and only 200 inspectors who received over 25,000 illegal conversion complaints per year. The challenge was to distinguish nuisance complaints from those worth investigating where current methods were resulting in only 13% of the inspections resulting in vacate orders.
New York’s Analytics team created a dataset that combined data from 19 agencies including buildings, preservation, police, fire, tax, and building permits. By combining data analysis with expertise gleaned from inspectors (e.g., buildings that recently received a building permit were less likely to be a problem as they were being well maintained), the team was able to develop a rating system for complaints. Based on their analysis of this data, they were able to rate complaints such that in 70% of their visits, inspectors issued vacate orders; a fivefold increase in efficiency…
This paper provides an introduction to the concepts that underlie Big City Data. It explains the concepts of Open, Unified, Linked and Grounded data that lie at the heart of the Semantic Web. It then builds on this by discussing Data Analytics, which includes Statistics, Pattern Recognition and Machine Learning. Finally we discuss Big Data as the extension of Data Analytics to the Cloud where massive amounts of computing power and storage are available for processing large data sets. We use city data to illustrate each.”

Microsensors help map crowdsourced pollution data

Curated on June 28, 2013August 15, 2018 by Stefaan Verhulst

Elena Craft in GreenBiz: Michael Heimbinder, a Brooklyn entrepreneur, hopes to empower individuals with his small-scale air quality monitoring system, AirCasting. The AirCasting system uses a mobile, Bluetooth-enabled air monitor not much larger than a smartphone to measure carbon dioxide, carbon monoxide, nitrogen dioxide, particulate matter and other pollutants. An accompanying Android app records and formats the information to an emissions map.
Alternatively, another instrument, the Air Quality Egg, comes pre-assembled ready to use. Innovative air monitoring systems, such as AirCasting or the Air Quality Egg, empower ordinary citizens to monitor the pollution they encounter daily and proactively address problematic sources of pollution.
This technology is part of a growing movement to enable the use of small sensors. In response to inquiries about small-sensor data, the EPA is researching the next generation of air measuring technologies. EPA experts are working with sensor developers to evaluate data quality and understand useful sensor applications. Through this ongoing collaboration, the EPA hopes to bolster measurements from conventional, stationary air-monitoring systems with data collected from individuals’ air quality microsensors….
Like many technologies emerging from the big data revolution and innovations in the energy sector, microsensing technology provides a wealth of high-quality data at a relatively low cost. It allows us to track previously undetected air pollution from traditional sources of urban smog, such as highways, and unconventional sources of pollution. Microsensing technology not only educates the public, but also helps to enlighten regulators so that policymakers can work from the facts to protect citizens’ health and welfare.

Capitol Words

Curated on June 27, 2013November 13, 2018 by Stefaan Verhulst

About Capitol Words: “For every day Congress is in session, Capitol Words visualizes the most frequently used words in the Congressional Record, giving you an at-a-glance view of which issues lawmakers address on a daily, weekly, monthly and yearly basis. Capitol Words lets you see what are the most popular words spoken by lawmakers on the House and Senate floor.

Methodology

The contents of the Congressional Record are downloaded daily from the website of the Government Printing Office. The GPO distributes the Congressional Record in ZIP files containing the contents of the record in plain-text format.

Each text file is parsed and turned into an XML document, with things like the title and speaker marked up. The contents of each file are then split up into words and phrases — from one word to five.

The resulting data is saved to a search engine. Capitol Words has data from 1996 to the present.”

Analyzing the Analyzers

Curated on June 27, 2013August 3, 2018 by Stefaan Verhulst

An Introspective Survey of Data Scientists and Their Work,By Harlan Harris, Sean Murphy, Marck Vaisman: “There has been intense excitement in recent years around activities labeled “data science,” “big data,” and “analytics.” However, the lack of clarity around these terms and, particularly, around the skill sets and capabilities of their practitioners has led to inefficient communication between “data scientists” and the organizations requiring their services. This lack of clarity has frequently led to missed opportunities. To address this issue, we surveyed several hundred practitioners via the Web to explore the varieties of skills, experiences, and viewpoints in the emerging data science community.

We used dimensionality reduction techniques to divide potential data scientists into five categories based on their self-ranked skill sets (Statistics, Math/Operations Research, Business, Programming, and Machine Learning/Big Data), and four categories based on their self-identification (Data Researchers, Data Businesspeople, Data Engineers, and Data Creatives). Further examining the respondents based on their division into these categories provided additional insights into the types of professional activities, educational background, and even scale of data used by different types of Data Scientists.
In this report, we combine our results with insights and data from others to provide a better understanding of the diversity of practitioners, and to argue for the value of clearer communication around roles, teams, and careers.”

Visualizing the Stunning Growth of 8 Years of OpenStreetMap

Curated on June 21, 2013May 29, 2019 by Stefaan Verhulst

Emily Badger in Atlantic Cities: “The U.S. OpenStreetMap community gathered in San Francisco over the weekend for its annual conference, the State of the Map. The loose citizen-cartography collective has now been incrementally mapping the world since 2004. While they were taking stock, it turns out the global open mapping effort has now mapped data on more than 78 million buildings and 21 million miles of road (if you wanted to drive all those roads at, say, 60 miles an hour, it would take you some 40 years to do it).
And more than a million people have chipped away at this in an impressively democratic manner: 83.6 percent of the changes in the whole database have been made by 99.9 percent of contributors.
These numbers come from the OpenStreetMap 2013 Data Report, which also contains, of course, more maps. The report, created by MapBox, includes a beautiful worldwide visualization of all the road updates made as OpenStreetMap has grown, with some of the earliest imports of data shown in green and blue, and more recent ones in white. You can navigate the full map here (scroll down), but we’ve grabbed a couple of snapshots for you as well.”