CityBeat: Visualizing the Social Media Pulse of the City


CityBeat is a an academic research project set to develop an application that sources, monitors and analyzes hyper-local information from multiple social media platforms such as Instagram and Twitter in real time.

This project was led by researchers at the Jacobs Institute at Cornell Tech,  in collaboration with the The New York World (Columbia Journalism School), Rutgers University, NYU, and Columbia University….

If you are interested in the technical details, we have published several papers detailing the process of building CityBeat. Enjoy your read!

Xia C., Schwartz, R., Xie K., Krebs A., Langdon A., Ting J. and Naaman M., CityBeat: Real-time Social Media Visualization of Hyper-local City Data. In Proceedings, WWW 2014, Seoul, Korea, April 2014. [PDF]

Xie K., Xia C., Grinberg N., Schwartz R., and Naaman M., Robust detection of hyper-local events from geotagged social media data. In Proceedings of the 13th Workshop on Multimedia Data Mining in KDD, 2013. [PDF]

Schwartz, R., Naaman M., Matni, Z. (2013) Making Sense of Cities Using Social Media: Requirements for Hyper-Local Data Aggregation Tools. In Proceedings, WCMCW at ICWSM 2013, Boston, USA, July 2013. [PDF]

#OpenGovNow: Open Government and how it benefits you


#OpenGovNow:  “Open Governments are built on two things: information and participation. A government that is open, actively discloses information about what it does with its money and resources in a way that all citizens can understand. Equally important, an Open Government is one that actively involves all citizens to be participants in government decision-making. This two-way relationship between citizens and governments, in which governments and citizens share information with one another and work together, is the foundation of Open Government….

Why should I care?

The water that you drink, the public schools that children go to, the roads that you use every day: governments make those a reality. Governments and what they do affect each and every one of us. How governments operate and how they spend scarce public resources have a direct impact on our everyday lives and the future of our communities. For instance, it is estimated that $9.5 trillion US dollars are spent by governments all over the world through contracts — therefore you have a role to play in making sure that your share in that public money is not lost, stolen, or misused….
The Global Opening Government Survey was conducted as a response to the growing demand to better understand citizens’ views on the current state and the potential impact of openness. Using the innovative “random domain intercept technology,” the survey was based on a brief questionnaire and collected complete responses from over 65,000 web-enabled individuals in the first 61 member countries of the Open Government Partnership (OGP) plus India. The survey methodology, as any other, has its advantages and its limitations. More details about the methodology can be found in the Additional Resources section of this page..

Can We Build a Safer Internet?


in the New York Times: “We often take it as a given that the Internet is a cruel place, a natural haven for those who seek to harass and threaten others. But to some people, social networks are not mere conduits for our worst impulses. They’re structures whose design can influence how we behave, for good as well as for ill.

Right now, having a social media account can mean facing down a torrent of harassment — including, for some, attacks that are misogynist, racist or both. “Just as you create a space for people to use something in innovative, creative ways, there are also people who will use it for other means,” Moya Bailey, a postdoctoral fellow at Northeastern University who writes about race, gender and media, told Op-Talk. She mentioned Anita Sarkeesian, the video game critic who has faced harassment for critiquing the portrayal of women in games.

“Because she is doing that work, she becomes a target of a lot of violence and hate,” said Ms. Bailey. The rise of online communication is “a gift and a curse always. It’s always both/and.”

And the way we behave online may depend on which site we’re using. Ms. Bailey cites Tumblr as an example. “I think there’s something about Tumblr that is really attractive to social-justice folks, and the kinds of conversations that people have on Tumblr are very different from what’s possible on Facebook,” she explained. “The platforms themselves help shape the kind of content that people post to those different sites.”

The design of those platforms can also determine who sees what we post. Kate Losse, a writer on technology and culture and a former product manager at Facebook, told Op-Talk that Facebook has widened the scope of some of our conversations.

“Pre-Facebook there would be all these different kinds of interactions you might have socially,” she said. “You might talk to one person, you might talk to three people, you might talk to a hundred people. But Facebook’s interesting because you’re always talking to a hundred people when you post, or more.”

“You have to look at something like Facebook as structuring social interactions,” she added. And interacting via what Ms. Losse called “large-scale announcements” can introduce problems. “The Internet is the classic case of tragedy of the commons,” she said. “If something that’s important to me gets viewed by someone across the world, who has no attachment to me, doesn’t care about me at all, doesn’t have any reason to know me or have empathy for me, it’s much easier for that person to do something hateful with the content than to be respectful of it.”

But if platforms can structure our interactions, can they steer us toward kindness rather than toward bile? Batya Friedman, a professor at the University of Washington’s Information School who studies the relationship between technology and human priorities, thinks it’s possible. “Any time people talk to each other,” she told Op-Talk, “we have all kinds of social norms that check how we say things to each other. We give each other social cues, we tell each other when somebody’s starting to go too far.”

The question for designers of online communities, she said, is “how do we either create virtual norms that are comparable, or how do we represent those things so that people are getting those cues, so they modulate their behavior?”…”

Americans know surprisingly little about their government, survey finds


The Annenberg Public Policy Center: “Americans show great uncertainty when it comes to answering basic questions about how their government works, a national survey conducted by the Annenberg Public Policy Center of the University of Pennsylvania has found.
The survey of 1,416 adults, released for Constitution Day (Sept. 17) in conjunction with the launch of the Civics Renewal Network, found that:

  • While little more than a third of respondents (36 percent) could name all three branches of the U.S. government, just as many (35 percent) could not name a single one.
  • Just over a quarter of Americans (27 percent) know it takes a two-thirds vote of the House and Senate to override a presidential veto.
  • One in five Americans (21 percent) incorrectly thinks that a 5-4 Supreme Court decision is sent back to Congress for reconsideration.

“Although surveys reflect disapproval of the way Congress, the President and the Supreme Court are conducting their affairs, the Annenberg survey demonstrates that many know surprisingly little about these branches of government,” said Kathleen Hall Jamieson, director of the Annenberg Public Policy Center (APPC). “This survey offers dramatic evidence of the need for more and better civics education.”

The Civics Renewal Network

To address the problem, APPC and 25 other nonpartisan organizations, including the Library of Congress, the National Constitution Center, the U.S. Courts, the National Archives, and the Newseum, announced the launch of the Civics Renewal Network, a unique partnership among some of the nation’s leaders in civics education. The network offers free, high-quality resources for teachers through the one-stop website www.civicsrenewalnetwork.org….”

Creating a national citizen engagement process for energy policy


Paper by Nick Pidgeon et al in the Proceedings of the National Academy of Sciences (PNAS): “This paper examines some of the science communication challenges involved when designing and conducting public deliberation processes on issues of national importance. We take as our illustrative case study a recent research project investigating public values and attitudes toward future energy system change for the United Kingdom. National-level issues such as this are often particularly difficult to engage the public with because of their inherent complexity, derived from multiple interconnected elements and policy frames, extended scales of analysis, and different manifestations of uncertainty. With reference to the energy system project, we discuss ways of meeting a series of science communication challenges arising when engaging the public with national topics, including the need to articulate systems thinking and problem scale, to provide balanced information and policy framings in ways that open up spaces for reflection and deliberation, and the need for varied methods of facilitation and data synthesis that permit access to participants’ broader values. Although resource intensive, national-level deliberation is possible and can produce useful insights both for participants and for science policy.”

France Announces An Ambitious New Data Strategy


at TechCrunch: “After four long months of speculations and political maneuvering, the French Government finally announced that France is getting its first Chief Data Officer….
First, it’s all about pursuing Etalab’s work when it comes to open data. The small team acted as a startup and quickly iterated on its central platform and multiple side projects. It came up with pragmatic solutions to complicated public issues, such as public health data or fiscal policy simulation. France is now the fourth country in the United Nations e-government survey.
Now, the CDO will have even more official and informal legitimacy to ask other ministries to release data sets. It’s not just about following open government theories — it’s not just about releasing public data to serve the public interest. The team can also simulate new policies before they are implemented, and share recommendations with the ministries working on these new policies.
When a new policy is written, the Government should evaluate all the ins and outs of it before implementation. Citizens should expect no less from their government.
At a larger scale, this nomination is very significant for the French Government. For years, its digital strategy was mostly about finding the best way to communicate through the Internet. But when it came to creating new policies, computers couldn’t help them.
Also announced today, the Government is modernizing and unifying its digital platform between all its ministries and services — it’s never too late. The CDO team will work closely with the DISIC to design this platform — it should be a multi-year project.
Finally, the Government will invest $160 million (€125 million) to innovate in the public sector when it makes sense. In other words, the government will work with private companies (and preferably young innovative companies) to improve the infrastructure that powers the public sector.
France is the first European country to get a Chief Data Officer…”

Barriers to the Open Government Data Agenda: Taking a Multi-Level Perspective


Article by Chris Martin in “Policy & Internet“: “A loose coalition of advocates for Open Government Data (OGD) argue that such data has the potential to have a transformative impact by catalyzing innovation across sectors of the economy and by fostering democratic participation and engagement. However, there is limited evidence to date of the OGD agenda having such a transformative impact. This article applies the Multi-level Perspective (MLP); an approach more typically applied to study transitions to a sustainable society, to explore the nature of the barriers currently faced by the OGD agenda. It argues that such barriers exist in two forms: implementation barriers and barriers to use. The empirical results presented include survey responses measuring the perceptions of U.K. OGD community members of 33 barriers to the OGD data agenda. These results are analyzed to identify implications for OGD policy and practice. The article concludes with a start at establishing a theoretical grounding for the study of barriers to the OGD agenda.”

Smart Inclusive Cities: How New Apps, Big Data, and Collaborative Technologies Are Transforming Immigrant Integration


New report by Meghan Benton for the Migration Policy Institute: “The spread of smartphones—cellphones with high-speed Internet access and geolocation technology—is transforming urban life. While many smartphone apps are largely about convenience, policymakers are beginning to explore their potential to address social challenges from disaster response to public health. And cities, in North America and Europe alike, are in the vanguard in exploring creative uses for these apps, including how to improve engagement.
For disadvantaged and diverse populations, accessing city services through a smartphone can help overcome language or literacy barriers and thus increase interactions with city officials. For those with language needs, smartphones allow language training to be accessed anywhere and at any time. More broadly, cities have begun mining the rich datasets that smartphones collect, to help attune services to the needs of their whole population. A new crop of social and civic apps offer new tools to penetrate hard-to-reach populations, including newly arrived and transient groups.
While these digital developments offer promising opportunities for immigrant integration efforts, smartphone apps’ potential to address social problems should not be overstated. In spite of potential shortcomings, since immigrant integration requires a multipronged policy response, any additional tools—especially inexpensive ones—should be examined.
This report explores the kinds of opportunities smartphones and apps are creating for the immigrant integration field. It provides a first look at the opportunities and tradeoffs that smartphones and emerging technologies offer for immigrant integration, and how they might deepen—or weaken—city residents’ sense of belonging…” (Download Report)

Mapping the Next Frontier of Open Data: Corporate Data Sharing


Stefaan Verhulst at the GovLab (cross-posted at the UN Global Pulse Blog): “When it comes to data, we are living in the Cambrian Age. About ninety percent of the data that exists today has been generated within the last two years. We create 2.5 quintillion bytes of data on a daily basis—equivalent to a “new Google every four days.”
All of this means that we are certain to witness a rapid intensification in the process of “datafication”– already well underway. Use of data will grow increasingly critical. Data will confer strategic advantages; it will become essential to addressing many of our most important social, economic and political challenges.
This explains–at least in large part–why the Open Data movement has grown so rapidly in recent years. More and more, it has become evident that questions surrounding data access and use are emerging as one of the transformational opportunities of our time.
Today, it is estimated that over one million datasets have been made open or public. The vast majority of this open data is government data—information collected by agencies and departments in countries as varied as India, Uganda and the United States. But what of the terabyte after terabyte of data that is collected and stored by corporations? This data is also quite valuable, but it has been harder to access.
The topic of private sector data sharing was the focus of a recent conference organized by the Responsible Data Forum, Data and Society Research Institute and Global Pulse (see event summary). Participants at the conference, which was hosted by The Rockefeller Foundation in New York City, included representatives from a variety of sectors who converged to discuss ways to improve access to private data; the data held by private entities and corporations. The purpose for that access was rooted in a broad recognition that private data has the potential to foster much public good. At the same time, a variety of constraints—notably privacy and security, but also proprietary interests and data protectionism on the part of some companies—hold back this potential.
The framing for issues surrounding sharing private data has been broadly referred to under the rubric of “corporate data philanthropy.” The term refers to an emerging trend whereby companies have started sharing anonymized and aggregated data with third-party users who can then look for patterns or otherwise analyze the data in ways that lead to policy insights and other public good. The term was coined at the World Economic Forum meeting in Davos, in 2011, and has gained wider currency through Global Pulse, a United Nations data project that has popularized the notion of a global “data commons.”
Although still far from prevalent, some examples of corporate data sharing exist….

Help us map the field

A more comprehensive mapping of the field of corporate data sharing would draw on a wide range of case studies and examples to identify opportunities and gaps, and to inspire more corporations to allow access to their data (consider, for instance, the GovLab Open Data 500 mapping for open government data) . From a research point of view, the following questions would be important to ask:

  • What types of data sharing have proven most successful, and which ones least?
  • Who are the users of corporate shared data, and for what purposes?
  • What conditions encourage companies to share, and what are the concerns that prevent sharing?
  • What incentives can be created (economic, regulatory, etc.) to encourage corporate data philanthropy?
  • What differences (if any) exist between shared government data and shared private sector data?
  • What steps need to be taken to minimize potential harms (e.g., to privacy and security) when sharing data?
  • What’s the value created from using shared private data?

We (the GovLab; Global Pulse; and Data & Society) welcome your input to add to this list of questions, or to help us answer them by providing case studies and examples of corporate data philanthropy. Please add your examples below, use our Google Form or email them to us at corporatedata@thegovlab.org”

Ants Are Cool but Teach Us Nothing


in Bloomberg View: “…For nearly seven decades, starting in boyhood, I’ve studied hundreds of kinds of ants around the world, and this qualifies me, I believe, to offer some advice on ways their lives can be applied to ours. I’ll start with the question I’m most often asked: “What can I do about the ants in my kitchen?” My response comes from the heart: Watch your step, be careful of little lives. Ants especially like honey, tuna and cookie crumbs. So put down bits of those on the floor, and watch as the first scout finds the bait and reports back to her colony by laying an odor trail. Then, as a little column follows her out to the food, you will see social behavior so strange it might be on another planet. Think of kitchen ants not as pests or bugs, but as your personal guest superorganism.
Another question I hear a lot is, “What can we learn of moral value from the ants?” Here again I will answer definitively: nothing. Nothing at all can be learned from ants that our species should even consider imitating. For one thing, all working ants are female. Males are bred and appear in the nest only once a year, and then only briefly. They are pitiful creatures with wings, huge eyes, small brains and genitalia that make up a large portion of their rear body segment. They have only one function in life: to inseminate the virgin queens during the nuptial season. They are built to be robot flying sexual missiles. Upon mating or doing their best to mate, they are programmed to die within hours, usually as victims of predators.
Many kinds of ants eat their dead — and their injured, too. You may have seen ant workers retrieve nestmates that you have mangled or killed underfoot (accidentally, I hope), thinking it battlefield heroism. The purpose, alas, is more sinister.
As ants grow older, they spend more time in the outermost chambers and tunnels of the nest, and are more prone to undertake dangerous foraging trips. They also are the first to attack enemy ants and other intruders. Here indeed is a major difference between people and ants: While we send our young men to war, ants send their old ladies.
The most complex societies of all ant species, and arguably of all animals everywhere, are the leafcutters of the American tropics. In lowland forests and grasslands from Mexico to South America, you find conspicuous long files of reddish ants. Many carry freshly cut pieces of leaves, flowers and twigs. The ants don’t eat this vegetation. They carry it deep into their nests, where they convert it into complex, spongelike structures. On this substrate they grow a fungus, which they do eat. The entire process employs a sequence of specialists: The leafcutters in the field are medium in size. As they head home with their burdens, tiny sister ant workers ride on their backs to protect them from parasitic phorid flies. Inside the nest, workers somewhat smaller than the gatherers scissor the leaf fragments into pieces. Still smaller ants chew the fragments into lumps and add their own fecal material as fertilizer. Even smaller workers use the gooey lumps thus created to construct the gardens. And workers as small as the fly guards plant and tend the fungus.
The largest caste of leafcutter ants have razor-sharp mandibles and the adductor muscles to close them with enough force to slice mammalian skin. These soldiers defend the nest against the most dangerous predators, including anteaters.
Species that have been able to evolve superorganismic colonies — almost purely on the basis of instinct — have as a whole been enormously successful. The 20,000 or so known species of social insects make up only 2 percent of the million known species of insects but three-fourths of the insect biomass.
With complexity, however, comes vulnerability, and that brings me to one of the other superorganism superstars, the domestic honeybee. When disease strikes solitary animals that we have embraced in symbiosis, such as chickens, pigs and dogs, veterinarians can usually diagnose and fix the problem. Honeybees, on the other hand, have by far the most complex lives of all our domestic partners. There are a great many more twists and turns in their adaptation to their environment that, upon failing, can damage some part of the colony life cycle. The intractability thus far of the honeybee colony collapse disorder of Europe and North America, which threatens so much of crop pollination, may represent an intrinsic weakness of superorganisms.
You may occasionally hear human societies described as superorganisms. This is a bit of a stretch. It is true that we form societies dependent on cooperation, labor specialization and frequent acts of altruism. But where social insects are ruled almost entirely by instinct, we base labor division on transmission of culture. Also, unlike social insects, we are too selfish to behave like cells in an organism. Human beings seek their own destiny. They will always revolt against slavery, and refuse to be treated like worker ants.”