Dataset Inventorying Tool


at US Open Data: “Today we’re releasing Let Me Get That Data For You (LMGTDFY), a free, open source tool that quickly and automatically creates a machine-readable inventory of all the data files found on a given website.
When government agencies create an open data repository, they need to start by inventorying the data that the agency is already publishing on their website. This is a laborious process. It means searching their own site with a query like this:

site:example.gov filetype:csv OR filetype:xls OR filetype:json

Then they have to read through all of the results, download all of the files, and create a spreadsheet that they can load into their repository. It’s a lot of work, and as a result it too often goes undone, resulting in a data repository that doesn’t actually contain all of that government‘s data.
Realizing that this was a common problem, we hired Silicon Valley Software Group to create a tool to automate the inventorying process. We worked with Dan Schultz and Ted Han, who created a system built on Django and Celery, using Microsoft’s great Bing Search API as its data source. The result is a free, installable tool, which produces a CSV file that lists all CSV, XML, JSON, XLS, XLSX, XML, and Shapefiles found on a given domain name.
We use this tool to power our new Let Me Get That Data For You website. We’re trying to keep our site within Bing’s free usage tier, so we’re limiting results to 300 datasets per site….(More)”

The Tricky Task of Rating Neighborhoods on 'Livability'


Tanvi Misra at CityLab: “Jokubas Neciunas was looking to buy an apartment almost two years back in Vilnius, Lithuania. He consulted real estate platforms and government data to help him decide the best option for him. In the process, he realized that there was a lot of information out there, but no one was really using it very well.
Fast-forward two years, and Neciunas and his colleagues have created PlaceILive.com—a start-up trying to leverage open data from cities and information from social media to create a holistic, accessible tool that measures the “livability” of any apartment or house in a city.
“Smart cities are the ones that have smart citizens,” says PlaceILive co-founder Sarunas Legeckas.
The team recognizes that foraging for relevant information in the trenches of open data might not be for everyone. So they tried to “spice it up” by creating a visually appealing, user-friendly portal for people looking for a new home to buy or rent. The creators hope PlaceILive becomes a one-stop platform where people find ratings on every quality-of-life metric important to them before their housing hunt begins.
In its beta form, the site features five cities—New York, Chicago, San Francisco, London and Berlin. Once you click on the New York portal, for instance, you can search for the place you want to know about by borough, zip code, or address. I pulled up Brooklyn….The index is calculated using a variety of public information sources (from transit agencies, police departments, and the Census, for instance) as well as other available data (from the likes of Google, Socrata, and Foursquare)….(More)”

Open data: how mobile phones saved bananas from bacterial wilt in Uganda


Anna Scott in The Guardian:”Bananas are a staple food in Uganda. Ugandans eat more of the fruit than any other country in the world. Each person eats on average 700g (about seven small bananas) a day, according to the International Food Policy Research Institute, and they provide up to 27% of the population’s calorie intake.
But since 2002 a disease known as banana bacterial wilt (BBW) has wiped out crops across the country. When plants are infected, they cannot absorb water so their leaves start to shrivel and they eventually die….
The Ugandan government drew upon open data – data that is licensed and made available for anyone to access and share – about the disease made available by Unicef’s community polling project Ureport to deal with the problem.
Ureport mobilises a network of nearly 300,000 volunteers across Uganda, who use their mobiles to report on issues that affect them, from polio immunisation to malaria treatment, child marriage, to crop failure. It gathers data from via SMS polls and publishes the results as open sourced, open datasets.
The results are sent back to community members via SMS along with treatment options and advice on how best to protect their crops. Within five days of the first SMS being sent out, 190,000 Ugandans had learned about the disease and knew how to save bananas on their farms.
Via the Ureport platform, the datasets can also be accessed in real-time by community members, NGOs and the Ugandan government, allowing them to target treatments to where they we needed most. They are also broadcast on radio shows and analysed in articles produced by Ureport, informing wider audiences of scope and nature of the disease and how best to avoid it….
A report published this week by the Open Data Institute (ODI) features stories from around the world which reflect how people are using open date in development. Examples range from accessing school results in Tanzania to building smart cities in Latin America….(More).”

How I used data-driven journalism to reveal racial disparities in U.S. nursing homes


at StoryBench: “In 2009, while at The Chicago Reporter, I took a deep look at racial disparities in the quality of care in nursing homes in Chicago, Illinois and nationally. For a project that the Center for Public Integrity published in November 2014, I brought together Medicaid cost reports, self-reported staffing figures, testimonies from advocates and lawyers, and personal stories from nursing home residents and their families to address a simple question: how much care is a loved one actually receiving at a nursing home? The conclusion? Nursing homes serving minorities offer a lot less care than those predominately housing whites. …(More)”

The Epidemic of Facelessness


Stephen Marche in the New York Times: “….Every month brings fresh figuration to the sprawling, shifting Hieronymus Bosch canvas of faceless 21st-century contempt. Faceless contempt is not merely topical. It is increasingly the defining trait of topicality itself. Every day online provides its measure of empty outrage.

When the police come to the doors of the young men and women who send notes telling strangers that they want to rape them, they and their parents are almost always shocked, genuinely surprised that anyone would take what they said seriously, that anyone would take anything said online seriously. There is a vast dissonance between virtual communication and an actual police officer at the door. It is a dissonance we are all running up against more and more, the dissonance between the world of faces and the world without faces. And the world without faces is coming to dominate…..

The Gyges effect, the well-noted disinhibition created by communications over the distances of the Internet, in which all speech and image are muted and at arm’s reach, produces an inevitable reaction — the desire for impact at any cost, the desire to reach through the screen, to make somebody feel something, anything. A simple comment can so easily be ignored. Rape threat? Not so much. Or, as Mr. Nunn so succinctly put it on Twitter: “If you can’t threaten to rape a celebrity, what is the point in having them?”

The challenge of our moment is that the face has been at the root of justice and ethics for 2,000 years. The right to face an accuser is one of the very first principles of the law, described in the “confrontation clause” of the Sixth Amendment of the United States Constitution, but reaching back through English common law to ancient Rome. In Roman courts no man could be sentenced to death without first seeing his accuser. The precondition of any trial, of any attempt to reconcile competing claims, is that the victim and the accused look each other in the face.

For the great French-Jewish philosopher Emmanuel Levinas, the encounter with another’s face was the origin of identity — the reality of the other preceding the formation of the self. The face is the substance, not just the reflection, of the infinity of another person. And from the infinity of the face comes the sense of inevitable obligation, the possibility of discourse, the origin of the ethical impulse.

The connection between the face and ethical behavior is one of the exceedingly rare instances in which French phenomenology and contemporary neuroscience coincide in their conclusions. A 2009 study by Marco Iacoboni, a neuroscientist at the Ahmanson-Lovelace Brain Mapping Center at the University of California, Los Angeles, explained the connection: “Through imitation and mimicry, we are able to feel what other people feel. By being able to feel what other people feel, we are also able to respond compassionately to other people’s emotional states.” The face is the key to the sense of intersubjectivity, linking mimicry and empathy through mirror neurons — the brain mechanism that creates imitation even in nonhuman primates.

The connection goes the other way, too. Inability to see a face is, in the most direct way, inability to recognize shared humanity with another. In a metastudy of antisocial populations, the inability to sense the emotions on other people’s faces was a key correlation. There is “a consistent, robust link between antisocial behavior and impaired recognition of fearful facial affect. Relative to comparison groups, antisocial populations showed significant impairments in recognizing fearful, sad and surprised expressions.” A recent study in the Journal of Vision showed that babies between the ages of 4 months and 6 months recognized human faces at the same level as grown adults, an ability which they did not possess for other objects. …

The neurological research demonstrates that empathy, far from being an artificial construct of civilization, is integral to our biology. And when biological intersubjectivity disappears, when the face is removed from life, empathy and compassion can no longer be taken for granted.

The new facelessness hides the humanity of monsters and of victims both. Behind the angry tangles of wires, the question is, how do we see their faces again?

(More)”

The Trouble With Disclosure: It Doesn’t Work


Jesse Eisinger at ProPublica: “Louis Brandeis was wrong. The lawyer and Supreme Court justice famously declared that sunlight is the best disinfectant, and we have unquestioningly embraced that advice ever since.
All this sunlight is blinding. As new scholarship is demonstrating, the value of all this information is unproved. Paradoxically, disclosure can be useless — and sometimes actually harmful or counterproductive.
“We are doing disclosure as a regulatory move all over the board,” says Adam J. Levitin, a law professor at Georgetown, “The funny thing is, we are doing this despite very little evidence of its efficacy.”…
Of course, some disclosure works. Professor Levitin cites two examples. The first is an olfactory disclosure. Methane doesn’t have any scent, but a foul smell is added to alert people to a gas leak. The second is ATM. fees. A study in Australia showed that once fees were disclosed, people avoided the high-fee machines and took out more when they had to go to them.
But to Omri Ben-Shahar, co-author of a recent book, ” More Than You Wanted To Know: The Failure of Mandated Disclosure,” these are cherry-picked examples in a world awash in useless disclosures. Of course, information is valuable. But disclosure as a regulatory mechanism doesn’t work nearly well enough, he argues.
First, it really works only when things are simple. As soon as transactions become complex, disclosure starts to stumble. Buying a car, for instance, turns out to be several transactions: the purchase itself, the financing, maybe the trade-in of old car and various insurance and warranty decisions. These are all subject to various disclosure rules, but making the choices clear and useful has proved nigh impossible.
In complex transactions, we then must rely on intermediaries to give us advice. Because they are often conflicted, they, too, become subject to disclosure obligations. Ah, even more boilerplate to puzzle over!
And then there’s the harm. Over the years, banks that sold complex securities often stuck impossible-to-understand clauses deep in prospectuses that “disclosed” what was really going on. When the securities blew up, as they often did, banks then fended off lawsuits by arguing they had done everything the law required and were therefore not liable.
“That’s the harm of disclosure,” Professor Ben-Shahar said. “It provides a safe harbor for practices that smell bad. It sanitizes every bad practice.”
The anti-disclosure movement is taking on the ” Nudge” school, embraced by the Obama administration and promoted most prominently by Cass R. Sunstein, a scholar at Harvard, and Richard H. Thaler, an economist at the University of Chicago. These nudgers believe that small policies will prod people to do what’s in their best interests.
The real-world evidence in favor of nudging is thin. …
The ever-alluring notion is that we are just one or two changes away from having meaningful disclosure. If we could only have annual Securities and Exchange Commission filings in plain English, we could finally understand what’s going on at corporations. A University of San Diego Law School professor, Frank Partnoy, and I called for better bank disclosure in an article in The Atlantic a few years ago.
Professor Ben-Shahar mocks it. ” ‘Plain English!’ ‘Make it simple.’ That is the deus ex machina, the god that will solve everything,” he said.
Complex things are, sadly, complex. A mortgage is not an easy transaction to understand. People are not good at predicting their future behavior and so don’t know what options are best for them. “The project of simplification is facing a very poor empirical track record and very powerful theoretical problem,” he said.
What to do instead? Hard and fast rules. If lawmakers want to end a bad practice, ban it. Having them admit it is not enough. (More)”

U.S. to release indexes of federal data


The Sunlight Foundation: “For the first time, the United States government has agreed to release what we believe to be the largest index of government data in the world.
On Friday, the Sunlight Foundation received a letter from the Office of Management and Budget (OMB) outlining how they plan to comply with our FOIA request from December 2013 for agency Enterprise Data Inventories. EDIs are comprehensive lists of a federal agency’s information holdings, providing an unprecedented view into data held internally across the government. Our FOIA request was submitted 14 months ago.
These lists of the government’s data were not public, however, until now. More than a year after Sunlight’s FOIA request and with a lawsuit initiated by Sunlight about to be filed, we’re finally going to see what data the government holds.
Sunlight’s FOIA request built on President Obama’s Open Data Executive Order, which first required agency-wide data indexes to be built and maintained. According to implementation guidance prepared in response to the executive order, Enterprise Data Inventories are intended to help agencies “develop a clear and comprehensive understanding of what data assets they possess” by accounting “for all data assets created or collected by the agency.”
At the time, we argued that “without seeing the entire EDIs, it is impossible for the public to know what data is being collected and stored by the government and to debate whether or not that data should be made public.”
When OMB initially responded to our request, it didn’t cite an exemption to FOIA. Instead, OMB directed us to approach each agency individually for its EDIs. This, despite the fact that the agencies are required to submit their updated EDIs to OMB on a quarterly basis.
With that in mind, and with the help of some very talented lawyers from the firm of Garvey Schubert Barer, we filed an administrative appeal with OMB and prepared for court. We were ready to fight for the idea that government data cannot be leveraged to its fullest if the public only knows about a fraction of it.
We hoped that OMB would recognize that open data is worth the work it takes to disclose the indexes. We’re pleased to say that our hope looks like it is becoming reality.
Since 2013, federal agencies have been required to construct a list of all of their major data sets, subject only to a few exceptions detailed in President Obama’s executive order as well as some information exempted from disclosure under the FOIA.
Having access to a detailed index of agencies’ data is a key step in aiding the use and utility of government data. By publicly describing almost all data the government has in an index, the Enterprise Data Inventories should empower IT management, FOIA requestors and oversight — by government officials and citizens alike….(More)”.

More Power to the People: How Cities Are Letting Data Flow


Stephen Taylor at People4SmarterCities: “Smart cities understand that engaging the public in decision-making is vital to enhancing services and ensuring accountability. Here are three ideas that show how cities are embracing new technologies and opening up data to spur civic participation and improve citizens’ lives.

 City Texts Help Keep Food on the Table
In San Francisco, about a third of the 52,000 people that receive food stamps are disenrolled from the program because they miss certain deadlines, such as filing quarterly reports with the city’s Human Services Agency. To help keep recipients up to date on their status, the nonprofit organization Code for America worked with the city agency to create Promptly, an open-source software platform that sends alerts by text message when citizens need to take action to keep their benefits. Not only does it help ensure that low-income residents keep food on the table, it also helps the department run more efficiently as less staff time is spent on re-enrollments.
Fired Up in Los Angeles Over Open Data
For the Los Angeles Fire Department, its work is all about responding to citizens. Not only does it handle fire and medical calls, it’s also the first fire agency in the U.S. to gather and post data on its emergency-response times on the Internet through a program called FireStat. The data gives citizens the opportunity to review metrics such as the amount of time it takes for stations to process emergency calls, the time for firefighters to leave the station and the travel time to the incident for each of its 102 firehouses throughout the city. The goal of FireStat is to see where and how response times can be improved, while increasing management accountability….(More)”

Moovit Crowdsources Public Transit Data, So You'll Never Get Stuck Waiting For The Bus Again


Ariel Schwartz at Co.Exist: “There are few things more frustrating than rushing to the bus, only to find out that it’s delayed indefinitely. Or arriving at the subway stop to find hundreds of others waiting for the same delayed train, making your chances of entry onto the next car nearly impossible.
In cities with open data, localized transit apps with real-time transit updates can alleviate the problem, but those apps aren’t available everywhere—and in most cases, they don’t offer up information about service alerts, like bus stops relocated due to construction.
Moovit, an Israeli startup that recently raised $50 million for its app, which takes a lot of the headache out of navigating public transportation. Available in over 500 cities globally, Moovit uses a combination of official transit information and crowdsourced live updates to provide accurate information on a city’s public transportation city at any given moment.
Getting started with the app is fairly intuitive; just put in your starting location and destination and Moovit spits out the best way to get there. It also notifies you when you’re getting close to your stop—a bonus in cities with transportation systems that don’t always make that clear (ahem, San Francisco)….
The vision for Moovit goes beyond public transportation (just look at the company’s investor list, which includes BMW i Ventures). “We want to make this an omni-search for non-car owners, including bike, taxi, and carsharing services,” says Erez. Later this year, Moovit plans to make a number of announcements around integrating multiple means of transportation into the app. (More)

Training the next generation of public leaders


Thanks to the generous support of the Knight Foundation, this term the Governance Lab Academy – a training program designed to promote civic engagement and innovation – is launching a series of online coaching programs.
Geared to the teams and individuals inside and outside of government planning to undertake a new project or trying to figure out how to make an existing project even more effective and scalable, these programs are designed to help participants working in civic engagement and innovation develop effective projects from idea to implementation.
Convened by leading experts in their fields, coaching programs meet exclusively online once a week for four weeks or every other week for eight weeks. They include frequent and constructive feedback, customized and original learning materials, peer-to-peer support, mentoring by topic experts and individualized coaching from those with policy, technology, and domain expertise.
There is no charge to participants but each program is limited to 8-10 project teams or individuals.
You can see the current roster of programs below and check out the website for more information (including FAQs), to sign up and to suggest a new program.

Faculty includes: 

  • Brian Behlendorf, Managing Director at Mithril Capital Management and Co-Founder Apache
  • Alexandra Clare, Founder of Iraq Re:Coded
  • Brian Forde, Senior Former Advisor to the U.S. CTO, White House Office of Science Technology and Policy
  • Francois Grey,  Coordinator of the Citizen Cyberscience Centre, Geneva
  • Gavin Hayman, Executive Director of the Open Contracting Partnership
  • Clay Johnson, CEO of The Department for Better Technology and Former Presidential Innovation Fellow
  • Benjamin Kallos, New York City Council Member and Chair of the Committee on Governmental Operations of the New York City Council
  • Karim Lakhani, Lumry Family Associate Professor of Business Administration at the Harvard Business School
  • Amen Ra Mashariki, Chief Analytics Officer of New York City
  • Geoff Mulgan, Chief Executive of NESTA
  • Miriam Nisbet,  Former Director of the Office of Government Information Services
  • Beth Noveck, Founder and CEO of The GovLab
  • Tiago Peixoto, Open Government Specialist at The World Bank
  • Arnaud Sahuguet, Chief Technology Officer of The GovLab
  • Joeri van den Steenhoven, Co-Founder and Chief Research and Development Officer of MaRS Solutions Lab
  • Stefaan Verhulst, Co-Founder and Chief Research and Development Officer of The GovLab