Stock Market Prediction using Google search terms


MIT Technology Review: “This week’s fleeting stock market crash prompted by a false report from the Associated Press’s hacked Twitter account has focused attention again on the growing Wall Street practice of mining news and social data to make trades.

A study in Nature Scientific Reports today illustrates just how lucrative the right combination of algorithms could potentially be.
Using Google Trends, researchers analyzed the Google search query volumes from 2004 to 2011 for a set of 98 mostly finance-related search terms, looked at how stock prices changed over that same time, and tried to see if they could retroactively tease out search patterns that showed “early warning signs” of market moves. They also tested trading strategies that would act on these signs.
srep01684-f2
The volume of the search term “debt” turned out to be the word that showed the most promise, and one trading plan based on changes in searches for this term would have yielded a return of 326 percent over the period analyzed, the authors found. For comparison, a “buy and hold” investment in the Dow Jones Industrial Average yielded 16 percent return.”

How to Clean Up Social News


verilyDavid Talbot in MIT Technology Review: ” New platforms for fact-checking and reputation scoring aim to better channel social media’s power in the wake of a disaster…Researchers from the Masdar Institute of Technology and the Qatar Computer Research Institute plan to launch Verily, a platform that aims to verify social media information, in a beta version this summer. Verily aims to enlist people in collecting and analyzing evidence to confirm or debunk reports. As an incentive, it will award reputation points—or dings—to its contributors.
Verily will join services like Storyful that use various manual and technical means to fact-check viral information, and apps such as Swift River that, among other things, let people set up filters on social media to provide more weight to trusted users in the torrent of posts following major events…Reputation scoring has worked well for e-commerce sites like eBay and Amazon and could help to clean up social media reports in some situations.

Investigating Terror in the Age of Twitter


Michael Chertoff and Dallas Lawrence in WSJ: “A dozen years ago when the terrorists struck on 9/11, there was no Facebook or Twitter or i-anything on the market. Cellphones were relatively common, but when cell networks collapsed in 2001, many people were left disconnected and wanting for immediate answers. Last week in Boston, when mobile networks became overloaded following the bombings, the social-media-savvy Boston Police Department turned to Twitter, using the platform as a makeshift newsroom to alert media and concerned citizens to breaking news.
Law-enforcement agencies around the world will note how social media played a prominent role both in telling the story and writing its eventual conclusion. Some key lessons have emerged.”

Knowing Where to Focus the Wisdom of Crowds


Nick Bilton in NYT: “It looks as if the theory of the “wisdom of crowds” doesn’t apply to terrorist manhunts. Last week after the Boston Marathon bombings, the Internet quickly offered to help find the people responsible. In a scene metaphorically reminiscent of a movie in which vigilantes swarm the streets with pitchforks and lanterns, people took to Reddit, the popular community and social news Web site, and started scouring images posted online from the bombings.
One Reddit forum told users to search for ”people carrying black bags,” and noted that “if they look suspicious, then post them. Then people will try and follow their movements using all the images.” In the process, each time a scrap of information was discovered — the color of a hat, the type of straps on a backpack, the weighted droop of a bag — it was passed out on Twitter like “Wanted” posters tacked to lampposts. It didn’t matter whether it was right, wrong or even completely made up (some images posted to forums had been manipulated) — off it went, fiction and fact indistinguishable. Some misinformation online landed on the front page of The New York Post, incorrectly identifying an innocent high school student as a suspect. Later in the week, the Web wrongly identified one of the suspects as  a student from Brown University who went missing earlier this month…
Perhaps the scariest aspect of these crowd-like investigations is that when information is incorrect, no one is held responsible.
As my colleague David Carr noted in his column this week, “even good reporters with good sources can end up with stories that go bad.” But the difference between CNN, The Associated Press or The New York Post getting it wrong, is that those names are held accountable when they publish incorrect news. No one is going to remember, or punish, the users on Reddit or Twitter who incorrectly identify random high school runners and missing college students as terrorists.”

Demystifying data centers


Wired: “If you walk into the lobby of the data center Facebook operates in the high desert in Prineville, Oregon, you’ll find a flatscreen display on the wall where you can check the pulse of this massive computing facility.
The display tracks the efficiency of the operation, which spans 333,400-square feet and tens of thousands of computer servers. Facebook built this data center in an effort to significantly reduce the power and dollars needed to serve up the world’s most popular social network, and — driven by CEO Mark Zuckerberg’s deep-seeded belief in the free exchange of ideas — the company aims to push the computing world in a similar direction. The display — which shows much the same information Facebook engineers use to monitor the facility — is an advertisement for the Facebook way.
Now, the company is taking this idea a step further. On Thursday, Facebook uncloaked a pair of web services that let anyone in the world track the efficiency of the Prineville data center and its sister facility in Forest City, North Carolina. “We’re pulling back the curtain to share some of the same information that our data center technicians view every day,” Facebook’s Lyrica McTiernan said in a blog post. “We think it’s important to demystify data centers and share more about what our operations really look like.”
http://www.wired.com/wiredenterprise/wp-content/uploads/2013/04/facebook-dashboard.png

Newark's Cory Booker: Social Media Can Help Fix Broken Government


Internet Evolution on Cory Booker’s panel at Ad Age Digital Conference: “Social media have been a part of a transformation of the City of Newark from a butt of jokes to a community experiencing economic growth, Booker told the Ad Age conference. Newark has a population of 300,000 in a state with 9 million people, and yet, Newark has a third of the economic growth in the state. The city population is growing for the first time in 60 years.
Social media can be a big part of the cure for government that has become unresponsive to the needs of its citizens, Booker said. He quoted California Lt. Governor Gavin Newsom, who uses the phrase “vending machine government.” Citizens pay for government services, and get prepackaged offerings in return. “If you don’t like what you get, you shake the vending machine,” Booker said…
When people lean back and disengage, government becomes unresponsive. But social media provide the tools for citizens to collaborate with government.  “We have all these tools pulling government away from citizens,” Booker said. These include special interest groups and moneyed corporate lobbies. “But social media brings us closer.”
Twitter helped Newark rebuild its reputation. The city had been a butt of jokes for years. When Conan O’Brien made a joke at Newark’s expense, Booker replied with an online video that said O’Brien was now on the no-fly list at Newark Airport. The TSA got into the act, issuing a statement that Booker didn’t have that power. Then-Secretary of State Hillary Clinton followed up with a plea for Booker and O’Brien to just get along.
And it’s not just a matter of public relations; social media have helped improve Newark in concrete ways — Newark’s government is more effective. For example, its inspectors are vastly more efficient at finding violations when citizens can use social media to point up problems, Booker said.
Video can be an even more powerful tool for getting a message out than microblogging services such as Twitter, Booker said. And that led to discussion of Booker’s startup, #waywire. The beta video service, updated this week to focus on video curation, is a place where people can collect and share online video.”
 

Public health, disaster recovery and social media


Janice Jacobs:  “Increasingly, social media is playing a key role in helping to ease the heavy burden of these tragedies by connecting individuals and communities with each other and with critical resources…
Social media, in its simplest form, can notify the masses in real-time about situations that are happening or are about to happen.

  • In August 2011, several New Yorkers learned of an earthquake on Twitter prior to feeling it. From the D.C. area, tweets began popping up in droves almost 30 seconds before anyone felt the tremors in New York City, and ahead of any media reports about it. Twitter said that more than 40,000 earthquake-related tweets were sent within a minute of the earthquake’s manifestation…...

Social media can be used to identify trouble spots and to react quickly during emergencies.

Social media can be used to foster communication among various healthcare, aid, government agencies and individuals.

  • Cory Booker, Mayor of Newark, NJ, a prolific Twitter user, consistently tweeted helpful information for the Newark community following Hurricane Sandy in late October 2012.”

 

Big Data, Big Brains


“This report on Big Data is the first MeriTalk Beacon, a new series of reports designed to shed light and provide direction on far reaching issues in government and technology. Since Beacons are designed to tackle broad concepts, each Beacon report relies on insight from a small number of big thinkers in the topic area. Less data. More insight. Real knowledge…Mankind created 150 exabytes (billion gigabytes) of data in 2005, and 1,800 exabytes in 20112; growth that only continues to accelerate. Every minute, users: Upload 48 hours of video to YouTube; Send 204 million emails; Spend $207,000 via the web; Create 571 new websites. Within the Federal government; U.S. drone aircraft sent back 24 years worth of video footage in just 2009. Every 24 hours, NASA’s Curiosity rover can send nearly three gigabytes of data, collecting in mere days the equivalent of all human knowledge through the death of Augustus Caesar – from Mars.”

Quarter of time online is spent on social networking


Experian: “Insights from Experian, the global information services company, reveals that if the time spent on the Internet was distilled into an hour then a quarter of it would be spent on social networking and forums across UK, US and Australia. In the UK 13 minutes out of every hour online is spent on social networking and forums, nine minutes on entertainment sites and six minutes shopping.”
Social Networking table

Data for the Boston Marathon Investigation Will Be Crowdsourced


WIRED: “The investigation of Monday’s deadly twin bombings in Boston will rely to an extraordinary extent on crowdsourced surveillance, provided by Marathon spectators’ cellphone photos, Vine videos, and Instagram feeds….There are limits to the crowdsourcing. The data used in the investigation will be crowdsourced. The investigation will not be. A crowdsourced investigation runs a high risk of becoming a witchhunt, as we saw in the Newton shooting spree.”