How Computers Trawl a Sea of Data for Stock Picks


An example from the Wall Street Journal of more data improving efficiency – in this case, allowing firms to more efficiently allocate resources in our economy to bring about the best growth. By leveraging increasingly large and diverse datasets, they can get a more accurate picture of the world’s needs and various risks. The equality of who gets the gains from these efficiencies can be a matter of debate, but more data are unequivocally more efficient than less data.

Two Sigma’s funds all take a big-data approach. Among its data sources are news bulletins, National Weather Service reports, market data, tweets and information from smartphone users who have agreed to be tracked by a retail-trend-analysis company.

Thirty years ago, it was easier to make investment picks because the world wasn’t as interconnected, Mr. Siegel says. “Here’s the problem: What affects the price of a share of Apple stock? The answer: Pretty much everything. Absolutely every little thing has some effect. Every sale, every earthquake.”

To comb through data 24 hours a day, the firm has more than 100 teraflops of power—more than 100 trillion calculations a second—and more than 11 petabytes of storage, the equivalent of five times the data stored in all U.S. academic libraries.