When machine learning packs an economic punch

A new study co-authored by an MIT economist shows that improved translation software can significantly boost international trade online — a notable case of machine learning having a clear impact on economic activity.

The research finds that after eBay improved its automatic translation program in 2014, commerce shot up by 10.9 percent among pairs of countries where people could use the new system.

“That’s a striking number. To have it be so clear in such a short amount of time really says a lot about the power of this technology,” says Erik Brynjolfsson, an MIT economist and co-author of a new paper detailing the results.

To put the results in perspective, he adds, consider that physical distance is, by itself, also a significant barrier to global commerce. The 10.9 percent change generated by eBay’s new translation software increases trade by the same amount as “making the world 26 percent smaller, in terms of its impact on the goods that we studied,” he says.

The paper, “Does Machine Translation Affect International Trade? Evidence from a Large Digital Platform,” appears in the December issue of Management Science. The authors are Brynjolfsson, who is the Schussel Family Professor of Management Science at the MIT Sloan School of Management, and Xiang Hui and Meng Liu, who are both assistant professors in the Olin Business School at Washington University in St. Louis.

Just cause

To conduct the study, the scholars examined what happened after eBay, in 2014, introduced its new eBay Machine Translation (eMT) system — a proprietary machine-learning program that, by several objective measures, significantly improved translation quality on eBay’s site. The new system initially was focused on English-Spanish translations, to facilitate trade between the United States and Latin America

Previously, eBay had used Bing Translator to render the titles of objects for sale. By one evaluation measure, called the Human Acceptance Rate (HAR), in which three experts accept or reject translations, the eMT system increased the number of acceptable Spanish-language item titles on eBay from 82 percent to 90 percent.

Using administrative data from eBay, the researchers then examined the volume of trade on the platform, within countries, after the eMT system went into use. Other factors being equal, the study showed that the new translation system not only had an effect on sales, but that trade increased by 1.06 percent for each additional word in the titles of items on eBay.

That is a substantial change for a commerce platform on which, as the paper notes, items for sale often have long, descriptive titles such as “Diamond-Cut Stackable Thin Wedding Ring New .925 Sterling Silver Band Sizes 4-12,” or “Alpine Swiss Keira Women’s Trench Coast Double Breasted Wool Jacket Belted.” In those cases, making the translation clearer helps potential buyers understand exactly what they might be purchasing.

Given the study’s level of specificity, Brynjolfsson calls it “a really fortunate natural experiment, with a before-and-after that sharply distinguished what happened when you had machine translation and when you didn’t.”

The structure of the study, he adds, has enabled the researchers to say with confidence that the new eBay program, and not outside factors, directly generated the change in trade volume among affected countries.

“In economics, it’s often hard to do causal analyses and prove that A caused B, not just that A was associated with B,” says Brynjolfsson. “But in this case, I feel very comfortable using causal language and saying that improvement in machine translation caused the increase in international trade.”

Larger puzzle: The productivity issue

The genesis of the paper stems from an ongoing question about new technology and economic productivity. While many forms of artificial intelligence have been developed and expanded in the last couple of decades, the impact of AI, including things like machine-translation systems, has not been obvious in economics statistics.

“There’s definitely some amazing progress in the core technologies, including in things like natural language processing and translation,” Brynjolfsson says. “But what’s been lacking has been evidence of an economic impact, or business impact. So that’s a bit of a puzzle.”

When looking to see if an economic impact for various forms of AI could be measured, Brynjolfsson, Hui, and Liu thought machine translation “made sense, because it’s a relatively straightforward implementation,” Brynjolfsson adds. That is, better translations could influence economic activity, at least on eBay, without any other changes in technology occurring.

In this vein, the findings fit with a larger postulation Brynjolfsson has developed in recent years — that the adoption of AI technologies produces a “J-curve” in productivity. As Brynjolfsson has previously written, broad-ranging AI technologies nonetheless “require significant complementary investments, including business process redesign, co-invention of new products and business models, and investments in human capital” to have a large economic impact.

As a result, when AI technologies are introduced, productivity may appear to slow down, and when the complementary technologies are developed, productivity may appear to take off — in the “J-curve” shape.

So while Brynjolfsson believes the results of this study are clear, he warns against generalizing too much on the basis of this finding about the impact of machine learning and other forms of AI on economic activity. Every case is different, and AI will not always produce such notable changes by itself.

“This was a case where not a lot of other changes had to happen in order for the technology to benefit the company,” Brynjolfsson says. “But in many other cases, much more complicated, complementary changes are needed. That’s why, in most cases with machine learning, it takes longer for the benefits to be delivered.”