Order online through the Harvard Business Review website or request a free reprint from Icosystem.

Three years ago, Nasdaq faced a prickly problem. It wanted to reduce the tick size of its trading, which was then 1/8 of a dollar, to 1/16 and eventually down to 1/100. Among market professionals, the perceived wisdom was that the change would let buyers and sellers negotiate in more precise terms, thus driving down the market’s spread between bid and ask prices. But the organization was wary that the move toward decimalization might backfire, leading to inefficiencies or, worse, loopholes that people could abuse. In the past, Nasdaq executives had analyzed the financial marketplace through economic studies, financial models, and other research. But the move to a smaller tick size was unknown territory that defied traditional analyses. How could Nasdaq ensure that decimalization would not muck up the system?

To answer that, Nasdaq worked with BiosGroup, a consultancy in Santa Fe, New Mexico, to develop a computer program that simulated the proposed changes. It was no ordinary program; the software created thousands of virtual individuals to represent market makers, institutional investors, pension fund managers, day traders, casual investors, and other market participants. Each of those software agents made decisions to buy and sell using real-world strategies. The technology, called agent-based modeling, enabled Nasdaq to explore stock market dynamics that pure mathematical methods could never unpick.

The results were eye opening. A smaller tick size could actually reduce the market’s ability to perform price discovery, leading to wider spreads between the bid and ask prices. Further tests helped Nasdaq confirm and understand this counterintuitive behavior, which allowed the organization to better plan its transition to decimalization.

Nasdaq is not the only pioneer to benefit from agent-based modeling. Macy’s, for instance, has used the technology to investigate better ways to design its department stores. Hewlett-Packard has run agent-based simulations to anticipate how changes in its hiring strategy would affect its corporate culture. And Société Générale has used the technology to determine the operational risk of its asset management group.