More than four decades ago, field ecologists set out to quantify the diversity of trees on a forested plot on Barro Colorado Island in Panama, one of the most intensively studied tracts of forest on the planet. They began counting every tree with a trunk wider than a centimeter. They identified the species, measured the trunks and calculated the biomass of each individual. They put ladders up the trees, examined saplings and recorded it all in sprawling spreadsheets.

As they looked at the data accumulating year after year, they began to notice something odd in it. With more than 300 species, the tree diversity on the tiny 15-square-kilometer island was staggering. But the distribution of trees among those species was also heavily lopsided, with most of the trees belonging to only a few species.

Since those early studies, that overstuffed, highly uneven pattern has been seen repeatedly in ecosystems around the world, particularly in rainforests. The ecologist Stephen Hubbell of the University of California, Los Angeles, who was part of the team behind the Barro Colorado surveys, estimates that less than 2% of the tree species in the Amazon account for half of all the individual trees, meaning that 98% of the species are rare.

Such high biodiversity flies in the face of predictions made by a leading theory of ecology, which says that in a stable ecosystem, every niche or role should be occupied by one species. Niche theory suggests that there are not enough niches to enable all the species the ecologists saw to stably exist. Competition over niches between similar species should have sent the rarities into extinction.

A new ecological modeling paper in Nature by James O’Dwyer and Kenneth Jops of the University of Illinois, Urbana-Champaign explains at least part of this discrepancy. They found that species that should seemingly be head-to-head competitors can share an ecosystem if details of their life histories — such as how long they live and how many offspring they have — line up in the right way. Their work also helps to explain why one of the most successful ways to model ecologies often arrives at accurate results, even though it glosses over almost all we know about how organisms function.

Back in 2001, the paradoxically high biodiversity on Barro Colorado Island inspired Hubbell to propose the groundbreaking neutral theory of ecology. Traditional ecology theory stressed the competition for niches between species. But Hubbell pointed out that species might not really matter in that equation because, in effect, individuals compete for resources with members of their own species too. He suggested that patterns of diversity in ecosystems might largely be the products of random processes.

For a theory that dealt with biodiversity, Hubbell’s neutral theory was sparse. It ignored variations in life spans, nutritional quirks and other details that distinguish one species from another. In models based on the theory, every individual in a theoretical ecosystem is identical. Once the clock starts, the ecosystem evolves stochastically, with individuals outcompeting and replacing each other at random. The theory was completely at odds with species-based approaches to ecology, and it provoked impassioned debate among ecologists because it seemed so counterintuitive.

Yet surprisingly, as the random walks in the neutral models progressed, they reproduced key features of what Hubbell and his colleagues saw in their data from Barro Colorado Island and what others have seen elsewhere. In this modeling that almost perversely acknowledges no differences, there are flashes of the real world.

That tension between the models and reality has long interested O’Dwyer. Why did neutral theory seem to work so well? Was there a way to bring in information about how species function to get results that might look still more realistic?

One of the things that make neutral models appealing, O’Dwyer said, is that there really are deep universalities among many living things. While animal species are not identical, they are remarkably similar at the level of, say, the circulatory system. The same numbers concerning physiology crop up again and again in animals and plants, reflecting perhaps the constraints of their shared evolutionary history. According to a principle called Kleiber’s law, for example, the metabolic rate of an animal generally increases with its size, scaling as a power law — the same power law, no matter the species. (Several theories about why Kleiber’s law is true have been offered, but the answer is still debated.)

Given those signs of underlying order, O’Dwyer wondered whether some details of how organisms live matter more than others in determining how successfully species will compete and survive over evolutionary time. Take metabolism again: If an ecosystem can be seen as an expression of its inhabitants’ metabolisms, then the organisms’ sizes are special, significant numbers. The size of an individual may be more useful in modeling its fate over time than any number of other details about its diet or species identity.

O’Dwyer wondered whether one of those crucial, privileged factors might be captured by life history, a concept that combines species statistics such as average number of offspring, time until sexual maturity and life span. Imagine a plot of 50 individual plants. Each has its own life span, its own pattern of reproduction. After three months, one plant might produce 100 seeds, while another similar one produces 88. Maybe 80% of their seeds will germinate, producing the next generation, which will go through its own version of this cycle. Even within a species, individual plants’ numbers will vary, sometimes by a little, sometimes by a lot, a phenomenon called demographic noise. If this variation is random, in the manner of Hubbell’s neutral theory, what patterns will emerge over successive generations?

O’Dwyer knew he had found someone who could help him explore that question when Jops joined his lab as a graduate student. Jops had previously studied whether models using life histories could predict whether a vulnerable plant species would survive or if it was on the way out. Together, they started to hammer out the math that would describe what happens when life history meets competition.

In Jops and O’Dwyer’s model, as in neutral models, stochasticity — the influence of random factors on deterministic interactions among the species — is important. The life histories of species, however, can amplify or reduce the effects of that randomness. “Life history is a kind of lens through which demographic noise works,” O’Dwyer said.

When the researchers allowed their model to progress through time, putting each simulated individual through its paces, they found that certain species could persist alongside each other for long periods even though they were competing for the same resources. Looking deeper into the numbers for an explanation, Jops and O’Dwyer found that a complex term called effective population size seemed useful for describing a kind of complementarity that could exist among species. It encapsulated the fact that a species could have high mortality at one point in its life cycle, then low mortality at another, while a complementary species might have low mortality at the first point and high mortality at the second. The more similar this term was for two species, the more likely it was that a pair could live alongside each other despite competing for space and nutrition.

“They experience demographic noise at the same amplitude,” O’Dwyer said. “That’s the key for them to live together a long time.”

The researchers wondered if similar patterns prevailed in the real world. They drew on the COMPADRE database, which houses details about thousands of plant, fungal and bacterial species collected from a variety of studies and sources, and they zeroed in on perennial plants that all lived together in the same research plots. They discovered that, as their model had predicted, the plant species that lived together had closely matching life histories: Pairs of species living in the same ecosystem tend to be more complementary than randomly drawn pairs.

The findings suggest ways in which species that are not necessarily in direct competition could work well alongside each other without invoking distinct niches, said Annette Ostling, a professor of biology at the University of Texas, Austin. “The coolest part is that they are highlighting that these ideas … can extend to species that are pretty different but complementary,” she said.

To William Kunin, a professor of ecology at the University of Leeds in England, the paper suggests one reason why the natural world, for all its complexity, can resemble a neutral model: Ecological processes may have a way of canceling each other out, so that what seems like endless variety can have a simple outcome he described as “emergent neutrality.” Hubbell, for his part, appreciates the expansion of his initial work. “It offers some thoughts on how to generalize neutral models, to tweak them to put in a bit of species differences, expanding and contracting to see what happens to diversity in a local community,” he said.

This is just one bite out of the problem of understanding how biodiversity arises and why it persists, however. “In ecology, we struggle with the relationship between pattern and process. Many different processes can produce the same pattern,” Ostling said. O’Dwyer hopes that in the coming years, more data about the real world can help researchers discern whether effective population size is consistently able to explain coexistence.

Kunin hopes that the paper will inspire others to keep working with ideas from neutral theory. In a field where the unique qualities of individuals, rather than their commonalities, have long held sway, neutral theory has forced ecologists to be creative. “It’s kicked us out of our mental ruts and made us think about which things really matter,” he said.

Hubbell, who unleashed neutral theory on ecology so many years ago, wonders whether truly immense data sets about real forests could yield the kind of detail needed to make the relationship between life history and biodiversity clearer. “This is the kind of building on neutral theory that I was hoping would happen,” he said of the new paper. “But it’s only a baby step toward really understanding diversity.”