Biodiversity describes the variety of organisms on planet earth. Ecologists have long hoped for a synthesis between analyses of biodiversity and analyses of biotic interactions among species, such as predation, competition, and mutualism. However, it is often unclear how to connect details of these interactions with complex modern analyses of biodiversity. Using methods pioneered in studies of ecological-evolutionary dynamics, we link biotic interactions and changes in measures of biodiversity such as Hill numbers. We show that analyses of biodiversity obscure details about biotic interactions. For example, identical changes in biodiversity can arise from predation, competition or mutualism, locally or across a metacommunity. Our approach indicates that traditional models of community assembly miss key facets of diversity change. Instead, we suggest that analyses of diversity change should focus on partitions, which measure mechanisms that directly shape changes in diversity, notably relative fitness and immigration, rather than traditional analyses of biotic interactions.

Long-term ecological studies have consistently reported slower than expected changes in biodiversity over time. One explanation for this phenomenon is that commonly used diversity measurements such as species richness are too coarse to detect mechanisms shaping community assembly. It remains unclear whether abundance based diversity measurements are susceptible to the same problem. To test this, we study temporal changes in abundance based diversity indices across 3341 observations from 880 plots from 15 long-term vegetation plot studies. We then partition diversity change into mechanisms of interest to ecologists: selection, drift, and immigration. We show that these mechanisms are an imperfect predictor of temporal diversity change, creating a mismatch between changes in species abundances and changes in diversity, particularly when shifts in diversity are rapid. To resolve this mismatch, we quantify a less studied mechanism “rarity shifts”, which measure how an individual’s contributions to diversity changes over time. We found rarity shifts are an important component of diversity change across many studies. Furthermore, rarity shifts tend to oppose other mechanisms, particularly selection. Therefore, rarity shifts obscure changes in relative abundance from abundance based diversity measurements, revealing why diversity changes appear slower than expected. Ultimately, understanding rarity shifts can lead to a more accurate understanding of the rate and nature of temporal diversity change in ecology and conservation.

Changes in biodiversity reflect processes acting on the success of individual species at multiple spatial scales, including in communities, biogeographic regions, and globally. This complexity makes it difficult to analyse the mechanisms shaping diversity change using traditional approaches. To resolve this, we propose a novel approach to partition total biodiversity changes according to mechanisms reflecting species' success at multiple scales. We apply this approach to study changes in the diversity of invertebrate herbivores from a large-scale, plant community experiment. This partitioning showed that rapid changes in the relative abundances of individual species resulted in surprisingly small changes in diversity across scales. Our novel analytical method reveals how strong ecological effects at different hierarchical levels can counteract each other, resulting in weak effects on diversity across broad spatial scales.