I was reading this interesting study on the impact of fear on the stability of a food web which led me to start thinking about principles of sound experimental design, and how such designs can yield valuable insight into a variety of systems, natural or man made. From the authors:
When it comes to conserving biodiversity and maintaining healthy ecosystems, fear has its uses. By inspiring fear, the very existence of large carnivores on the landscape, in and of itself, can provide a critical ecosystem service human actions cannot fully replace, making it essential to maintain or restore large carnivores for conservation purposes on this basis alone.
The experimental design behind this study was fascinating. Using two islands off the coast of British Columbia, Canada, the team setup an experiment:
To see whether fear alone could impact raccoon behavior, Suraci’s team played the sounds of wild dogs barking along the shoreline. As a control, they played the sounds of harbor seals, stellar sea lions and other pinnipeds along a separate shoreline. Throughout the month, they used cameras to track the number of raccoons prowling the beaches, as well as to survey the number of red rock and shore crabs and intertidal fish before and after the experiment.
When the raccoons heard the dog barks, they startled and shied away. Overall, raccoons on this fear-filled island spent 66 percent less time foraging in their favorite spots. The innocuous seal sounds, by contrast, produced no such effect.
The impact of those barking dog sounds reverberated through the food chain. At the end of this month-long experiment, the number of shore crabs nearly doubled, while there was an 81 percent jump in the number of intertidal fish and a 61 percent increase in red rock crabs, likely because fewer raccoons were scooping them up. Meanwhile, the small invertebrates that those crabs ate decreased, likely because the crustacean numbers increased.
This struck me as an interesting experiment with conditions on a grand scale. This type of experimental design is something often lacking in the commercial software development industry. We often discount the importance of investing in designing experiments that provide data on how our products behave under different conditions. Typically it is far too difficult to design a series of test conditions that would accurately mirror production environments. Instead, we can setup experiments to breakdown our product workloads into atomic units of work, which can then be reassembled to reflect the expected usage pattern for a specific customer.
One useful technique is using linear regression analyses for identifying expected latencies for computationally intensive workloads. In this case, people matching comparison functions. In this example we’re looking at comparing the behavior of a new linguistic comparison function (COMP2) versus a standard comparison function (COMP1). Initial runs indicated a non-linear increase in comparison latency as the number of tokens in a name increased. This required further study to determine the cause of the latency increase, so I designed an experiment to further isolate the behavior.
I sorted the name list by token count that was used as the basis for comparing names. I randomly selected 1,000 names from each name length, and used those as test criteria. So now we’ve got a set of 6,000 names that range from 2 tokens to 7 tokens per name, with 1,000 names each. This basic test framework allowed for the creation of a baseline with COMP1, and successive test runs with different optimizations using COMP2:
With this experimental setup, we were able to first assess the latency impact of increasing name length on comparison time, identify the development of a secondary population at an elevated latency as token count increased, and test multiple optimizations. Without developing a specific test framework for isolating comparison behavior, we would not have been able to identify the latency pattern, and the underlying cause.