Unlocking large-scale killer whale analysis

Raincoast Conservation Foundation has joined forces with Earth Species Project (ESP), a non-profit research lab using machine learning to advance our understanding of animal communication. At the heart of this project is a deceptively simple idea: to watch and listen at the same time.

By synchronizing drone footage, underwater acoustic recordings, and time-stamped behavioral observations to the nearest second, the team aims to uncover how killer whales use sound to coordinate movements, share prey, and maintain social bonds.

Ultimately, our objective is to understand and mitigate how noise pollution interferes with these processes. We’re chatting with David Robinson, Senior AI Research Scientist at Earth Species Project.

Hi David, can you tell us a little bit about who you are and what your background is? 

Before joining Earth Species Project I worked on a series of medical AI startups, most recently as the Machine Learning (ML) lead working on large language models for radiology and on structuring medical data. 

But I had always dreamed of working with whales. I realized I could use the same ML techniques I was using to study animal communication, and that this could have a big impact from a conservation perspective.

I published a paper on a model called BioLingual, which learned to pair animal vocalizations with human language descriptions of species and context. After I was invited to present this work to Earth Species Project, I eventually joined the team.

At ESP, I work on the team that builds foundation models – tools that can help speed up ethology workflows across multiple tasks and species. One example is NatureLM-audio, our language model for animal sounds. 

How did you end up working on this killer whale project with Raincoast? 

Honestly, I think the team at ESP got me involved early on knowing I love whales! I joined the 2025 fieldwork aboard Raincoast’s research sailboat Achiever, essentially as a research assistant – both to learn more about the data and how the Raincoast team collects it, and to give Earth Species a chance to share input on the data collection process. Co-designing from the beginning with biologists is hugely important: it lets us build a data pipeline together that works end-to-end, from collection all the way through to machine learning and analysis.

How is AI helping advance this conservation research?

At Earth Species Project, we are leveraging advances in foundation models and large language models, and using both self-supervised and supervised approaches to analyze animal vocalizations and their surrounding context at scale. 

Our models are trained on massive data sets across human speech and music, bioacoustic data and environmental sounds. They learn foundational representations of bioacoustic signals, which makes them generalizable across tasks and species. Whether studying birds or whales or jumping spiders, our models and methods help researchers automatically detect, classify, and find new answers in massive multimodal datasets.

Our tools are designed to accelerate the science already happening by automating some of the most manual and time-intensive parts of the work, so researchers can spend more time out in the field and on the biological questions that matter. 

We see AI as a tool for overcoming a challenge that has long constrained conservation and animal communication research: the gap between the scale of data we can collect and our ability to make sense of it. 

AI lets us move from monitoring to understanding – identifying not just presence or absence, but deeper patterns like changes in communication that may signal disruption before other indicators catch up. That kind of early signal can support conservation efforts that are timely and tailored to the needs of a specific population. 

Is the AI you’re referring to the same as using ChatGPT or an AI agent? Can you clarify the distinction for us? 

Models like ChatGPT and Claude are built on the same fundamental computer science breakthroughs that have inspired us at Earth Species Project. But those models try to do everything for everyone. 

By contrast, we build models that solve the tasks we need for studying animal communication. For example, during the collaboration with Raincoast, we built a model that is very good at detecting killer whale calls based on underwater hydrophone data. 

Most comparable to ChatGPT and AI agents is our model NatureLM-audio, which allows researchers to interact with their bioacoustic data with natural language and is able to perform a wide variety of tasks in bioacoustics. 

What does the project look like today? What has been built so far?

After the initial data collection pilot, we worked with annotators at Raincoast to build a working pipeline that turns raw field data into something we can actually analyze. 

The first step was developing a model that can automatically detect when orcas are vocalizing in long audio recordings. Instead of manually labeling hours of data, we can now process over 100 hours in about 30 minutes – a task that used to take days or even weeks of work. 

Once we know when calls happen, we can then connect those sounds to behavior using structured observations from drone footage and field notes, allowing us to align what the whales are saying with what they are doing.

Early indicators are really promising. One of our success metrics is whether our analysis can validate what our biology partners already know. For example, our analysis found that orcas are most vocal when the pod is in a high-energy state, like hunting, playing, or actively moving. We also found that they tend to call more when they’re spread out, using sound to stay connected across distance. These are known patterns, but seeing it clearly emerge from the data is an important proof of concept and helps validate that the pipeline is working.

The next step is to go deeper by linking different call types with specific behaviors. This is where tools like NatureLM-audio will play a bigger role, helping us move from simply detecting sounds to starting to understand what those sounds might mean.

The key milestone at this stage is that we’ve moved from raw, unstructured data to a system that offers a glimpse into a world that has, until now, remained beneath the surface.

What are you most excited about when it comes to this initiative? 

We already know orcas are an incredibly complex species. They have distinct dialects, with populations across the world and even within our study site using entirely different calls. Yet we still know very little about what those calls actually mean. This collaboration is an opportunity to examine their communication far more precisely, allowing us to understand it at a much deeper level. 

It can also sharpen conservation efforts. For example, are particular calls used for critical functions in group coordination or foraging being suppressed in the presence of boat noise? 

The most exciting part of the project for me is being able to ask these new types of questions for the first time.

What are you most looking forward to for the upcoming field season? 

We have some incredible data from our initial pilot study and we’ve achieved a proof of concept analysis pipeline. We learned a lot in collaborating with the team at Raincoast, so I’m most excited about building on those insights and gathering even more data that can help us start to associate call type with behavior.