When out on Puget Sound with killer whale experts, I’m often impressed by how quickly such folks can identify individual whales by name or number, even when a group of orcas is swimming a fair distance away. I can’t begin to do that, because I have never studied the catalogs of dorsal fins and markings used to distinguish one whale from another. But now an advanced computer system can provide IDs for even casual whale observers.

Thanks to artificial intelligence with deep learning, amateur whale watchers can now contribute to science by submitting photos of orcas to a website. From there, a computer model takes over, compares each submitted photo to images of known whales, and then “predicts” each whale’s identification. The system — something like facial recognition technology used to identify people — is designed to get faster and more accurate over time.

“The model creates a mental map of all the features of an individual,” said Jared Towers, executive director of Bay Cetology, a research group that led development of the system, called Finwave. “It is accurate in most cases — about 92 percent accurate.”

That 92 percent is the current success rate for predicting Bigg’s, or transient, killer whales — the kind of orcas that hunt marine mammals along the West Coast, including Puget Sound. The computer model for Bigg’s killer whales was released for public use in February. Now anyone willing to follow prescribed guidelines may submit original photos of Bigg’s whales and get back an identification of the animal.

To “train” the computer model, killer whale experts review the submitted photos and tell the model whether its result is right or wrong, Jared told me. Over time, based on past successes and failures, the model has advanced each time the software is updated. Human operators can’t be sure which physical attributes of the whales the computer uses in the identification process.

“We’re not exactly sure how it is doing what it is doing,” Jared said. “There’s a mysterious component to it. We updated the model a few weeks ago, and our team was amazed at how accurately it was predicting this whale and that whale.”

For example, he added, the model was asked to identify two female orcas with nearly identical dorsal fins in a photo taken from a distance. Jared did his best to identify the animals himself.

“I was not sure who was who until I zoomed in all the way, and then it took time for me to identify them,” Jared said. “The model predicted them accurately before I had a chance to confirm.”

In addition to Bigg’s killer whales, the model is being used for the endangered southern resident orcas, the fish-eating whales frequently seen in Puget Sound during the summer and fall. At this time, use of the system for southern residents is generally limited to experienced observers and photographers, although anyone can request access.

“I don’t think the model will ever be as good as those of us who work with whales all the time,” Jared added, “but it will be better than everybody else.”

Lightning-fast analysis

The Finwave algorithm was developed by researchers at Bay Cetology in British Columbia in affiliation with Alexander Barnhill at Friedrich-Alexander-Universität, one of the largest universities in Germany. As designed, Finwave can be adapted and used to identify any animal with a dorsal fin on its back. At least a dozen research groups around the world have employed the model for local populations of marine mammals. They include Lahille’s bottlenose dolphins in Brazil, Bryde’s whales in South Africa, Sei whales in Argentina and at least five killer whale populations. Theoretically, it could be used to identify sharks as well.

“The whole point is that you are using existing knowledge and applying data that you want analyzed,” Jared said. “It’s like putting together pieces of a puzzle, and (Finwave) can do this very quickly.”

Bay Cetology was founded by Jared Towers in 2017. Jared grew up around orcas as part of a family that operated a whale-watching business out of Alert Bay near northeast Vancouver Island.

“As a kid, I got to be on the water all summer with killer whales,” he said. “In the 1980s, this part of the world was one of the few places that people could go to study killer whales.”

Jared’s interest in observing orcas turned into serious studies that eventually took him throughout the world, as he expanded his research to other species of marine mammals. Back home, he continued to watch the various orca populations grow and evolve over the past five decades.

In addition to Jared, Bay Cetology includes two biologists, two data analysts and a developer/programmer.

Among the group’s many research projects is the ongoing monitoring and census reports for more than 300 Bigg’s killer whales, including records of births and deaths. In 2019, Jared was the primary author for a photo-ID catalog of the known Bigg’s population along the West Coast. These projects were funded by Fisheries and Oceans Canada.

This population of marine-mammal-eating orcas appears to be growing at about 4 percent per year, according to the latest report (PDF). That growth stands in contrast to the endangered southern residents, whose population is about the same as 50 years ago, following the capture era, despite periods of increases and declines.

Advancing technology

The idea to use artificial intelligence to identify killer whales and other marine mammals grew out of a struggle to manage a burgeoning mass of data involving killer whales throughout the Northwest, including photos and observations from a multitude of sources, Jared said. Being able to speed up identifications, curate the data and share the information widely are the key features of Finwave. The algorithm itself, FIN-PRINT, was first described in 2021 in the journal Nature Scientific Reports.

Finwave follows earlier efforts in the rapidly growing field of AI to tell one species from another in all sorts of terrestrial and marine habitats plus refinements that allow identification of individual animals, such as North Atlantic right whales.

“It has been an experimental learning process,” Jared said. “We’re now four years into the model and we’re thinking, ‘Holy smokes; it is getting better than we thought possible.’”

The computer model begins its analysis by searching a photo for triangle-like shapes in the image. That sets the target to dorsal fins. A square is drawn around each fin and its associated “saddle patch” of unique gray markings behind the fin. Deep learning is accomplished through a convolutional neural network (CNN) that allows the computer to first visualize and enhance the target by filtering out factors such as poor weather, blurring or odd angles. Finally, the model uses a multi-layered classification system to efficiently search for the best match to previously known killer whales.

Work on the model continues, and the next version of the algorithm will add the ability to key in on the eye patch — a white oval marking above each eye that is unique enough to provide another important identifying factor for killer whales.

Initial training for Finwave involved running the model to classify 121,000 images of individual killer whales, which had all been identified previously by humans, forming the knowledge backbone of the system.

 Although whale-sighting information can be widely shared among researchers using the system, ownership of the data is maintained as the intellectual property of those submitting it. The compilation of vast information across time and space helps support studies of the orca population, including changes in numbers, movements, social structure and behavior.

Widely used system

So far, more than 500 people have submitted Bigg’s photos and data gathered during more than 6,000 encounters along the West Coast. Naturalists, who serve as expert tour guides on whale-watching boats, frequently contribute high-quality photos, sighting locations and times, along with other details that have expanded scientific knowledge of the Bigg’s orcas, Jared said.

Erin Gless, executive director of the Pacific Whale Watch Association, said she finds that Finwave can serve as an online catalog of whale IDs, especially considering that the latest official catalog is six years old. “There have been a lot of births and changes since then,” she said in an email. “It’s great for letting you see when and where a family was last seen, confirm when a specific individual was born, etc.”

As with human experts, successful identification depends in large part on the quality of the photo, especially around the dorsal fin and saddle patch. High-quality images normally require a good camera with a telephoto lens to stay a safe and legal distance from the whales.

While anyone can use Finwave for Bigg’s killer whales, Jared said, “we don’t usually encourage people to send in photos from their iphone.” The truth is the resolution and wide-angle nature of most cell phones don’t provide the image quality needed for the model to identify individual whales, he said.

Michael Weiss, research director for the Center for Whale Research, said he sees the benefits of Finwave not only for researchers but also for casual observers who would like to know more about the whales they see — whether they are viewing from shore or from a boat. CWR maintains the official census records for the southern resident killer whales, which currently number 74.

“The time-consuming part for me is to actually load up the images and correct the model’s predictions,” Michael told me in an email. “The predictions themselves are very, very fast.”

Michael said the model’s prediction rate for southern residents is relatively high and getting better, but his team does not use the system for initial identification of southern residents, because the 74 whales are well known to CWR researchers.

 “At the moment, it’s actually quicker to just ID them yourself,” he said. “I think it does have value for very large populations with some rarely seen individuals… and I can definitely see how it speeds up IDs for Bigg’s and other large populations.” 

While Finwave continues to advance and expand to other marine mammal populations, other new systems based on artificial intelligence also are moving forward. Jared contributed to a new system based on 39 catalogs of 24 species of whales and dolphins from throughout the world. A single animal in a photo can be identified not only by species but by individual with 87 percent accuracy — or up to 95 percent accuracy for seven species. Model development involved 56 researchers from six continents.

That computer model, described in the journal Methods in Ecology and Evolution, was the winning entry in a 2022 competition to develop an algorithm that could recognize individuals from a multiplicity of species. It was submitted by a team led by Philip Patton at the Marine Research Program at the University of Hawaii.

“There are a variety of platforms out there using AI,” Jared noted. “I think we will continue to see studies as AI is applied to many more species. It is exciting to see what will come next.”