Before you can trust the AI, find out if your humans agree

The problem

Corkscrew Swamp Sanctuary’s trap cameras produced images faster than the team could review them. Images were downloaded by hand every day and reviewed three times each, twice by volunteers and once by staff, with results tracked in a Google Sheet. The work depended on volunteer availability and could not keep up with the backlog.

We expected to build something. The research pointed somewhere else: first find out whether an existing platform could handle the sanctuary’s images reliably.

Check the labels before judging the models

We researched six platforms against the sanctuary’s needs: classification, supported media, storage, data-sharing terms, and fit with how the team already worked.

Before scoring any of them, we compared two volunteer-labeled datasets covering 2,026 images. They agreed on 63% of labels, and frames marked blank accounted for much of the disagreement. If the people disagree that often, a model’s accuracy score against either set means very little.

So we built a reference set for the evaluation, and sanctuary staff reviewed it and corrected the labels we had gotten wrong.

Compare what each tool actually does

We evaluated selected tools, including Wildlife Insights, WildTrax, and MegaDetector, against that reference set. Spotting an animal in a frame and naming its species are different jobs, so a single accuracy ranking would have misled the team. We recommended trying Wildlife Insights.

The outcome

The sanctuary adopted Wildlife Insights. It went on to classify more than 90,000 images, cutting manual review and getting the team, in Alex Meinders’ words, “caught up for the first time ever.”

The team also told us Corkscrew used the evaluation to help other local preserves adopt the same approach.