How an MIT Photo Remake Became an AI Benchmark

For decades, a single black and white photograph known as the 'camera man' has been an iconic staple for engineers in image processing. The image, showing a man with a camera in a field, was a go-to test image within Matlab, a popular programming platform for scientists and engineers.

The Legacy of the Camera Man

For Kush Varshney, now an IBM Fellow, this image was a familiar sight during his graduate studies at MIT. But one day, a detail in the background sparked a moment of recognition. “I had an epiphany while reading a paper at my desk in [MIT’s] Stata Center,” Varshney recalls. The imposing building in the distance was none other than MIT’s Great Dome. “I know where this is!”

A Modern Homage to a Classic

This realization inspired Kush and his twin brother, Lav, to recreate the famous shot. They went to the MIT soccer fields, found the right spot, and with a borrowed tripod and camcorder, they captured their own version. Kush, in an overcoat and dress shirt, posed as the modern camera man while Lav took the picture. For years, the photo served as the profile picture on Kush's personal website.

Becoming the New Standard

A decade later, this personal project took on a new life. Developers for scikit-image, the image-processing library for the popular scikit-learn machine learning framework, ran into a problem. They wanted to include the original camera man photo in their library but couldn't track down the copyright holder. During a search on GitHub for a suitable, rights-cleared replacement, a friend who knew of Varshney's photo pointed them to his remake. After the Varshney brothers granted permission, their homage officially became one of the new test photos in the scikit-image library, as Kush later announced on Twitter.

Rethinking AI Benchmarks in a Modern World

This unique story has given Varshney a platform to reflect on the state of AI development. He sees the 'camera man' as an early form of a benchmark, similar to how HELM and MMLU are used to evaluate modern chatbots today. However, he cautions that these standardized tests often lack real-world context. Varshney advocates for a shift in AI evaluation, arguing, “We need fewer ‘camera men’ and more real-world testing.” He believes AI models should be evaluated in environments that closely resemble where they will ultimately be deployed.

Copyright and Creativity in the Age of AI

The journey of his photo from a personal project to an open-source standard naturally leads to a discussion of copyright in the era of generative AI. Varshney credits Creative Commons for making vast amounts of data available, which later proved essential for training powerful AI models. He emphasizes that IBM Research takes copyright seriously, only training its models on data it has permission to use. However, he also points out the legal complexities, noting that a Harvard law professor found that copyright standards for AI training vary uniquely across 17 different countries.

Varshney offers a philosophical take, viewing copyright as a “product of capitalism.” He compares the content produced by LLMs to the work of ancient bards like Homer, who composed and performed epic stories without a concept of authorship. “We should support this ‘tradition’,” he suggests, by paying for the computational costs of AI, thereby allowing human creators to focus on producing truly original work.

The Future of AI Attribution

With the rise of AI-generated content, the question of attribution becomes critical. Is a new standard, like a Creative Commons license for AI, needed? Varshney believes so. “We need a way to mark AI-generated content so we can recognize what’s original,” he states. His team at IBM has already developed a potential solution: a self-reported AI attribution mark that can be placed at the end of a text. It’s a simple but powerful idea, and as he puts it, “Let’s see if it catches on.”