Last year, Apple deployed a mechanism on its devices that identifies landmarks in user photos and then allows them to find pictures on users’ mobile devices and computers based on the names of those same landmarks. The manufacturer enabled this feature without the explicit consent of device owners, and they have only now begun to notice this.
Image source: apple.com
The feature was called “Enhanced Visual Search,” and was first announced to the public by programmer Jeff Johnson. According to him, the new service began working on October 28, 2024 on iOS 18.1 and macOS 15.1, and Apple did not bother to clearly explain how this technology works.
В датированном 6 ноября 2024 года документе на сайте Apple говорится: «Улучшенный визуальный поиск в приложении „Фото“ позволяет искать фото по достопримечательностям и другим ориентирам. Ваше устройство конфиденциально сопоставляет места на Ваших фото с каталогом мировых локаций, который хранится на серверах Apple. Мы применяем технологии гомоморфного шифрования и дифференциальной конфиденциальности, а также используем узел протокола OHTTP, который скрывает IP‑адрес. Это не даёт Apple возможности собрать какую‑либо информацию о том, что изображено на Ваших фото. Функцию „Улучшенный визуальный поиск“ можно выключить в любой момент. Для этого на устройстве iOS или iPadOS откройте „Настройки“ –> „Приложения“ –> „Фото“. На Mac откройте приложение „Фото“ и выберите „Настройки“ –> „The main ones.”
A detailed explanation of how the technology works is provided in an article on Apple’s website dated October 24, 2024, around the same time that Enhanced Visual Search was launched. It is based on a machine learning model that runs locally on devices. She analyzes photographs for the presence of landmarks. If the AI finds a match, the system calculates a vector embedding—an array of numbers representing that portion of the image. The vector embedding is then encrypted using homomorphic encryption, which allows computations to be performed on the encrypted data without decrypting it.
After this, the data is sent to a remote server, where a search is performed in the database of attractions. The results are returned to the user’s device, which Apple claims is the only one capable of decrypting the information. An additional level of protection is the differential confidentiality mechanism, which excludes the identification of a subject in the database, even if an attacker gains access to the entire database.
However, experts interviewed by the British publication *The Register* expressed doubts about the new Apple service. The company has enabled “Enhanced Visual Search” by default for all users, including those who have turned off photo uploads to iCloud for privacy reasons.
More than three years ago, Apple planned to scan images on users’ devices for child abuse material, but abandoned the initiative due to privacy concerns. The new system, critics say, forcibly processes locally stored photos and sends data about all images, not just those with suspicious hashes.
Additionally, it remains unclear why Apple launched “Enhanced Visual Search” without making sure users of its devices were properly notified.