iPhone, Galaxy, and Pixel Users Can Now Search On-Screen Content Visually

Apple's latest iOS 27 update introduces Visual Intelligence, a feature enabling iPhone users to search for objects appearing on their screen through screenshots. This functionality mirrors capabilities already available to Samsung Galaxy and Google Pixel users via Google's Circle to Search.

Visual Intelligence on iPhone

With iOS 27, Visual Intelligence expands beyond identifying external objects through the camera to analyzing content directly on the iPhone display. Users can take a screenshot of any app or screen and access Visual Intelligence tools at the bottom. These tools can include visual search for similar items, asking questions about the screen's content, summarizing text, or performing actions like adding event details to the Calendar. This integration allows users to, for example, search for a lamp seen in a video or convert an invitation from Messages into a calendar event without manually entering information.

However, advanced features like Visual Intelligence powered by Apple Intelligence require specific hardware. Supported devices for Apple Intelligence on iOS 27 include the iPhone 15 Pro, iPhone 15 Pro Max, or iPhone 16 models and later. While iOS 27 itself is compatible with a wider range of iPhones, the full visual search capabilities are limited to newer, more powerful models.

To use Visual Intelligence with screenshots on iOS 27:

1. Open the desired app or screen.
2. Press and release the side button and volume up button simultaneously to take a screenshot.
3. Utilize the Visual Intelligence tools displayed at the bottom of the screen, such as 'Search,' 'Ask,' 'Add to Calendar,' or 'Summarize.'
4. Swipe over a specific object to narrow the search.
5. Tap 'X' to delete or a check mark to save the screenshot.

Visual Intelligence can also be used with the iPhone camera. Supported iPhones can identify objects, translate text, or provide details about businesses. With Siri AI enabled in iOS 27, users can ask questions or search based on what their camera sees.

Circle to Search on Samsung Galaxy and Google Pixel

Samsung Galaxy devices running Android 14 with One UI 6.1 or later, and Google Pixel phones from the Pixel 6 series onwards, can utilize Circle to Search. This feature allows users to search for on-screen content without leaving their current application.

For Samsung Galaxy users:

1. Ensure Circle to Search is enabled in Settings > Display > Navigation bar.
2. Open the content to be searched (e.g., a webpage, image, or app).
3. Press and hold the Home button (circle icon) or the navigation handle (horizontal line) at the bottom of the screen.
4. Circle, tap, or drag over the desired object or text.
5. Search results will appear.

For Google Pixel users:

1. Confirm Circle to Search is enabled in Settings.
2. Open the content to be searched.
3. Touch and hold the navigation handle (horizontal line) or the Home button (center button) at the bottom of the screen.
4. Circle, highlight, scribble over, or tap the item.
5. Google search results will display.

Both platforms offer options to copy or translate text detected on the screen. Some applications may restrict screen capture or analysis due to security or copyright reasons, which could prevent Circle to Search from functioning within them.

Practical Uses and Privacy Considerations

These visual search tools are useful for various scenarios, including online shopping, identifying landmarks or restaurants while traveling, and quickly accessing information from text. Users can search for products seen in videos or photos, or translate foreign text encountered online.

Regarding privacy, Apple states that many Apple Intelligence requests are processed on the device, with complex requests using Private Cloud Compute where data is not retained. Users have control over Visual Intelligence settings under Settings > Siri. Google notes that Circle to Search activity may be saved with Google Search history, and users can manage these settings. Both companies advise caution when using these tools on highly sensitive screens such as banking or medical information.

Visual search tools offer a convenient way to gather information quickly. However, users are advised to verify the search results, as they may occasionally misidentify objects or provide inaccurate information. The tools are best viewed as an aid for initiating searches rather than a definitive source of truth.