Google's New Search: Combining Images & Text

Google Enhances Search with Multitask Unified Model (MUM) and Google Lens

At its annual I/O developer conference earlier this year, Google unveiled a significant advancement in artificial intelligence known as the Multitask Unified Model, or MUM. This innovative technology possesses the capability to comprehend information presented in diverse formats.

Understanding MUM's Capabilities

MUM can simultaneously process text, images, and videos. It then identifies relationships and draws connections between various topics, concepts, and ideas. Google recently announced plans to integrate MUM into its products, starting with an update to Google Lens.

Google Lens: Expanding Visual Search Functionality

Google Lens is a powerful image recognition tool. It utilizes a phone’s camera to facilitate numerous tasks. These include real-time translation, plant and animal identification, text extraction from images, and finding visually similar items.

Google intends to leverage MUM’s abilities to enhance Google Lens. The upgrade will allow users to combine visual searches with text-based queries.

How the New Feature Works

Imagine you find a shirt you like online. You can use Google Search and then activate the Lens icon. You could then ask Google to locate the same pattern on a different item, such as socks.

By adding a text query like “socks with this pattern,” you provide Google with more specific direction. This approach can yield more relevant results than using text input alone.

Addressing Complex Queries

This feature will be especially beneficial for searches that Google currently finds challenging. These are often queries where a visual element is difficult to describe accurately with words. Combining images and text into a single query improves the chances of receiving pertinent search results.

Consider a scenario where a bicycle part is broken, and you need repair instructions. If you don’t know the part’s name, you can use Google Lens to identify it. Then, type in “how to fix.”

This could directly link you to the relevant segment of a helpful video tutorial.

Google's Strategy for Staying Relevant

Google views these AI-driven initiatives as a means of enhancing the utility of its products. By integrating the phone’s camera into the Search experience, Google aims to maintain its relevance in a changing market.

Many core search use cases are now shifting to alternative platforms. For example, a significant number of shopping searches now begin directly on Amazon.

iPhone users frequently turn to Siri, Spotlight, the App Store, or dedicated apps for assistance. Apple is also actively developing its own search alternative, as evidenced by the iOS 15 update to Spotlight search.

Broader Applications of MUM

Google also announced at its Search On event that it is applying MUM to other areas of Google Search and video searches. The company is committed to expanding the use of this powerful AI model.

Rollout and Testing

The Google Lens update is scheduled for release in the coming months. However, it will first undergo “rigorous testing and evaluation,” a standard procedure for all new AI models before deployment.