🚧 This prototype is actively under construction. 🚧   Please provide feedback at github/danielsgriffin/SearchRights/issues or via email to daniel.griffin@berkeley.edu.

This table is optimized for larger screens. Rotate your device to landscape mode for the best experience.

Andi Search Metaphor Perplexity AI Phind You.com

Example searches

Help users practice new ways of imagining and formulating queries.

Research suggests that effective prompting of LLMs can be challenging (Zamfirescu-Pereira et al., 2023) and query formulation even in mainstream web search engines is complicated (Tripodi, 2018). Mollick (2023), not specifically re search systems, has called for "large-scale public libraries of prompts".

Are there default example searches?

Comment: It may be interesting to also consider whether these queries demonstrate an advantage of the generative search capabilities or not.
Yes

Ex.

  • [history of the birkin bag]
  • [What is the Mandalorian?]
  • [write a review of the best michelin star chefs in the world]
  • Go to apps, eg [go aws console]
Note that these are not links to searches because search-by-URL is not available in Andi.

Yes
Yes
Yes
Yes
Do default example searches have contextually relevant explanation on sourcing?
No
No
Partial

On mobile app there are example searches listed above the search bar, explained only by the word 'Popular'

No
No
Is there a "searchable repository of examples"?

The phrase "searchable repository of examples" comes from Zamfirescu-Pereira et al., 2023. They point to a "prompt book" for DALL·E. Other examples, also in the image generation domain, include Lexica (marketing itself as "The Stable Diffusion search engine"). See also the searchable LangChain Hub prompt repository for developers.
No
Partial

`# show-and-tell` channel in the Discord channel: 'Show off what you've found with Metaphor!'

Partial

**(1)** `# 💗 | sharing` channel in the Discord channel: 'Share cool results that you got with Perplexity! Please keep everything safe and friendly.'

**(2)** `# 🔎 │ ask` channel in the Discord channel: 'ask \@perplexity anything. Check pinned for usage.' (with a weekly challenge for the best question; pinned message show example queries (note these are queries of their Discord bot))

**(3)** A [Discover](https://www.perplexity.ai/discover) page shares 30 example queries.

Partial

`# 🤯impressive-results` channel in the Discord channel

No

`# create-and-share` channel in the Discord channel appears focused on image generation rather than searching with You.com All or YouChat.

Sharing searching

Support users in sharing their search experience with others.

Research demonstrates significant value in users communicating about their experiences with tools. Better support for sharing interactions with these systems may improve users collective ability to effectively question/complain, teach, & organize about/around/against these tools (i.e. "working around platform errors and limitations" & "virtual assembly work" (Burrell et al., 2019), "repairing searching" (Griffin, 2022), search quality complaints (Griffin & Lurie, 2022), and end-user audits (Metaxa et al., 2021; Lam et al., 2022)) and improve our "practical knowledge" of the systems (Cotter, 2022).

Can you share a link to the results or conversation?

Is there a share interface? For example, this is supported on OpenAI's ChatGPT and Google's Bard. It is not supported on Anthropic's Claude.
No
No
Yes
Yes
Yes

Requires sign-in.

Is there an explanation of how the share interface works provided or linked to in the interface itself?

Examples:

(1) The share interface in OpenAI's ChatGPT has a "More Info " link to their FAQ page: ChatGPT Shared Links FAQ:
OpenAI's ChatGPT share interface showing a more info link at the bottom.

(2) The share interface in Google's Bard has a "Learn more" link to a Bard Help page: Share your Bard chats: Bard share chat interface, showing a learn more link at the bottom.
N/A
N/A
Partial

The share link interface consists of the button, with a lock and labelled 'Share', and a popup showing the current 'Secret' state and an option to make 'Shareable:
Perplexity AI share link showing the current 'Secret' state and an option to make 'Shareable'.
Clicking 'Shareable' copies the link and presents options to facilitate sharing directly to Twitter, WhatsApp, and Facebook:
Perplexity AI share link showing 'shareable' selected, that it copied link to clipboard, and share options.

No

The share link interface consists of the share button icon and a popup:
Phind share link showing only Copy Link and Twitter.

Partial

The share link interface consists of the share button icon and a popup which says 'Share YouChat transcript' and 'Anyone with a link will be able to view your full transcript with YouChat in this session.' and presents a link to the search:
You.com Chat share link showing the text: 'Share YouChat transcript' and 'Anyone with a link will be able to view your full transcript with YouChat in this session.' and a link: 'https://you.com/search?q=who+are+you&tbm=youchat'.

Do shared links update if the sharee's interaction with the system continues?

Example: Here is the interaction in ChatGPT, per ChatGPT Shared Links FAQ:
If I continue the conversation after I create a shared link, will the rest of my conversation appear in the shared link?

No. Think of a shared link as a snapshot of a conversation up to the point at which you generate the shared link.
N/A
N/A

(1) Links to searches reproduce the search from the URL.

(2) Metaphor does not have follow-on queries/questions.

Sometimes

(1) It is not explicitly stated, but it appears that further interactions by the sharer on the same 'thread' may break the original link (`error=ThreadAccessNotAllowed` appears in the URL, and there is a temporary popup saying the same; try: example):
Text saying: 'Cannot view this thread'.

(2) Perplexity has another way to share as well: Collections. A search shared to a collection and then subsequently expanded is updated for the sharee(s).

No

It is not explicitly stated, but it appears that further interactions by the sharer on the same chat are NOT updated at the shared link.

No

It is not explicitly stated, but it appears that further interactions by the sharer on the same chat are NOT updated at the shared link.

Is there a social share card?

Quality of share cards may influence the engagement on social platforms.
No
Yes

Links to searches produce a social share card.

Yes

Share interface links produce a social share card.

No
Yes

Share interface links produce a social share card.

Are share links (or links to searches) indexed and searchable in search engines (or the system itself)?

Permitting share links to be indexed by search engines may allow searchers elsewhere to find and evaluate the quality of the system responses. Depending on the disclosure in the interface this may pose a privacy risk to users. It may be viewed explicitly as a growth strategy by a search system.

Evaluation details: I am currently looking only at `site:` searches on Google.

See a links to conversations about this on Twitter here.
N/A
Yes

You can find links to searches in Google: g[site:https://metaphor.systems/search].

Note: There are no 'conversations' on Metaphor.

Google result count: 6

Yes

You can find shared links on Google: g[site:https://www.perplexity.ai/search/]

Many of these searches may be sourced from the links provided on the Perplexity AI Discover page (which appear to be from Perplexity Bot (accessed via their Discord server). Perplexity Bot is identified as the account for many of the searches found on Google (example link here), others are anonymously sourced (example) or identified accounts (N.b. A few of the results are links I've shared myself: g[site:perplexity.ai/search danielsgriffin].

Google result count: 3,210

Yes

You can find shared links on Google: g[site:phind.com/search]

Note: Finding a Jan 19, 2023 tweet from Phind's CEO is what initiated my review of this criteria.

Google result count: 6

Partial

You can find links to searches on Google: g[site:you.com/search]

Google result count: 267

It does not appear that shared YouChat links are currently searchable on Google. @vijayabhaskarj reported finding chat links on Google on Jun 4, 2023.

Standards & Openness

Support interoperability and extensibility.

This includes checking if a platform supports searching by URL and if it offers an API.

Coming soon: Examine support for the OpenSearch protocol (mdn | GitHub | Wikipedia); record comments about open source; record contributions to & integrations from open source.

Can you search by URL?

Searching by URL has long been a staple approach to searching. This allows people to link to (live) searches (to bookmark or share with others) and to integrate searching within simple scripts. This also supports auditing.
No
Yes

Example: What is an LLM?

Yes

Example: What is an LLM?

Yes

Example: What is an LLM?

Yes

Example: What is an LLM?

Is there an API for search?

Searching by API may better support evaluations of the search results and the development of modifications or extensions building on or with the search system.

We do not currently evaluate the performance of the API or restrictions on access. You.com, for instance, says this:
If you are interested in being an early access partner please email api@you.com with your use case, background, and expected daily load.
Other search engines that provide some form of API include the Brave Search API, Google (through their Programmable Search Engine's Custom Search JSON API), Microsoft's Bing Web Search API, and the Yandex Search API.

You can also get programmatic or tool-based access to web search results from many third party providers that scrape search results from search engines. For example, academic researchers have conducted research using data from SerpAPI (Zade et al., 2022 ) and Ahrefs (Williams & Carley, 2023).
No
Yes

Metaphor announced beta access to their search API on Apr 21, 2023. It is now generally available at platform.metaphor.systems

No

Perplexity provides an API for interacting with open-source LLMs (see Oct 4, 2023 blogpost: "Introducing pplx-api". While not a search API, Perplexity also wrote in the blogpost that it would be providing a "Perplexity RAG-LLM API with grounding for facts and citations" in the near future.

No
Yes

Updated Oct 18, 2023
On Oct 18 LangChain announced support for YouRetriever, using a search API from You.com. LangChain shared a guest blog post from You.com ("You.com x LangChain") and an interactive notebook. Some documentation is also available on You.com's website (with an example query: [reasons to smile]: Documentation > Using the Search API
(Note: "To use the You.com Search API, you will need an API key. Please reach out via email at api@you.com for more details.").


Previously:
You.com does not offer an API for search or chat at this time. However, we are considering creating one in the future. If you would like to be notified when this happens, fill out the form here.
source

Does the system provide open-source models?

Providing open source models can advance the general search landscape as well as help users better understand the capabilities and limitations of the search system.
No
No
No
Yes

Phind provides three models, built on fine-tuning CodeLlama-34B on their proprietary dataset. Here is the Phind page on Hugging Face. Here is a Phind blogpost from Aug 25, 2023: Beating GPT-4 on HumanEval with a fine-tuned CodeLlama-34B.

No

Supporting feedback

Help users develop, submit, and share feedback.

Mechanisms to support feedback will shape user expectations, experiences, and future improvements of the search landscape. Some approaches may excessively conceal or otherwise control complaint rather than focusing on the users.

Is there a clear contextually relevant feedback method for core search functionality?

This is distinct from having a Contact or Feedback link buried off-page or in a footer.
No

Clicking 'Contact' on the left sidebar will open a page with links to a Feedback form and Bug Report form and a note that you can initiate them within the interface by typing `/feedback` or `/bug`.

No
Yes

There is a flag ("Not accurate") button at the bottom of the response. There is a question mark button (a dropup with options including a Help & FAQ link). The FAQ page has a Feedback link in the footer that sends the user to a survey.

Yes

There are thumbs up ("Good answer.") / down ("Bad answer.") buttons at the bottom of the response fields.

Yes

There are thumbs up ("This answer is helpful and/or accurate. Provide feedback on this result.") / down ("This answer is not helpful, accurate, and/or safe. Provide feedback on this result.") buttons at the bottom of the response fields. There is a general 'Give feedback' button at the bottom left of the screen (which links to You.com's Canny.io page).

Do users receive a copy of feedback?
Not yet evaluated
Not yet evaluated
Not yet evaluated
Not yet evaluated
Partial

Feedback left on You.com's Canny.io page appears to be persistent.

Is the feedback record easily shared by the user with a web-accessible link?
Not yet evaluated
Not yet evaluated
Not yet evaluated
Not yet evaluated
Partial

Feedback left on You.com's Canny.io page can be linked to.

Can users opt-in to contribute feedback to a commons-based infrastructure?

Sharing feedback from users in a transparent and privacy-preserving manner may improve the larger search landscape. Something like commons-based infrastructures, interfaces, or record repositories already exist for other factors, like IndexNow, Lumen, the ClaimReview schema, robots.txt, the OpenSearch protocol, research datasets and benchmarks, etc.

These are resources that search systems may contribute to, respect, or draw on.
Research shows that search systems already rely heavily on Wikipedia (see especially, McMahon et al. (2017 )).

Note: The proximate cause of adding this criteria was the responses from the CEOs of Perplexity AI and You.com to a Oct 15, 2023 comment from Yann LeCun (Chief AI Scientist at Meta):
Human feedback for open source LLMs needs to be crowd-sourced, Wikipedia style.

It is the only way for LLMs to become the repository of all human knowledge and cultures.

Who wants to build the platform for this?

No
No
No
No
No

In response to the question from Yann LeCun, Richard Socher (You.com's CEO) said (on Oct 15, 2023):

I've been saying similar things for a while. We'd be down to help coordinate at you.com and allow users to opt-in to contribute feedback for specific answers..

Works Cited

  1. Burrell, J., Kahn, Z., Jonas, A., & Griffin, D. (2019). When Users Control the Algorithms: Values Expressed in Practices on Twitter. Proc. ACM Hum.-Comput. Interact., 3(CSCW). https://doi.org/10.1145/3359240
  2. Griffin, D. (2022). Situating Web Searching in Data Engineering: Admissions, Extensions, Repairs, and Ownership [PhD thesis, University of California, Berkeley]. https://danielsgriffin.com/assets/griffin2022situating.pdf
  3. Griffin, D., & Lurie, E. (2022). Search quality complaints and imaginary repair: Control in articulations of Google Search. New Media & Society, 0(0), 14614448221136505. https://doi.org/10.1177/14614448221136505
  4. Metaxa, D., Park, J. S., Robertson, R. E., Karahalios, K., Wilson, C., Hancock, J., & Sandvig, C. (2021). Auditing Algorithms: Understanding Algorithmic Systems from the Outside In. Foundations and Trends® in Human–Computer Interaction, 14(4), 272–344. https://doi.org/10.1561/1100000083
  5. Lam, M. S., Gordon, M. L., Metaxa, D., Hancock, J. T., Landay, J. A., & Bernstein, M. S. (2022). End-User Audits: A System Empowering Communities to Lead Large-Scale Investigations of Harmful Algorithmic Behavior. Proc. ACM Hum.-Comput. Interact., 6(CSCW2). https://doi.org/10.1145/3555625
  6. Cotter, K. (2022). Practical knowledge of algorithms: The case of BreadTube. New Media & Society, 1–20. https://doi.org/10.1177/14614448221081802
  7. Tripodi, F. (2018). Searching for alternative facts: Analyzing Scriptural Inference in Conservative News Practices. Data & Society. https://datasociety.net/output/searching-for-alternative-facts/
  8. Mollick, E. (2023). One Useful Thing. Now Is the Time for Grimoires. https://www.oneusefulthing.org/p/now-is-the-time-for-grimoires
  9. Zamfirescu-Pereira, J. D., Wong, R. Y., Hartmann, B., & Yang, Q. (2023). Why Johnny Can’t Prompt: How Non-AI Experts Try (and Fail) to Design LLM Prompts. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3544548.3581388
  10. McMahon, C., Johnson, I., & Hecht, B. ( 2017 ). The Substantial Interdependence of Wikipedia and Google: A Case Study on the Relationship Between Peer Production Communities and Information Technologies . ICWSM , 11 ( 1 ), 142–151 . https://doi.org/10.1609/icwsm.v11i1.14883
  11. Zade, H., Wack, M., Zhang, Y., Starbird, K., Calo, R., Young, J., & West, J. D. ( 2022 ). Auditing Google’s Search Headlines as a Potential Gateway to Misleading Content . Journal of Online Trust and Safety , 1 ( 4 ). https://doi.org/10.54501/jots.v1i4.72
  12. Williams, E. M., & Carley, K. M. (2023). Search engine manipulation to spread pro-Kremlin propaganda. Harvard Kennedy School Misinformation Review, 4(1). https://doi.org/10.37016/mr-2020-112