We took Glean Chat for a test drive – here’s how it went

Glean recently unveiled Glean Chat – a ChatGPT for the enterprise that interacts with all of your company’s knowledge and data to provide answers for any work-related questions you might have. We engineers on the Search & Intelligence team recently took it for a test-drive. 

The results were magical! It felt like talking to an all-knowing expert at the company. Here’s what really impressed us: 

• It gets its answers correct!
• It’s able to reference its sources.
• Its answers are tailored to our enterprise.
• Even for queries that aren’t company-specific, its answers are precise and focused on our own internal use-case.
• It’s able to combine information from varied sources to generate comprehensive answers in a way that’s hard and time-consuming for even subject-matter experts to do.
• It generates text and code just as clear as, if not clearer than, what we write ourselves.

Our Glean Chat session journey

The Search & Intelligence team at Glean is obsessed with making sure that users are able to find their desired information near the top of the search results page. One of the metrics we use to measure user search experience is Mean Average Precision (MAP). So, we started a Glean Chat session to see how well it understood this important metric.

We started by asking what the MAP metric is.

Glean Chat came back with a complete answer. It not only explained what the metric is and how it’s computed, but it also described clearly what the metric is trying to capture. It also discovered from our enterprise corpus what the typical value of this metric is across our customers. We didn’t ask for it, but it does belong in a comprehensive answer to our question.

More impressively, this complete answer wasn’t extracted from just a single resource in our enterprise corpus. The metric is defined in a few internal documents, and the typical values are discussed in other resources. To be able to combine all these sources of information into a coherent, comprehensive answer within seconds was amazing.

The answer left some terms undefined – for example, it didn’t describe what the “average precision” of a clicked document is. So, we asked it to give a detailed algorithm for the metric.

This is a truly awesome answer! Firstly, it’s correct – just as good as if we’d answered it. Secondly, and just as importantly, it’s crystal clear. Our internal documentation doesn't contain a description of this algorithm as clear as this. 

The internal documentation is decent, but it was written by busy people with limited time to spend on polishing it for thoroughness and consistency. For Glean Chat to be able to digest all that documentation and come up with a clear, correct description is incredibly useful.

OK, so Glean Chat can talk. But can it code? We then asked it to implement the MAP metric in Python.

Beautifully written, compact, clean, and correct code – generated within seconds. We were blown away! 

The same session with ChatGPT

MAP, the topic of our Glean Chat session above, is an industry-standard metric. So, even ChatGPT should be able to answer questions about it reasonably well. However, there are differences.

Here’s what ChatGPT had to say about the MAP metric.

ChatGPT also gave a decently comprehensive answer. But it had no idea which MAP metric we were interested in as Search & Intelligence engineers at Glean, so it chose to enumerate three of the most common acronym expansions for MAP. Reasonable response, but it read like a generalized answer that touched on only surface-level details. 

On the other hand, Glean Chat sits on top of our internal company corpus of documents and data. It understands our work, and so it knew exactly which MAP metric we were talking about. Glean Chat’s answer was precise, to the point, and used the exact terminology we use internally when talking about MAP. 

Not only that, Glean Chat was able to make the answer more relevant for us by providing the range of values this metric takes among our customers. This level of customization isn’t possible without a deep understanding of the enterprise corpus and the user context. In addition, Glean Chat was able to provide useful references to internal documents that talk about MAP.

Next, we asked ChatGPT for a detailed algorithm for MAP.

Again, ChatGPT’s answer is decent, but generic.

• Steps 1-2 were spent setting up the search context – which was unnecessary from our perspective.
• Step 3 used the notion of “relevant documents'' – this is not wrong, but following our internal vocabulary, using the specialization of “clicked documents” would have been more appropriate, and it would have made the answer more comprehensible to members of our team.
• Also, Step 3 uses a boolean variable “Relevant at K” to run a summation over all documents on the page. It’s functionally equivalent to Glean Chat’s description involving only clicked documents, but Glean Chat’s description would feel more familiar to anyone reading or editing our internal documents on this topic. It’s also closer to the actual implementation of the concept in our system.
• The note at the end doesn’t apply to our context.

ChatGPT gave serviceable, generalized answers more appropriate for a broad audience, whereas Glean Chat sounded like a local enterprise expert who might actually have written our internal documentation – capable of providing precise guidance. It’s what ChatGPT for the enterprise should look like! 

Conclusion

In just one test drive, Glean Chat really impressed us with its abilities to synthesize information in the context of our enterprise and generate comprehensive answers and citations to help us in our workflows. This is incredibly exciting for the future of work – imagine an all-knowing expert from your company that’s always ready to answer any query. It’s enterprise AI on another level. Magic! 

If you’re interested in learning more, sign up for a demo today!