There is more to chat than just Q&A as Vectara debuts new RAG powered chat module

In the generative AI era chatbots have become even more pervasive than ever before – but are they actually more helpful and accurate?

Gen AI platform builder Vectara is out today with a new module for its platform that aims to help enterprises build and deploy highly accurate chatbots. Vectara’s platform takes a Retrieval Augmented Generation (RAG) approach with its Boomerang vector embeddings to provide the most up-to-date information and reduce the risk of hallucination. Vectara first emerged from stealth in October 2022 and has been incrementally growing its platform capabilities to meet user requirements. The addition of a chat module further extends the platform.

“When you are using our chat feature, it’s actually using RAG to generate the answer that you get,” Tallat Shafaat, co-founder and Chief Architect at Vectara told VentureBeat. “So the answer is not just coming blindly, it’s coming from your own documents using the RAG platform.”

A modern chatbot is more than Q&A it’s a conversational AI

So what’s different with the new Vectara Chat module from what the company has been doing to date?  According to Amr Awadallah, co-founder and CEO of Vectara, it’s all about scale and persistence.

“Historically, our API if you were using us was focused more on Q&A, you ask a question, you get back the answer,” Awadallah explained to VentureBeat.

If the user wanted to ask a follow-up question, the user would have had to restate the original question because Vectara was taking a stateless session approach where it didn’t have the history of the conversation. Awadallah noted that Vectara’s customers have had to build their persistence layer to maintain a stateful conversation. That situation will now change with the Vectara Chat module that has persistent memory on the Vectara site to maintain the state.

“What we have launched is an ability that is an extension to our API that retains the history of each conversation,” Awadallah said. “So you don’t have to rewrite and rephrase the questions and refer to things from the past.”

In terms of deployment, Awadallah said that Vectara provides both an API and simple widgets that make it easy for organizations to deploy and use the chat module. A chat widget can be dropped into a website or application with just a few lines of JavaScript and HTML.

Going further, Shafaat added that the plan is to expand Vectara Chat with more enterprise management features. For example, an account owner whose customers are the ones who are using the chat feature will be able to look at customer chat histories in a semantic way to find out how people are feeling about different things and the types of queries being made. Vectara will also enable RAG-based queries against the Vectara Chat module, to ask questions about user chats.

Reducing hallucinations and bias in gen AI

A primary concern for enterprise usage of gen AI is the risk of hallucination. The RAG approach is one of many that Vectara is taking to help reduce the risk of inaccurate responses for its chat module.

Awadallah noted that the Vectara approach provides explainability with citations to help ensure accuracy. The system also integrates bias mitigation capabilities through an approach the company has pioneered known as – maximal marginal relevance. 

“Maximal marginal relevance increases the diversity of the results we’re bringing back,” he explained. 

Awadallah said that on questions where it’s a topic of debate, where there are different opinions if there isn’t a good algorithm that can fetch all points of view, the system will give a biased response.

“We make sure that we’re getting the primary point of view but the secondary point of views even if they are less relevant,” Awadallah explained.

Originally appeared on: TheSpuzz