Artificial Intelligence (AI) has often been hailed as a “blue pill” solution, promising to resolve every conceivable challenge, said Bharani Subramaniam, CTO of Thoughtworks for India and the Middle East, but he brings forth a perspective, “Implementing AI is just the start of it, the real challenge lies in scaling AI effectively and navigating the intricacies of retrieval and embedding systems.”
AI’s journey begins with converting human-readable text into numbers through a process called embedding. “When you give AI a piece of text, it needs to convert it into numbers because that’s what models understand,” Bharani explained. However, this process is more complex than it appears. Effective AI systems require not just exact matches but also a hybrid approach that includes embedding and similarity searches.
“If you give an input, you’re supposed to ask something relevant,” said Subramaniam of Thoughtworks.
RAG Architecture
Many banks have ventured into AI by experimenting with the RAG (Retriever-Augmented Generator) architecture. Here, the retriever component is substantial, while the generator remains small. “It’s super easy to start,” Bharani remarked. However, he cautioned, “Once you have it, it’s extremely difficult to scale.”
The retrieval process can compress vast amounts of data but may not yield 100% accuracy. “It can be 80%, but not wholly,” Bharani explained. This inherent limitation poses challenges in scaling AI systems and raises questions about their reliability.
Despite these challenges, Bharani firmly believed that the AI industry is not a bubble. “Are there bugs? Yes. But this similarity match, even though it isn’t very accurate, you cannot live without it.” Exact matches, like those used in traditional databases, fail to scale in AI-driven environments.
The Hybrid Approach
According to Bharani, the key to success lies in adopting a hybrid retrieval strategy. “You have to index your document using two ways: traditional exact matches and embedding-based similarity searches.” He emphasised that no single approach can address all challenges, advocating for a “bag of techniques” approach to achieve optimal results.
Challenges in Retrieval and Context Management
Retrieval remains a persistent problem in AI, one that has evolved over two decades. Bharani highlighted how search capabilities have become integral to modern AI systems, likening it to a human-like behavior, “If you give a very big context to a model, it behaves like a human. The model will pick the beginning and end, and the middle will be lost.”
