Corporate Decision-making with AI

Introduction

One of the most powerful applications enabled by Large Language Models (LLMs) is the development of sophisticated Question-Answering (Q&A) chatbots. These chatbots are designed to interact with users by providing answers to questions based on specific information sources. They utilize an advanced technique known as Retrieval Augmented Generation (RAG), which combines the capabilities of retrieving relevant data from vast databases and generating accurate, coherent responses. This integration of data retrieval and response generation makes Q&A chatbots exceptionally effective in providing precise and contextually relevant information to users. The right question answered at the right time can facilitate effective decision-making.

In this article, we will describe how RAG can be used for decision-making. We’ll also explore how an old concept of computer science—agents—can be leveraged to harness even more power from large datasets and LLMs. This article is intended for decision-makers, managers of AI solutions, and data scientists who are responsible for AI adoption in their businesses.

RAG with Agents

Agents are self-conscious (🤔) entities that perform tasks to achieve specific goals on behalf of governments, organizations, or even for self-serving purposes. Similarly, software agents are designed to perform tasks autonomously or semi-autonomously to accomplish various objectives. In the past, these agents assisted in only IT-related activities, such as network management, data processing, and user support. However, with the rise of Large Language Models (LLMs), these agents can now undertake more complex tasks, involving decision-making and acting based on their judgments.

Decision Making

Decision-making is a crucial aspect for humans. But what is decision-making in philosophical terms?

A decision is seen as the resolution to behave in a certain way and as such it involves both a choice and a commitment to that choice. It focuses on how one makes choices among alternatives and also on how one identifies the available alternatives to influence the goal.
Philosophy of Decision Making

When we involve software agents in this task, they can also make certain choices to achieve their goals. These choices can be open-ended or confined by third parties. They need information, resources, and capabilities to make the right choices, which raises the question of intelligence. A pre-trained LLM is sufficiently intelligent to make low-level choices. However, with Reinforcement Learning with Human Feedback (RLHF), this capability becomes more powerful, especially in domain-specific scenarios.

As of today, according to my evaluation, the following models make smarter decision in most cases:

Gpt-4o from OpenAI
Claude 3.5 Sonnet from Anthropic

Acting According Judgment

Acting according to judgment is purely an engineering challenge involving several steps. Typically, it starts with setting a goal and demands resources, information, and the capability to move towards that goal. Intelligence is then employed for optimal decision-making to progress towards the goal, acting on choices, evaluating outcomes in terms of goal achievement, making further smart decisions, reevaluating, and so on. This cycle continues until the goal is achieved.

Answer evaluation is also a fundamental part of this cycle, requiring not only intelligence but also experience. It is the user's responsibility to provide adequate experience to the agent to evaluate and assess progress towards the goal.

Many papers have been published on various iterations of these solutions, and numerous frameworks have been developed by brilliant developers. At their core, they all aim to achieve specific goals using available resources and intelligence. It's important to remember that intelligence is no longer solely a human attribute.

Importance of RAG in Business Applications

There are numerous ways an organization can benefit from RAG-based agents. While I won't delve into the details, here are some key points in brief:

Informed Decision-Making: RAG provides comprehensive and context-specific data, supporting strategic planning and effective problem-solving.
Enhanced Customer Insights: By improving customer service interactions, RAG offers valuable insights into customer needs and preferences, aiding in better decision-making.
Increased Efficiency: Automating data retrieval and analysis with RAG boosts productivity, allowing quicker and more accurate decision-making.
Competitive Edge: Leveraging RAG technology helps organizations stay ahead of market trends and seize new opportunities, leading to better strategic decisions.
Scalable Knowledge Management: RAG ensures access to up-to-date information and best practices, enhancing an organization's ability to make informed decisions based on the latest knowledge.

Agentic RAG

In the data science community, Agentic RAG refers to the integration of RAG with LLM Agents. In our AI lab, we have conducted some research on implementing RAG with medical documents related to autism. An Agentic RAG is built on a small number of research papers just to demonstrate the capability of Agentic RAG.

Our sample Agentic RAG is designed to investigate autism and analyze possible treatments based on research papers. We can ask various questions focused on autism and expect answers derived from the information provided in these papers.

As a user, I will provide the goal to find the answer to the following question:

—------ GOAL

Q: What are the treatments for autism?

—----------------

As a user, I am also providing the resources for the agent to work with: the latest research papers on autism and the capability of web search.

—------- Resources

Latest research papers
Web-search cability

—-----------------------

Now, our agent will operate according to the following diagram. First, it will retrieve relevant information from the research papers and then generate an answer based on this information. At this point, it will need to choose between the following two options:

If the generated answer is satisfactory, it will submit the answer as the final response;
The generated answer is not satisfactory -

it will conduct a web search for more information and regenerate the answer based on the new data.

There are cyclic tasks involved in answer generation, evaluation, and regeneration until the answer is satisfactory. The aspect of satisfaction is twofold:

Groundedness (how much the answer is grounded on the fact)
Answer Relevance (how much the answer is relevant to the actual question)

We can delve into more detail in future articles. However, these satisfaction levels determine whether the agent will continue using different resources to answer the initial question. This decision can be challenging—even for humans. The powerful aspect of these agents is their ability to investigate relentlessly, utilizing all available resources until a satisfactory solution is achieved.

Here is the output from different points of execution for the quest of answering our question `What are the treatments for autism?`:

====================================================

--- AGENT NODE---

INPUT :What are the treatments for autism?

OUTPUT : (decision making …)

--- GENERATE NODE ---

INPUT : Context information: a developmental disability or mental …

OUTPUT :This text focuses on Functional Communication Training (FCT) as a treatment for behavioral issues, particularly in individuals with autism. It does not provide a comprehensive list of all autism treatments.

--- EVAL CONDITIONAL EDGE---

INPUT :This text focuses on Functional Communication Training (FCT) as a treatment for behavioral issues, particularly in individuals with autism. It does not provide a comprehensive list of all autism treatments.

OUTPUT :No, This answer does not satisfy the question asked.

--- WEB SEARCH NODE ---

INPUT :No, This answer does not satisfy the question asked.

OUTPUT :Initiating web-search …

--- GENERATE NODE ---

INPUT :However, most people with ASD respond best to highly structured and specialized programs…

OUTPUT :This text mentions highly structured and specialized programs, and educational treatments like the TEACCH approach. It also mentions hyperbaric oxygen treatments and intravenous immunoglobulin (IVIG) infusions but notes that these are not proven effective.

--- EVAL CONDITIONAL EDGE---

INPUT :This text mentions highly structured and specialized programs, and educational treatments like the TEACCH approach. It also mentions hyperbaric oxygen treatments and intravenous immunoglobulin (IVIG) infusions but notes that these are not proven effective.

OUTPUT :Yes, This answer does satisfy the question asked.

============================================

The first evaluation failed as the agent could not prepare a proper answer for the actual question but the next evaluation was satisfactory, because the agent fetched more relevant information from the web-search tool. Thus could generate a satisfactory answer for the question.

This example demonstrates that decision-making is significantly enhanced with RAG, making it a valuable tool for corporate settings when making critical business decisions, such as investments. By leveraging RAG, organizations can ensure that decisions are well-grounded in comprehensive data analysis and relevant information retrieval. This thorough approach allows for more informed and strategic choices, ultimately driving better outcomes and fostering success in the competitive business landscape.

As users, it's time to change our mindset about how we use intelligent software systems. Today, we mostly rely on API-based systems and expect outcomes within minutes. However, with these new breeds of systems, we need to be more patient in obtaining the final answer, allowing time for proper evaluation and decision-making. It’s worth the wait because a better decision leads to greater success.