Kforce AI chatbot empowers team performance with dynamic sales insights

Given the client’s initial uncertainty about the possible functionalities of the large language model (LLM) chatbot, our KCS Data and Analytics Practice Leader provided initial recommendations and conducted a workshop with client stakeholders. The goal of this workshop was to better understand the types of data features and inferences that would be most beneficial in the initial deployment.

After identifying the needs and desires of end-users and stakeholders and considering the client’s institutional rules around LLM API use and development, Kforce developed a plan that outlined the creation of a chatbot MVP for testing by client users for later iteration.

• USERS ISSUE A QUERY THROUGH AN API TO THE CHATBOT SERVICE
• The chatbot is only accessible through existing client company internal portals
• This both ensures access security and allows user credentials to be easily verified and passed via API to the chatbot service environment
• User credentials are used to ensure only the proper users get access to any given data through the client’s data entitlement layer
• THE CHATBOT SERVICE CATEGORIZES THE QUESTIONS INTO SEVERAL TOPICS WHICH PERTAIN TO DIFFERENT DATA TOPICS AND/OR QUERY TEMPLATES
• Classification was initially done using the LLM through prompting and examples
• Once more sample questions were amassed in logs, a simple classification model was trained from Google t5-small
• PROMPTING SEQUENCES INSTRUCT THE LLM TO CREATE OR USE SQL QUERIES BASED ON THE CLASSIFICATION OF THE QUERY/TOPIC
• Simple chain-based prompting strategy
• Used for both ease of logging and because agent-based methods (e.g., LangChain) were incompatible with the rate limits imposed upon the API institutionally by the client
• Simple to set up guardrails around improper API uses (e.g., asking about unrelated topics)
• Different query/topic classifications lead to different prompt sequences within the chatbot architecture
• For some simple and common queries, SQL queries are stored to retrieve data without LLM input
• For most queries and topics, classification allows for targeted prompting and passing only essential table metadata relevant to the topic(s) involved in the user’s query. This both improves performance (in accuracy and time) and reduces costs from excess tokens sent to the LLM
• SQL QUERIES RETRIEVE RELEVANT DATA FROM THE SNOWFLAKE DATA WAREHOUSE
• All data passes through the client’s entitlement layer
• This ensures all data access protocols are followed and users only see data they are entitled to access
• RETRIEVED DATA IS PASSED BACK TO THE LLM FOR FORMATTING AND INFERENCE
• Formatting/inference instructions are passed to the LLM alongside data according to the topic classification of the initial query
• RESPONSES ARE PASSED BACK TO THE CHATBOT PROCESS AND SENT ALONG TO THE USER’S UI APPLICATION
• Responses are, depending on the query and user preference settings, displayed in bullet points, text and/or table(s)

• Trained logistic regression models in the Snowflake environment to create columns for some inference topics
• Customer retention risk
• Cross-selling/up-selling opportunity prediction
• Creation of logging for user and session-based analysis of chatbot performance
• Logging also allows for history context implementations in future iterations
• User feedback features implemented and incorporated into session and user logging for process analysis and improvement
• Classification process and modular prompting strategy
• Provides future proofing against changes in data warehousing or additions/changes to desired question scope

We built a custom-designed team from our portfolio of proven technologists to address the customer's specific needs and integrate successfully within their environment. The initial team composition for product development included:

• Lead Data Scientist
  This role served as the primary communication and planning touchpoint interacting with client teams and resources, determining and enacting a prompting strategy in consultation with Kforce Consulting Solutions Practice Leadership and client teams.
• Data Scientist
  This role was responsible for creating the code and prompting framework for the prompting strategy and conducting feature testing for chatbot features.
• AI/ML Engineer
  This role worked with the client’s data engineering team to validate data going into the AI environment, maintain and manage CI/CD pipelines for development and ensure security measures at all stages of development.

With efficient technical discovery and immersion, actual work on the project began quickly after the initial scoping workshops. The custom nature of this team enabled them to quickly deliver outcomes. Delivery Performance developed a Playbook to introduce and acclimate the team with the project scope and team structure before the formal workshops kicked off. Team leadership worked in tandem with the client supervisor to enable access to tools, applications and documentations that assisted the team ramp up and deep dive into the existing infrastructure.

The primary technical requirement was to leverage the client’s API to GPT 3.5 Turbo (licensed through Microsoft/OpenAI) alongside existing data warehousing of sales data in Snowflake. This setup allowed for on-demand data retrieval and basic inference for Sales Leadership.

Other technical requirements and limitations that shaped the needs of the project included:

• No retrieval augmented generation (RAG) as the client’s internal AI rules and administration had not yet approved the use of the Microsoft Azure GPT API for use in RAG or RAG-assisted applications.
• All data must be accessed through existing data entitlement processes.
• Tokenization costs of using the LLM API meant tokens sent to the LLM should be minimized to reduce production and testing costs.
• Query rates to the LLM API were limited on a per-team basis.

For the purposes of development and initial MVP deployment, a cloud based VM was created to host the chatbot code, which runs as Python Flask applications within containers on the VM. The chatbot service is accessible as a service through a secure API through which queries and metadata (username, session ID if applicable, etc.) are passed to the chatbot system.

The chatbot service ensures that the LLM can retrieve the right data to answer user questions and provide summary and/or inference while abiding by technical and institutional constraints. The KCS team created a methodology that ensured sufficient flexibility to answer free-form questions while also limiting computational costs and ensuring adherence to security measures.

Kforce’s data and analytics experts partnered with the client to develop a generative AI framework using the client’s internal, licensed instance of OpenAI’s GPT 4 and their internal sales and customer data to serve as a live chatbot.

The final product was delivered on schedule for MVP testing internally by the client with all agreed upon features succeeding all tests the Kforce team and initial users could devise. The product also included several more features than what was initially planned (for example, adding more than present quarter data to allow for some time series analysis of sales data).

The chatbot was used to answer questions and provide data summaries pertaining to team and regional sales performance, customer/opportunity status and other necessary queries. Sales leadership is now able to make queries pertaining to any data present in the customer or sales team databases and is also provided sample prompts for quick access to answers to frequently asked questions.

CS_126: Modernizing legal operations with scalable agentic AI workflows

LP_034: Preparing your organization for generative AI

CS_049: Energy industry: Using AI to enhance operational outcomes