Community Expert Article #2: LLM APIs as Core Product Features

The Community Expert program is a new program we’re launching to gather and share insight from real world practitioners in software and AI. We will periodically send these out as they come in. If you would like to contribute, please apply here.

This article is by Ashay Satav, Director of Product Management at eBay

Introduction

In the past year, large language models (LLMs) have quickly evolved from a novelty to a core part of product strategy for many companies. Product managers across industries are discovering that LLM APIs – from providers like OpenAI (ChatGPT/GPT-4), Anthropic (Claude), Google (PaLM 2 and the upcoming Gemini), Cohere, and even open-source models like Llama 2 or Mistral – can be integrated directly into their product stacks to unlock new capabilities. What began as simple chatbot add-ons is now becoming deeply embedded functionality that differentiates products, improves user experience, and even drives new revenue streams.

This article provides an overview of how LLM APIs are woven into products, with concrete examples in SaaS, e-commerce, and customer service. 

LLMs in the Modern Product Stack

Adding AI to an application not long ago meant long development cycles and training custom models. However, companies can plug ready-made LLMs into their products via APIs. This API-driven approach allows teams to experiment and deploy AI features much faster. For example, OpenAI's GPT-4 and GPT-3.5 models (via the ChatGPT API) sparked an explosion of integrations in 2023, and competing APIs from Anthropic, Cohere, and others soon followed. Even Google is set to offer its Gemini model to developers, underlining that all major tech players are making LLM capabilities accessible.

How LLM Integration Creates Business Value

Key benefits that integrating LLMs as core product features can bring to a business:

• Improved User Experience and Engagement: LLM features can make products more intuitive, interactive, and helpful. Features like Notion's AI writing assistant or Snapchat's My AI add real utility, keeping users engaged longer. When users can ask the product for what they need (whether it's a piece of information, a recommendation, or content), it creates a frictionless experience. This often leads to higher usage frequency and user satisfaction. For example, Snapchat+ subscribers used the My AI chatbot to send 20 billion messages monthly​.

• Differentiation in a Competitive Market: AI capabilities are helping products stand out from their competitors. Instacart differentiates itself from other grocery apps by offering an intelligent AI shopping assistant. Shopify gives merchants tools that marketplaces like Amazon don't provide natively (such as AI-generated content with minimal effort). In B2B SaaS, having an AI "copilot" can be a unique selling proposition when courting customers. 

• New Revenue Streams or Pricing Models: Some companies directly monetize their LLM features. A great example is Snapchat – they included generative AI features (such as My AI chatbot and AI image generation) in their $3.99/month Snapchat+ subscription, which helped propel that subscription to 7 million paying users by the end of 2023. Enterprise software companies are also charging more for AI. Notion initially offered its AI features as an extra $10 per user add-on, later including them in higher-tier plans. 

• Efficiency and Cost Savings (Internal and External): Internally, companies use LLMs to automate tasks that would otherwise incur high labor costs. The most transparent case is customer support automation. Efficiency gains can improve margins or allow staff to be reallocated to higher-value activities. Product managers should consider these ROI angles: sometimes, an LLM feature can justify itself through cost savings, even if it isn't monetized directly.

• Unlocking New Use Cases: LLMs can enable functionalities previously not feasible in a product. For example, Duolingo (the language learning app) integrated GPT-4 to create Duolingo Max, which offers AI conversation practice and on-demand explanations of answers. Teaching casual learners via free-form conversation was a new use case unlocked by LLMs' ability to generate endless interactive dialogues.

Product Strategy for PMs Integrating LLMs

Building with LLMs is a beast different from traditional feature development. Product managers need to think through how to integrate AI thoughtfully and sustainably. Here are some key considerations and best practices emerging as PMs gain experience with LLM features:

• Use Case Selection & Feature Prioritization: Start with clear problems that an LLM uniquely suits to solve. Not every feature needs AI; you should identify where generative or conversational ability will significantly improve the user experience or unlock value. Good candidates are often tasks that involve unstructured language data, such as answering questions, writing content, parsing requests, or complex decision trees that can frustrate users.

• LLM Vendor Choice & Model Strategy: Product managers have a menu of AI model providers and types. Choosing (and possibly continually re-evaluating) the right model is crucial. OpenAI's GPT-4 is often the gold standard for quality and knowledge, but it comes at a higher cost and sometimes higher latency. Cheaper or faster models, such as OpenAI's GPT-3.5 or Cohere's command models, suffice for less critical tasks. Anthropic's Claude offers a 100k-token context window, which can be advantageous for applications that need to digest a lot of data (Claude-powered agents can intake entire knowledge bases or long transcripts). Some companies adopt a multi-model strategy, e.g., using GPT-4 for complex queries but falling back to a smaller model for simple ones to save cost.

• Infrastructure & Data Management: Incorporating LLMs is not just a UI feature build – it often requires supporting infrastructure. One common approach is Retrieval-Augmented Generation (RAG), which pairs the LLM with a vector database of relevant data. This helps reduce hallucinations and makes responses more accurate. Product managers should ensure their team has a plan for how the AI will access up-to-date information via RAG, fine-tuning the model on their data, or other prompt engineering techniques. 

• Cost Management and ROI: LLM API calls have a cost per use, which can add up quickly at scale. Part of product planning for AI features is figuring out the business model behind them. Monitoring usage analytics is key: track how often users invoke the AI feature, how it correlates with outcomes (e.g., support resolution, time saved, purchases made), and calculate a rough return on investment (ROI).

• Quality, Testing, and Trust: Ensuring the AI's outputs are accurate and appropriate is one of the most complex yet critical parts of deploying LLM features. Hallucinations (confident but incorrect answers) can erode user trust or even cause harm if the domain is sensitive. Product managers should implement rigorous testing regimes for AI responses. This might include manual reviews of outputs, user feedback loops (e.g., thumbs up or down on AI answers), and automated checks for specific keywords or patterns. Many products use disclaimers (e.g., "AI-generated, may be incorrect") or UI cues to set expectations. Transparency helps maintain trust.

• Privacy and Ethical Considerations: When integrating third-party AI APIs, you often send user data, such as queries and content, to an external service. PMs must consider user privacy and compliance. Can you opt out of specific sensitive data from being sent? Do you need user consent, especially in regulated industries or regions like the EU's GDPR? Many LLM providers offer enterprise options where they don't train on your data or allow deployment in a private cloud. 

The Road Ahead: Opportunities /Challenges

Although the integration of LLM APIs into products is still in its early days, the landscape is evolving quite rapidly, and PMs should prepare for this trend to make their products create more value for customers. Here are some forward-looking insights:

• AI Everywhere – User Expectations Evolving: We are likely moving towards a world where almost every digital product has some form of intelligent assistant or AI-powered functionality. As mobile apps became a standard for businesses a decade ago, AI capabilities may become a baseline expectation. Users might soon expect to ask any app for help in plain language or have content auto-generated wherever they create something. PMs should prepare for a future where "AI inside" is assumed and focus on unique value or proprietary data they can combine with AI to stand out.

• Advancements in Models (and New Players): The pace of LLM improvement is blistering. OpenAI is continuously refining its models (GPT-4 was a leap, and future versions or specialized variants will follow). Google's Gemini would be multimodal (processing text, images, etc., together) and aimed to outperform GPT-4. We also see startups releasing competitive open models (e.g., Claude 2 from Anthropic or Mistral's following models) that could close the gap with the big names. For product strategy, this means more options and possibly lower costs over time. 

• Deeper Personalization and Contextual Understanding: In the future, LLM integrations will likely become more personalized by leveraging user-specific data. There is a balance with privacy here, but technically, this is feasible through fine-tuning or using extended context. We could also see products chaining multiple specialized models: a large general model plus a smaller domain-specific model working together. This could improve reliability and enable more complex, multi-step operations using AI.

• Reliance on Third Parties – Risk Mitigation: Relying on an external AI service means ceding some control. There's also the risk of policy changes; for instance, an AI provider might change what content is filtered or how pricing works. Over time, if an AI feature is mission-critical, some organizations will gravitate towards self-hosted models despite the overhead, just for guaranteed uptime and control. Another risk is quality drift: if the model update suddenly behaves differently, it could introduce bugs into your user experience. Hence, monitoring and testing are continuous requirements. As a PM, it's wise to have a Plan B for critical AI dependencies and to keep an open dialogue with your AI vendors about their roadmap and support.

• Regulation and Trust Factors: Society and governments are increasingly scrutinizing the use of AI. We might see regulations surrounding transparency (e.g., requiring the disclosure of AI-generated content), data usage, or liability for AI errors. For example, the EU is discussing an "AI Act" that could impose obligations on companies using AI in customer-facing ways. Product managers should monitor compliance needs, especially in sensitive domains like finance, healthcare, or education. Being proactive about ethical AI practices can also become a competitive advantage – demonstrating that your AI feature is safe and unbiased and respects user data can help build trust and a strong brand reputation.