When a client asked whether we should throw an LLM at a simple FAQ bot, I stopped the meeting after ten minutes. The project budget couldn't cover the API cost, the latency was already a deal-breaker, and the knowledge base was just a few hundred static answers. In the end we built a keyword-matcher with a tiny embedding index and saved weeks of integration work.

When an LLM is the right tool

LLMs shine when you need:

  • Open-ended generation, creative copy, code snippets, or summarisation where the output space is huge.
  • Few-shot learning, the ability to adapt to a new task with only a handful of examples.
  • Understanding nuanced language, sentiment analysis on slang, multi-turn dialogue, or ambiguous queries.

If your problem meets any of those, an LLM can reduce development time dramatically. The trade-off is cost (both compute and API fees) and less deterministic behaviour.

When a smaller model or rule-based approach wins

I keep a checklist for every new AI request:

  1. Is the output bounded? If you only need a classification among ten categories, a fine-tuned BERT-size model or even a classic logistic regression will be faster and cheaper.
  2. Do you have enough labelled data? Training a small model on a few thousand rows often outperforms prompting a massive LLM that hasn’t seen your domain.
  3. Latency matters? Real-time UI interactions (e.g., autocomplete) benefit from on-device inference; a 200 ms LLM call feels sluggish.
  4. Regulatory constraints? Storing user data in a third-party LLM service can breach GDPR or HIPAA. A self-hosted model gives you full control.
  5. Explainability is required? Rule-based systems or shallow models let you trace why a decision was made, something LLMs hide behind billions of parameters.

When the answer to any of those is yes, start with the lightweight option.

A practical workflow to decide

  1. Define the success metric, accuracy, latency, cost, or compliance.
  2. Prototype with a simple model, use sklearn or tensorflow to get a baseline.
  3. Benchmark, measure the metric against a small LLM call (e.g., OpenAI gpt-3.5-turbo).
  4. Iterate, if the baseline fails, consider a medium-sized transformer (DistilBERT, MiniLM) before jumping to the biggest model.

This step-by-step approach keeps you from over-engineering and lets you justify the expense of an LLM with data.

Common pitfalls when you jump straight to an LLM

  • Prompt drift, changing the prompt slightly can cause wildly different outputs; version control becomes a nightmare.
  • Hidden costs, token-based pricing adds up quickly, especially for long context windows.
  • Hallucinations, the model may fabricate facts that look plausible, breaking trust.
  • Rate limits, production traffic can hit API caps, forcing you to implement complex retry logic.

I learned these the hard way when a client’s marketing copy generator started inserting brand-inconsistent slogans. The fix was to add a post-processing filter, but it would have been cheaper to start with a template-driven system.

Code snippet: simple keyword matcher with embeddings 

from sentence_transformers import SentenceTransformer, util
import numpy as np

# Load a tiny embedding model (≈ 50 MB)
model = SentenceTransformer('all-MiniLM-L6-v2')

# FAQ pairs
questions = [
    "How do I reset my password?",
    "What is your refund policy?",
    "Can I change my subscription plan?"
]
answers = [
    "Click ‘Forgot password’ on the login page and follow the email link.",
    "Refunds are processed within 7 business days after request.",
    "Yes, go to Account → Plan and select a new tier."
]

# Encode once
embeddings = model.encode(questions, normalize_embeddings=True)

def get_answer(user_query):
    q_emb = model.encode([user_query], normalize_embeddings=True)
    # cosine similarity
    scores = util.cos_sim(q_emb, embeddings)[0]
    best_idx = int(np.argmax(scores))
    if scores[best_idx] > 0.75:
        return answers[best_idx]
    return "I’m not sure, please contact support."
  • SentenceTransformer loads a model that fits on a cheap VPS.
  • embeddings are pre-computed for the static FAQ list.
  • cos_sim gives a fast similarity score; a threshold of 0.75 filters out unrelated queries.

This approach handled a 5 k-question knowledge base with sub-100 ms latency on a single CPU core, far cheaper than any LLM call.

FAQ

When is it acceptable to use a free LLM API?

If the use-case is low-volume, non-commercial, and you can tolerate occasional latency spikes, a free tier can be a quick proof-of-concept. Move to a paid plan or a self-hosted model once usage scales.

How do I know if my model is hallucinating?

Run a set of known-answer queries and compare the output. If the model generates information not present in the source data, flag it and add a verification step (e.g., a retrieval-augmented pipeline).

Can I fine-tune a small model to replace a large one?

Yes. Fine-tuning a 80 M parameter model on your domain data often yields performance comparable to prompting a 175 B model, with a fraction of the cost and full control over the weights.

Wrapping up

The first question you should ask isn’t “Which LLM should I buy?” but “Do I really need an LLM at all?” A quick prototype with a lightweight model, a clear metric, and a cost check will usually answer that. When the problem truly needs open-ended generation or few-shot learning, then the investment in a large model makes sense. Otherwise, keep it simple, stay in control, and you’ll avoid a lot of hidden headaches.