When AI Fails: Troubleshooting

• Recognize common AI failure patterns
• Know the difference between AI limitations and prompt problems
• Have a toolkit of troubleshooting strategies
• Know when to give up and try a different approach
• Stay calm when AI is not working (it happens to everyone)

AI will fail you. It will misunderstand your question. It will make things up. It will refuse to help with something reasonable. It will give you confidently wrong answers.

This is not a bug. It is the nature of the technology. Understanding how AI fails helps you work around it effectively.

This lesson gives you a troubleshooting toolkit. When AI stops working, you will know what to try.

When AI gives a bad response, the problem is usually one of two things:

1. Your prompt (fixable with better wording)
2. AI's limitations (requires working around)

Before troubleshooting, figure out which one you are dealing with.

Signs of a Prompt Problem

• AI understood the topic but focused on the wrong aspect
• The response is generic and could apply to many situations
• AI asked clarifying questions (it knew it needed more info)
• A small wording change would probably fix it

Signs of an AI Limitation

• AI says it cannot help or refuses
• AI confidently states something you know is wrong
• AI seems confused or produces nonsense
• Multiple attempts with different prompts get the same bad result
• The task involves recent events, math, or real-time data

Ambiguity

Symptom: AI answers a different question than you meant.

Example: "What is the best way to learn Python?"

AI might not know if you mean: Python the programming language? Python for data science? For a beginner? For someone switching from another language?

Fix: Add context that removes ambiguity. "I am a marketing professional who knows Excel well but has never coded. What is the best way to learn Python specifically for automating data tasks? I have about 5 hours per week."

Missing Context

Symptom: AI gives generic advice that does not fit your situation.

Example: "How should I respond to this email?"

AI has no idea what email you mean, who sent it, what your relationship is, or what outcome you want.

Fix: Provide the actual context. "A client I have worked with for 2 years just sent this email: [paste email]. They sound frustrated about a delay that was our fault. How should I respond to acknowledge the problem without being overly apologetic? I want to keep the relationship strong."

Wrong Framing

Symptom: AI goes in a completely different direction than you expected.

Example: "Tell me about Apple's new features."

AI might describe fruit nutrition, the company's hardware, software features, or their services. The word "features" is too vague.

Fix: Reframe with specifics. "List the three most important new features in iOS 19 for someone deciding whether to update their iPhone."

Hallucination (Making Things Up)

What happens: AI confidently states false information as if it were true.

Common triggers:

• Obscure topics with limited training data
• Specific details like names, dates, statistics
• Questions about the AI's own capabilities

Workarounds:

1. Ask AI to qualify its confidence: "Rate your confidence in that answer on a scale of 1-10 and explain why."
2. Ask for sources (but verify them yourself): "What sources would I check to verify this information?"
3. Cross-reference with other sources before trusting factual claims.

Knowledge Cutoff

What happens: AI does not know about recent events.

Example: If you ask about something that happened after the AI's training data ended, it will not know about it (or worse, will make something up).

Workarounds:

1. Ask when the knowledge ends: "What is your knowledge cutoff date?"
2. Provide the recent information yourself: "Based on this recent news article I am pasting below, analyze the implications for [topic]. Here is the article: [paste]"
3. Use AI tools with web search capability for current information.

Math and Counting

What happens: AI makes calculation errors or miscounts items.

Example: Ask AI to count letters in a word or do multi-step math, and it often fails.

Workarounds:

1. For critical math, use a calculator or spreadsheet instead
2. Ask AI to show its work step by step
3. Verify any numbers before using them

Refusals

What happens: AI refuses to help with a request, even if the request seems reasonable to you.

Common causes:

• AI interprets the request as potentially harmful
• The topic touches on sensitive areas
• The request is ambiguous and could be interpreted badly

Workarounds:

1. Clarify your intent: "I understand you have limitations. Can you explain what about my request is problematic? Maybe I can rephrase."
2. Add legitimate context: "I am a nurse studying medication interactions for patient safety. Can you explain how [medication A] and [medication B] interact?"
3. Break the request into smaller, clearly benign parts.

Sometimes a conversation gets stuck. AI might be:

• Confused by contradictory context
• Stuck on a wrong interpretation
• Producing increasingly unhelpful responses

When this happens: start a new conversation.

Do not just repeat your original prompt. Rethink it:

• What did AI misunderstand?
• What context was missing?
• Is there a simpler way to frame this?

Reset pattern: "Let's start over with a fresh approach. Forget what we discussed. New question: [completely reframed request]"

When AI is not the right tool:

AI is powerful but not universal. Knowing its limits helps you use it appropriately.

• Diagnose first: Is this a prompt problem or an AI limitation?
• Prompt problems are fixable: Add context, remove ambiguity, reframe
• AI limitations have workarounds: Hallucination (verify), cutoff (provide info), math (double-check), refusals (clarify intent)
• Reset when stuck: New conversation with rethought approach
• Know when AI is not the answer: Some tasks need other tools or human expertise

Intentionally break AI to see how it fails:

1. Test hallucination: Ask about a very obscure topic you know well. Does AI make things up? "Tell me about [obscure person, place, or thing you know about]. Be specific."
2. Test counting: Try this classic: "How many r's are in the word strawberry?"
3. Test knowledge cutoff: "What happened in the news yesterday?"
4. Now practice recovery. For each failure:

• Identify the failure type
• Try a workaround from this lesson
• Note what worked

This builds your troubleshooting instincts.

• Research on LLM failure modes: https://arxiv.org/abs/2302.04023
• Anthropic model card on Claude limitations: https://www.anthropic.com/research/claude-character
• OpenAI on model limitations: https://platform.openai.com/docs/guides/prompt-engineering/strategy-verify-outputs