Debug Decision Tree

Advanced13 min read

Assumes: All previous modules

TL;DR

Debug automotive SoCs layer by layer: power/hardware (is the board alive?) → boot mode (which mode am I in?) → interface (what can I see?) → OS identification (which OS has the problem?) → tool selection (which tool for this OS?) → root cause classification (hardware, BSP, kernel, driver, app, config). This systematic approach eliminates guesswork and works for any failure — from a bricked board to a display flicker.

Every page in this curriculum taught you a piece of the puzzle: bench connectors, boot modes, UART wiring, per-OS tools. This page is the assembly diagram. When something breaks on an SA8155P/SA8295P cockpit platform running QNX + AAOS under QHEE, you do not start by guessing. You climb the debug layer cake from the bottom, prove each layer works, and only then reach for the tool that matches the OS owning the symptom. “Have you tried turning it off and on again?” works 30% of the time. The other 70% needs a real methodology.

Memorization trick: P-B-I-O-T-R — Power, Boot mode, Interface, OS, Tool, Root cause. Say it out loud at the start of every session.

The Debug Layer Cake

Six horizontal layers. Always start at Layer 1 and work up. Never jump to Layer 5 (tools) without confirming Layers 1–4.

flowchart BT
    subgraph L6 ["Layer 6 — Root Cause Classification"]
        RC["Hardware fault · BSP config · Kernel bug<br/>Driver issue · App bug · Config error"]
    end

    subgraph L5 ["Layer 5 — Tool Layer"]
        TL["slog2info · logcat · QXDM · ftrace · KGDB<br/>(Page 10.4)"]
    end

    subgraph L4 ["Layer 4 — OS Layer"]
        OS["QNX · AAOS · AMSS identification<br/>(Module 6, Module 7)"]
    end

    subgraph L3 ["Layer 3 — Interface Layer"]
        IF["UART · JTAG · USB · Ethernet<br/>(Page 10.3)"]
    end

    subgraph L2 ["Layer 2 — Boot Mode Layer"]
        BM["EDL · fastboot · recovery · normal · crashdump<br/>(Page 10.2)"]
    end

    subgraph L1 ["Layer 1 — Hardware Layer"]
        HW["Power · connectors · bench setup<br/>(Page 10.1)"]
    end

    L1 --> L2 --> L3 --> L4 --> L5 --> L6

    NOTE["Always start at Layer 1 and work UP.<br/>Never jump to Layer 5 without confirming Layers 1–4."]
    L1 -.-> NOTE

Layer	Question You Must Answer	If This Layer Fails	Primary Reference
1 — Hardware	Is the board alive? Power rails OK? Cables connected?	No UART, no USB, no LEDs	Page 10.1 — Debug Bench
2 — Boot Mode	Which rung of the mode ladder am I on?	Wrong tool for the mode (ADB in EDL)	Page 10.2 — Boot Modes
3 — Interface	What visibility do I have right now?	Cannot reach the OS that owns the bug	Page 10.3 — Debug Interfaces
4 — OS	Which OS owns this symptom?	Debugging AAOS when QNX is broken	Module 6 — Three OSes
5 — Tool	Which log or debugger for this OS?	KGDB when logcat would answer in 30 seconds	Page 10.4 — Software Debug Toolkit
6 — Root Cause	What category of fix is needed?	Escalating to wrong team, wrong fix path	This page — taxonomy below

The 5-Minute Triage Flowchart

This is the artifact you will use most. Print it. Tape it to your bench. When a colleague says “the board is broken,” start here.

flowchart TD
    START["System is not working as expected"]
    S1["Step 1: Check UART — any output?"]

    NO_UART["No output at all"]
    PWR["Check power supply, UART cable, baud rate<br/>Try different UART port"]
    DEAD["Still nothing: board may be dead or in EDL<br/>Try force-EDL via test point (Page 10.2)"]

    YES_UART["Yes — output visible"]
    S2["Step 2: Identify boot stage from log prefix"]

    PBL["PBL messages — then stops"]
    PBL_FIX["Stuck in PBL. Likely XBL partition corrupt<br/>Enter EDL, reflash XBL via QFIL (Page 8.2)"]

    XBL["XBL / fastboot messages"]
    XBL_FIX["Stuck in XBL/fastboot<br/>Check boot.img, vendor_boot, DTB<br/>Use fastboot to reflash (Page 8.3)"]

    QNX["QNX startup messages"]
    QNX_FIX["QNX is booting<br/>Check startup/driver init<br/>Use slog2info, pidin (Page 10.4)"]

    LINUX["Linux kernel messages [0.000000]"]
    LINUX_FIX["Linux kernel loading<br/>Check panic, module failures<br/>Use dmesg, logcat (Page 10.4)"]

    ANDROID["Android framework messages"]
    ANDROID_FIX["Android running<br/>Issue is app/framework level<br/>Use logcat, dumpsys (Page 10.4)"]

    S3["Step 3: Is the issue reproducible?"]

    ALWAYS["Always reproducible"]
    ALWAYS_FIX["Systematic — check config, DT, build<br/>Likely BSP or driver issue"]

    INTERMITT["Intermittent"]
    INTERMITT_FIX["Race condition or hardware marginal<br/>Use ftrace, tracelogger, stress testing"]

    OTA["Only after OTA"]
    OTA_FIX["Partition mismatch<br/>Check A/B slot state<br/>boot.img vs vendor_boot version match"]

    START --> S1
    S1 -->|No| NO_UART --> PWR --> DEAD
    S1 -->|Yes| YES_UART --> S2

    S2 --> PBL --> PBL_FIX
    S2 --> XBL --> XBL_FIX
    S2 --> QNX --> QNX_FIX
    S2 --> LINUX --> LINUX_FIX
    S2 --> ANDROID --> ANDROID_FIX

    PBL_FIX --> S3
    XBL_FIX --> S3
    QNX_FIX --> S3
    LINUX_FIX --> S3
    ANDROID_FIX --> S3

    S3 --> ALWAYS --> ALWAYS_FIX
    S3 --> INTERMITT --> INTERMITT_FIX
    S3 --> OTA --> OTA_FIX

Boot Log Prefix Quick Reference

Log Signature	Boot Stage	You Are Here	First Action
PBL / PBL-FW banner, then silence	PBL ROM	Below XBL — chain cannot continue	Force EDL → QFIL reflash XBL (Page 8.2)
XBL / UEFI / Fastboot mode	XBL / ABL	Bootloader stage — no HLOS	fastboot getvar all, reflash boot/vendor_boot (Page 8.3)
QNX / startup-* / procnto	QNX IFS load	Hypervisor host booting	slog2info, UART from beginning
[ 0.000000] Linux timestamp	AAOS guest kernel	Linux loading inside vCPU	dmesg, check VirtIO probe
boot_progress / ActivityManager	Android framework	AAOS fully up	logcat, dumpsys for owning service

Critical habit: Read the UART log from the beginning, not from the error line. The context before the failure often reveals the root cause. A kernel panic at line 4000 means nothing if you missed the driver probe failure at line 200.

Think About It

A field return shows ‘intermittent black screen after 2 hours of driving.’ Using the P-B-I-O-T-R framework, which layer do you investigate first?

Reveal Answer

Layer 1 (Power/Hardware) — P-B-I-O-T-R always starts at the bottom, and a 2-hour soak failure often indicates thermal throttling, PMIC margin, or display power rail droop under load. Reproduce on the bench with a thermal camera and scope VDD_CX during extended run before assuming software. Once Layer 1 checks out, climb to Layer 4 (OS): cluster black screen points to QNX display RM or AAOS SurfaceFlinger depending on which physical display failed.

Symptom-to-Layer Map

Bookmark this table. Find your symptom, confirm the layer, run the first tool, know what to look for.

Symptom	Likely Layer	Which OS	First Tool	What to Look For	Reference Page
Board completely dead (no UART, no USB)	Hardware	N/A	Multimeter, check power rails	Power supply, PMIC, loose connector	10.1
UART shows PBL then stops	Boot Mode	N/A	QFIL	XBL partition corrupt, enter EDL	8.2, 10.2
Stuck in fastboot	Boot Mode	N/A	fastboot CLI	boot.img or DTB corrupt	8.3, 10.2
QNX boots but AAOS doesn’t start	OS	QNX + AAOS	slog2info + UART	Hypervisor VM config, AAOS image load	5.3, 6.2
Black screen, touch works	OS/Driver	AAOS	dumpsys SurfaceFlinger	Display driver, framebuffer, HWC layers	6.4, 5.6
No ADB connection	Interface	AAOS	Check USB port routing	Wrong USB port, adbd not running, USB mux	10.1, 8.3
Audio from wrong speaker	App/Config	AAOS	dumpsys audio	Zone routing, audio HAL config	6.4
Random reboot	Kernel	AAOS or QNX	dmesg / core dump / minidump	Kernel panic, watchdog timeout	6.6, 10.4
OTA fails at 0%	Config/Network	Multiple	lb status + SOME/IP trace	Service discovery, gRPC connection	8.6
Camera preview frozen	Driver	AAOS	logcat + dumpsys media.camera	ISP driver, CSI lane config	5.6
Vehicle speed shows 0	Cross-OS	QNX → AAOS	VHAL logs + CAN trace	VHAL property not propagating	6.3
Modem no network	AMSS	Modem	QXDM	RF calibration, NAS registration	7.1

Root Cause Classification Flow

Once you have identified the layer and OS, classify the root cause before choosing a fix path. Different classifications mean different owners and different escalation paths.

flowchart LR
    SYM["Symptom confirmed<br/>at correct layer"]

    HW["Hardware fault<br/>Bad solder, ESD, power marginal"]
    BSP["BSP config<br/>DT wrong, defconfig, pin mux"]
    KERN["Kernel bug<br/>Panic, NULL deref, deadlock"]
    DRV["Driver bug<br/>Timeout, register seq, race"]
    FW["Framework bug<br/>Service crash, wrong API"]
    APP["App bug<br/>Logic error, resource leak"]
    CFG["Config error<br/>Wrong build, OTA mismatch, slot"]

    SYM --> HW
    SYM --> BSP
    SYM --> KERN
    SYM --> DRV
    SYM --> FW
    SYM --> APP
    SYM --> CFG

    HW --> HW_FIX["Hardware team · RMA · bench swap"]
    BSP --> BSP_FIX["BSP team · DT overlay · defconfig"]
    KERN --> KERN_FIX["Kernel team · minidump · KGDB"]
    DRV --> DRV_FIX["Driver owner · register trace · ftrace"]
    FW --> FW_FIX["Platform team · dumpsys · service logs"]
    APP --> APP_FIX["App team · logcat · repro steps"]
    CFG --> CFG_FIX["Release team · slot check · reflash"]

Common Debug Patterns

These patterns recur across programs. Recognizing the pattern saves hours of random log grepping.

“Works on bench A, fails on bench B”

Root cause template: BSP or board variant difference.

Check	Why
Device tree overlay	Different carrier board revision, pin mux, display panel
BSP version	Bench A on debug BSP, bench B on production
Board revision	PMIC variant, fuse state, connector population
Fuse state	Debug policy, JTAG availability, EDL signing

Reference: Page 5.6 — BSP, Page 10.1 — Debug Bench

“Works after fresh flash, fails after OTA”

Root cause template: Partition mismatch across A/B slots.

Check	Why
A/B slot versions	boot_a updated but vendor_boot_b stale
boot.img vs vendor_boot	Kernel and vendor ramdisk must match
Super partition merge state	Dynamic partition resize incomplete
OTA payload vs running slot	Update applied to inactive slot, switch failed

Reference: Page 8.1 — Partition Map, Page 8.6 — OTA In Practice

“Works in debug BSP, fails in production BSP”

Root cause template: Debug feature dependency stripped in production.

Check	Why
CONFIG_* flags	ftrace, KGDB, verbose slog disabled
JTAG/UART policy	Production DT disables debug nodes
Symbol stripping	Stack traces useless without symbols
Assert behavior	Debug builds halt; production builds continue into corruption

Reference: Page 6.6 — Kernel Internals, Page 8.4 — Secure Boot

“Intermittent crash, no pattern”

Root cause template: Race condition or hardware marginality.

Tool	Purpose
ftrace function_graph	See call timing and unexpected delays
QNX tracelogger	Kernel event timeline for System Profiler
Increased log verbosity	Catch the rare path before crash
Stress test suspect paths	Thermal, load, rapid connect/disconnect

Reference: Page 10.4 — Software Debug Toolkit

“Worked yesterday, broke today”

Root cause template: Build change.

Check	Why
git log / build manifest	What changed in the last 24 hours
Build diff	New vendor module, HAL version bump
Dependency version	gRPC proto change, SOME/IP service ID shift
Config overlay	Someone flashed wrong variant

Reference: Page 8.3 — Fastboot & ADB, Page 3.4 — gRPC

Mental Model

Debug like a mechanic, not a magician. Mechanics don’t guess — they follow a diagnostic tree. Check fuel, check spark, check air, check timing. Our tree: check power, check boot, check interface, check OS, check app. The tree never skips a step.

P-B-I-O-T-R: Physical → Boot → Init → OS → Transport → Result. Debug top-down until the layer breaks.

Common Gotcha

Beginner thinks they can skip straight to logcat when something breaks. Reality: the most expensive debug mistake is assuming which layer the bug is in. “It must be a software bug” when the power rail is drooping. “It must be hardware” when the device tree has the wrong pin config. Always start at Layer 1 and prove each layer works before moving up.

The 60-second pre-debug ritual

Before every debug session:

1. Read the bug report / symptom description — note reproducibility, timing, and which build.
2. Identify the suspected layer from the symptom table above.
3. Connect UART + start logging — tee to timestamped file.
4. Reproduce — one clean reproduction with full logs beats ten partial attempts.
5. Read the UART log from the beginning — don’t search for the error first. The context before the error is often more important than the error itself.

Deep Dive

Root cause classification taxonomy — each category has a different fix path and a different person to escalate to.

Classification	Examples	Fix Path	Escalate To
Hardware fault	Bad solder, ESD damage, power rail marginal, failed PMIC	Swap board, rework, scope rails	Hardware / EE team
BSP config	DT wrong panel node, missing status = "okay", pin mux error, wrong clock	DT overlay, defconfig patch, reflash	BSP / platform team
Kernel bug	Panic, NULL deref, deadlock, use-after-free	Minidump analysis, KGDB, kernel patch	Kernel / OS team
Driver bug	Probe timeout, wrong register sequence, IRQ storm, DMA misconfig	Driver logs, ftrace, register dump	Driver owner
Framework bug	System service crash, wrong binder API, Car Service state error	dumpsys, service-specific logs	Android platform team
App bug	Logic error, resource leak, wrong permission	logcat, repro APK, systrace	App developer
Config error	Wrong build flashed, OTA slot mismatch, wrong super merge	Reflash matched set, slot switch, manifest audit	Release / integration team

The taxonomy matters because misclassification wastes everyone’s time. A “random reboot” classified as an app bug when it is a kernel watchdog timeout sends the wrong team on a three-day chase. Confirm layer and OS first, then classify.

Connect the Dots

This is the hub page — the knowledge graph center of the curriculum.

• Module 0 (Hardware Foundations) — Layer 1: power, memory, boot media
• Module 5 (Qualcomm Cockpit SoC) — Layers 2–3: boot chain, hypervisor, BSP, inter-VM comms
• Module 6 (Three Operating Systems) — Layer 4: QNX, AAOS, VHAL, display/audio, kernel internals
• Module 7 (Connectivity & Telematics) — AMSS/modem layer for telematics and RF issues
• Module 8 (Build, Flash & Update) — Recovery paths: QFIL, fastboot, OTA, secure boot
• Module 9 (Safety, Security & Diagnostics) — Production diagnostics (UDS/DoIP) when bench debug is not available
• Module 10 — 10.1 Debug Bench, 10.2 Boot Modes, 10.3 Debug Interfaces, 10.4 Software Debug Toolkit, 10.5 Board Bring-Up, 10.6 Debug Decision Tree (this page)

When you finish this module, you have a complete debug workflow from dead board to classified root cause.

What to Remember

1. P-B-I-O-T-R: Power → Boot mode → Interface → OS → Tool → Root cause. Never skip a layer.
2. UART first, always. If UART is silent, fix Layer 1 before opening logcat.
3. Read boot logs from the top. The failure context precedes the error message.
4. Match tool to OS: slog2 (QNX), logcat (AAOS), QXDM (modem).
5. Recognize patterns: bench variant, OTA mismatch, debug-vs-production BSP, build change.
6. This page is your cheat sheet. Symptom table + triage flowchart + layer cake = systematic debug.

Check Your Understanding

1. What does the P-B-I-O-T-R debug mnemonic stand for?

Recall

A. PBL, Boot.img, Init, Overlay, Test, ReleaseB. PMIC, BSP, IPC, OTA, Telemetry, ResetC. Power, Boot mode, Interface, OS, Tool, Root causeD. Partition, Bootloader, Interface, OS, Trace, Recovery

2. No UART output after power-on. You confirm 12V input and steady-state rails with a multimeter. Per the 5-minute triage flowchart, what is the immediate next step?

Recall

A. Parse a SOME/IP Wireshark capture to look for service discovery failuresB. Check UART cable, port, baud (115200), TX/RX; scope VDD_CX if wiring is OKC. Run QXDM on the diag port to check if the modem subsystem is respondingD. Open logcat — AAOS may be running silently

3. AAOS boots to the homescreen but the camera app shows a black preview. Touch and other apps work fine. Where do you start on the layer cake, and which tool comes first?

Applied

A. Layer 5 — QXDM for modem RF interferenceB. Layer 4 (OS) AAOS up, then Layer 5 logcat + dumpsys media.camera for HAL/ISPC. Layer 1 — check power rails with a multimeterD. Layer 2 — force EDL and reflash via QFIL

4. The board worked yesterday. A new build was flashed today; now it is stuck at the splash screen. Which debug pattern applies, and which tools should you use first?

Applied

A. Build change — read UART, find boot stage, slog2info (QNX) or dmesg (Linux)B. Partition mismatch after OTA — check A/B slot state and vbmeta integrityC. Intermittent race condition — use ftrace and tracelogger immediately to captureD. Hardware fault — RMA the board to the vendor for failure analysis and rework

5. Vehicle speed shows 0 km/h in AAOS but CAN bus traces show correct speed signals. Which layers are involved, and which cross-OS tools do you use?

Synthesis

Connects to: Module 6 — VHAL

A. Layer 2 boot mode — reflash boot.img via fastboot to fix a kernel-level issueB. Layer 4 cross-OS: slog2info + dumpsys car_service + CAN trace to find the breakC. Layer 1 hardware — replace the vehicle speed sensor on the drivetrain moduleD. Layer 5 AAOS only — logcat ActivityManager for the speedometer app lifecycle

6. UART shows PBL banner then silence. USB enumerates as QDLoader 9008 after a failed OTA. A colleague suggests `fastboot flash boot`. What is the correct decision-tree path?

Synthesis

Connects to: Module 8 — Build, Flash & Update

A. Layer 5 — run logcat for OTA agent errorsB. Layer 2 — board is in EDL, not fastboot; QFIL + Firehose to recover partitionsC. Layer 1 — replace the PMIC before attempting any software flash or debug stepD. Layer 4 — QNX slog2info for hypervisor failure

Module Review: Debug Engineering

Question 1 of 12

Board boots QNX (slog2 shows startup) but AAOS never starts — stuck at Android splash. UART shows Linux kernel loading then panic. You have adb unavailable. Next tools in order?

🔗 Connects to: Module 6 — Kernel Internals

A. Reflash QNX IFSB. slog2info for QNX errorsC. QXDM for modem debugD. L2: boot mode → L3: UART dmesg panic → L4: Linux/AAOS → L5: panic backtrace

Further Reading

• Linux Kernel Debugging Guide — Kernel.org systematic debugging index covering oops analysis, tracing, and KGDB — the Layer 5–6 escalation path for AAOS kernel issues in the decision tree.
• Android Debugging Documentation — Google’s debug workflow hub for logcat, bugreport, systrace, and dumpsys — the AAOS tool layer referenced in the triage flowchart.
• QNX Neutrino System Architecture — QNX fault containment and microkernel model context for classifying resource manager crashes vs full system failures in the root cause taxonomy.
• NASA Fault Tree Analysis Handbook — Structured fault-tree methodology for decomposing complex failures into provable sub-causes, analogous to the layer cake and triage flowchart on this page.
• Android Bootloader Architecture — Boot stage reference for interpreting UART log prefixes (PBL → XBL → ABL → HLOS) in the 5-minute triage flowchart.
• Android Verified Boot — AVB and rollback index concepts behind the “build change + AVB verify failed” pattern in the common debug patterns section.
• Qualcomm Developer Network — Cross-cutting platform resources for QFIL recovery, crashdump collection, and modem QXDM when the decision tree routes to a Qualcomm-specific tool.
• Vector — Root Cause Analysis in Automotive Development — Automotive industry RCA methodology for classifying hardware, BSP, driver, and configuration faults before escalating to the wrong team.