Software Debug Toolkit

Advanced12 min read

Assumes: 6.1 QNX, 6.2 AAOS, 10.1 Debug Bench

TL;DR

Each OS on the cockpit SoC has its own debug toolkit. For AAOS: adb logcat, dumpsys, KGDB, ftrace, Qualcomm minidump/RAMParser. For QNX: slog2info, dumper, pidin, GDB via qconn, Momentics IDE. For modem (AMSS): QXDM, diag port, modem crash dumps. Knowing which tool to reach for — and on which OS — eliminates “where do I even start” paralysis.

Hardware interfaces (Page 10.3) get you visibility below the OS. This page covers the software tools that run once HLOS is up — or immediately after a crash. On SA8155P/SA8295P platforms running QNX + AAOS under QHEE, you will switch between three toolchains in a single debug session. The skill is not memorizing every flag; it is knowing which floor of the building owns the broken component and reaching for that OS’s primary log first. Sometimes reaching for a hardware debugger is like bringing a forklift to move a potted plant.

Memorization trick: AAOS = ADB first. QNX = Slog2 first. Modem = QXDM first. Always start with the primary logging tool for the OS that owns the symptom.

Think About It

The cluster shows a black screen but Android logs show no errors. Which OS should you investigate first?

Reveal Answer

QNX — the cluster display pipeline runs through the QNX display resource manager, so clean logcat means the IVI stack is fine and the break is on the QNX side. Start with slog2info | grep -i display and pidin on the display RM process to check driver init and framebuffer allocation. Only cross-check AAOS with dumpsys SurfaceFlinger if QNX confirms the cluster path is exporting frames correctly.

AAOS Debug Stack

AAOS runs Linux inside a guest VM. Debug tools mirror standard Android development, plus automotive-specific services and Qualcomm crash collection. Think in layers — highest level first, drill down only when higher layers look clean.

flowchart TB
    subgraph L1 ["Layer 1 — App / Framework"]
        LC["adb logcat<br/>ActivityManager · WindowManager<br/>SurfaceFlinger · AudioFlinger · update_engine"]
    end
    subgraph L2 ["Layer 2 — System Services"]
        DS["dumpsys<br/>activity · window · SurfaceFlinger<br/>audio · car_service"]
    end
    subgraph L3 ["Layer 3 — Kernel Log"]
        DM["dmesg / kmsg<br/>kernel ring buffer"]
    end
    subgraph L4 ["Layer 4 — Kernel Tracing"]
        FT["ftrace<br/>function · event · MMIO tracers"]
    end
    subgraph L5 ["Layer 5 — Kernel Debugger"]
        KG["KGDB<br/>live kernel debug over UART"]
    end
    subgraph L6 ["Layer 6 — Post-Crash"]
        MD["Qualcomm minidump / ramdump<br/>RAMParser offline analysis"]
    end

    L1 --> L2 --> L3 --> L4 --> L5 --> L6

Layer 1: App and Framework — adb logcat

Your first stop for AAOS issues. Filter by tag and priority to cut noise.

# Errors only across all buffers — start here
adb logcat -b all *:E

# Live tail with timestamps, filter SurfaceFlinger
adb logcat -v threadtime SurfaceFlinger:V *:S

# OTA / update_engine progress
adb logcat -s update_engine

# Capture to file while reproducing
adb logcat -b all -v threadtime > aaos_$(date +%Y%m%d_%H%M%S).log

Tag	What It Shows
ActivityManager	App lifecycle, ANRs, process death
WindowManager	Display layers, focus, rotation
SurfaceFlinger	Compositor, vsync, buffer dequeue failures
AudioFlinger	Audio track creation, routing, underruns
update_engine	A/B OTA download, verify, apply status

Layer 2: System Services — dumpsys

Snapshot of live service state. Faster than parsing log streams for “what is the system doing right now?”

adb shell dumpsys activity top          # foreground activity stack
adb shell dumpsys window displays       # display topology, layers
adb shell dumpsys SurfaceFlinger        # compositor state, HWC layers
adb shell dumpsys audio                 # streams, devices, volume, routing
adb shell dumpsys car_service           # VHAL connection, properties, zones
adb shell dumpsys media.audio_flinger   # active tracks per zone

Automotive-specific: dumpsys car_service exposes occupant zone routing, VHAL property cache, and UX restriction state — essential for “wrong zone” bugs.

Layer 3: Kernel Log — dmesg / kmsg

Kernel ring buffer. Driver init failures, panics, and VirtIO errors appear here before they surface in logcat.

adb shell dmesg | tail -100
adb shell dmesg | grep -i error
adb shell dmesg | grep -iE "panic|oops|virtio|display|audio"
adb shell cat /proc/kmsg                 # live kernel stream (root)

Layer 4: Kernel Tracing — ftrace

Dynamic kernel instrumentation via debugfs. Requires CONFIG_DYNAMIC_FTRACE and debug BSP (Page 6.6).

adb root
adb shell "echo 0 > /sys/kernel/debug/tracing/tracing_on"
adb shell "echo function > /sys/kernel/debug/tracing/current_tracer"
adb shell "echo 1 > /sys/kernel/debug/tracing/tracing_on"
# ... reproduce issue ...
adb shell "echo 0 > /sys/kernel/debug/tracing/tracing_on"
adb shell cat /sys/kernel/debug/tracing/trace > ftrace_capture.log

Tracer	Use Case
function / function_graph	Call flow, latency hotspots
nop + event filters	Scheduler, IRQ, block I/O events
MMIO trace (platform-specific)	Register access during driver bring-up

Layer 5: Kernel Debugger — KGDB

Live kernel debugging over serial. Boot with KGDB wait so the kernel pauses before init.

# Append to kernel cmdline (boot.img or fastboot oem config)
kgdboc=ttyMSM0,115200n8 kgdbwait nokaslr

# Host — connect GDB to target over UART
aarch64-linux-android-gdb vmlinux
(gdb) set serial baud 115200
(gdb) target remote /dev/ttyUSB0
(gdb) break do_sys_open
(gdb) continue

Set breakpoints, inspect variables, single-step kernel code. Requires matching vmlinux with debug symbols. Not available on production-fused boards without debug policy.

Layer 6: Post-Crash — Qualcomm Minidump and Ramdump

When the kernel panics, UART may truncate the oops. Qualcomm crash collection dumps memory over USB.

# Enable minidump mode (kernel cmdline or SCM register)
qcom_scm.download_mode=mini

# Trigger dump collection from running system (requires policy)
adb reboot dumpmode

Dump Type	Size	Contents	Analysis Tool
Minidump	50–200 MB	Vendor-registered regions: kernel log, task structs, driver state	RAMParser, QCAP
Full ramdump	4–8 GB	Complete DRAM image	RAMParser, vendor tools

Minidump is faster to collect and sufficient for most kernel panics. Full dump is needed when corruption spans unregistered memory regions.

QNX Debug Stack

QNX owns hardware that AAOS accesses through VirtIO and resource managers. When UART shows interleaved QNX + Linux output, learn to read QNX lines (no kernel timestamp prefix) and reach for QNX-native tools.

flowchart TB
    subgraph Q1 ["Layer 1 — System Log"]
        SL["slog2info<br/>live tail · buffer filter"]
    end
    subgraph Q2 ["Layer 2 — Process Info"]
        PI["pidin<br/>arg · mem · thread · fd"]
    end
    subgraph Q3 ["Layer 3 — Core Dumps"]
        DU["dumper daemon<br/>/var/dumps/ on crash"]
    end
    subgraph Q4 ["Layer 4 — Kernel Trace"]
        TL["tracelogger<br/>→ System Profiler in Momentics"]
    end
    subgraph Q5 ["Layer 5 — Remote Debug"]
        QC["qconn<br/>GDB / Momentics over IP"]
    end
    subgraph Q6 ["Layer 6 — Startup Debug"]
        SU["startup-* -v · procnto -v<br/>boot-time hardware init"]
    end

    Q1 --> Q2 --> Q3 --> Q4 --> Q5 --> Q6

Layer 1: System Log — slog2info

QNX equivalent of scanning system-wide logs. The closest analog to Linux dmesg for day-to-day work.

# Live tail all buffers
slog2info -w

# Filter by buffer name (display, audio, network RM)
slog2info -w | grep -i display
slog2info -b qnx_slogger2 -w

# Via ADB bridge when QNX console is exposed
adb shell slog2info -w

Layer 2: Process Info — pidin

Process and thread inspection. First tool when a QNX service hangs or leaks memory.

pidin                    # process list with PID, state, CPU
pidin arg                # command-line arguments
pidin mem                # memory maps and usage
pidin thread             # thread list per process
pidin fd                 # open file descriptors
pidin -p <pid> thread    # threads for one process

Layer 3: Core Dumps — dumper

Daemon that captures core files when a process crashes. Default location: /var/dumps/.

# Verify dumper is running
pidin | grep dumper

# List recent dumps
ls -lt /var/dumps/

# Configure dump directory (buildfile or runtime)
# dumper -d /var/dumps -N 10    # keep last 10 dumps

Load core files into Momentics or gdb with matching debug symbols for stack trace.

Layer 4: Kernel Trace — tracelogger

Captures kernel events: scheduling, interrupts, IPC, syscalls. Analyze in QNX System Profiler (Momentics IDE).

# Capture 10 seconds of kernel trace
tracelogger -n 10 -s 1000 -f /tmp/trace.kev

# Copy to host and open in System Profiler
scp root@<target>:/tmp/trace.kev .

Layer 5: Remote Debug — qconn

Debug agent for remote GDB and Momentics IDE attachment over IP or serial.

# Start qconn on target (often in buildfile)
qconn port=8000

# Host GDB
ntoaarch64-gdb ./my_app
(gdb) target qnx <target-ip>:8000
(gdb) attach <pid>

Momentics IDE: Run → Attach to Process via qconn for breakpoints, variable inspect, and source-level stepping.

Layer 6: Startup Debug — Verbose Boot

When boot hangs before slog2 is available, UART is your only tool. Enable verbose flags on startup and the microkernel.

# In QNX buildfile or startup script
startup-sa8155 -v
procnto-smp-instr -v

Look for driver probe failures, resource manager registration errors, and IFS mount issues in the UART stream.

Modem Debug (AMSS)

The modem is a separate domain — not visible in adb shell ps or QNX pidin. Debug through the diag port and QMI, not by poking modem registers from the AP.

QXDM — Qualcomm eXtensible Diagnostic Monitor

Real-time modem log capture on the host PC. Shows protocol stack messages, call flows, NAS/RRC state machines, and RF measurements.

QXDM View	What It Shows
Message View	Raw diag messages filtered by log code
Item View	Parsed fields (RSRP, RSRQ, cell ID, registration state)
Graph View	RF metrics over time

Typical workflow: connect diag port → start log capture → reproduce signal/registration issue → save .qmdl log for offline analysis.

Diag Port Routing

The modem diag interface routes through the SoC USB composite device:

Modem (AMSS) → SMD/Diag router → USB composite function → Host PC (QXDM)

On automotive ADP boards, the diag interface is typically a separate USB composite function from ADB. Check lsusb on the host — look for Qualcomm diag enumeration (often alongside QDLoader, ADB, and RNDIS). Some platforms route diag through a dedicated USB port or require an OEM-specific composite configuration. If QXDM shows no messages, verify the correct USB port and that the modem firmware loaded successfully (SSR logs on AP side).

Modem Crash Dumps

AMSS/MPSS crash generates a subsystem dump — separate from AP kernel minidump.

Collection Path	When
SSR (Subsystem Restart) logs	Modem auto-restarts; AP-side logs capture restart reason
Dedicated dump partition	Full modem crash dump stored on UFS for offline parse
QXDM live capture	Last messages before crash visible in .qmdl

Parse offline with Qualcomm tools (QCAT, QXDM log parser). Cross-reference Page 7.1 (AMSS) for QMI architecture and Page 7.3 for modem firmware update paths.

Cross-OS Debug Scenarios

This table is the most valuable artifact on this page. When a symptom appears, find the row, run the tool, know what to look for. Do not start with KGDB when logcat would answer the question in thirty seconds.

Symptom	OS to Check	Primary Tool	What to Look For
Display doesn’t turn on	QNX (display routing) + AAOS (SurfaceFlinger)	slog2info (QNX display RM), dumpsys SurfaceFlinger	Display driver init, framebuffer allocation, HWC layer count, vsync state
Display freezes but touch works	AAOS (compositor)	dumpsys SurfaceFlinger, adb logcat SurfaceFlinger:V	Buffer dequeue stall, HWC composition failure, GPU hang; touch OK means QNX input path is alive
Audio plays from wrong zone	AAOS (AudioFlinger + car_service)	dumpsys audio, dumpsys car_service	Zone routing config, occupant zone mapping, audio HAL device selection
No audio at all	QNX (audio RM) + AAOS (AudioFlinger)	slog2info (audio buffer), dumpsys audio	QNX audio resource manager init, VirtIO sound device, AAOS audio policy
Touch input unresponsive	QNX (input driver)	pidin, slog2info (HID/input RM)	Input resource manager registration, event injection, I2C/SPI touch controller probe
Network unreachable	QNX (io-pkt) or AAOS (netd)	ifconfig / netstat -ni (QNX), adb shell ip addr (AAOS)	Ethernet/Wi-Fi driver init, IP config, DHCP, VirtIO net backend
Modem no signal	AMSS	QXDM	RF metrics (RSRP/RSRQ), NAS registration state, SIM status, band selection
Telematics OTA stuck	QNX (OTA agent) + AMSS	slog2info (gRPC/SOME/IP agent), QXDM (data call)	OTA agent state machine, modem data session, QMI WDS connection
Boot hangs at splash	QNX (early boot)	UART log	startup-* verbose output, driver init failures, IFS load errors
AAOS never starts, QNX OK	AAOS (guest boot)	UART + dmesg (if partial boot)	VirtIO probe, guest kernel panic, ABL load failure for boot.img
Random crash / reboot	Kernel (AAOS or QNX)	dmesg + minidump / core dump	Panic string, stack trace, watchdog timeout, SSR reason code
VHAL property stale	QNX (VHAL server) + AAOS (Car Service)	slog2info (VHAL), dumpsys car_service	vsock/gRPC connection, property update rate, CAN signal source
Camera preview black	QNX (camera RM) + AAOS (Camera HAL)	slog2info (camera), adb logcat Camera*:V	CSID/ISP init, buffer sharing across VM boundary, HAL open failure
USB gadget not enumerating	QNX (USB stack) + boot mode	UART, pidin (usb stack)	USB role switch, gadget config, boot mode (Page 10.2)
High CPU / thermal throttle	AAOS + QNX	adb shell top, pidin -F "%a %b %c %p"	Runaway process, interrupt storm, trace with ftrace/tracelogger
Inter-VM communication failure	QNX (VirtIO backend) + AAOS (VirtIO frontend)	dmesg (virtio), slog2info (vsock)	vsock connection refused, gRPC deadline exceeded, backend not ready

Mental Model

Each OS is a floor in a building. AAOS is the penthouse — fancy, user-facing, lots of windows (logcat). QNX is the basement — critical infrastructure, utility access (pidin, slog2). The modem is the communications room — specialized equipment (QXDM), restricted access. When something breaks, go to the floor where the broken thing lives.

AAOS: logcat → dumpsys → bugreport → tombstone. Layer by layer, surface to core.

Where You'll See This

The universal debug start sequence: 1) Open UART terminal. 2) adb logcat -b all *:E (errors only across all buffers). 3) adb shell slog2info -w (if QNX console is available via ADB bridge). 4) Reproduce the issue. 5) Capture logs. This 30-second setup catches 70% of issues.

Common Gotcha

If you enable function_graph ftrace on a production kernel, watchdog timeouts can crash the system. ftrace is powerful but dangerous on production builds — the overhead is enough to miss real-time deadlines. Always use ftrace on debug BSP builds only, and disable it before shipping.

Deep Dive

Qualcomm minidump vs full ramdump: Minidump collects only vendor-registered memory regions — kernel log buffer, task structs, specific driver state — typically 50–200 MB vs 4–8 GB for a full dump. Minidump is faster to collect (seconds vs minutes over USB) and sufficient for most kernel panics where the oops message and stack trace are in registered regions. Full dump is needed when the crash involves memory corruption outside registered regions — heap overwrites, DMA buffer corruption, or bugs in unregistered drivers. Your BSP team registers regions at build time; if a new driver is not registered, its state will be absent from minidump.

Connect the Dots

Page 6.1 (QNX) and Page 6.2 (AAOS) describe the OS environments these tools run in. Page 6.6 (kernel internals) covers the CONFIG flags that enable ftrace, KGDB, and debug symbols. Page 7.1 (AMSS) provides QMI and modem architecture context for QXDM sessions. Page 10.2 (boot modes) lists which tools are available in normal HLOS vs crashdump vs EDL. Page 10.6 (debug decision tree) routes symptoms to the tools on this page.

What to Remember

1. Three OSes, three primary tools: logcat (AAOS), slog2info (QNX), QXDM (modem).
2. Drill down in layers: logcat before dumpsys, dumpsys before dmesg, dmesg before ftrace, ftrace before KGDB.
3. Cross-OS bugs need cross-OS logs: capture UART + logcat + slog2info in the same reproduction window.
4. Minidump for speed, ramdump for depth: most panics resolve with minidump + RAMParser.
5. Match tool to boot mode: crashdump tools only work when dump policy is enabled (Page 10.2).

Check Your Understanding

1. What is the QNX equivalent of Linux's dmesg?

Recall

A. slog2infoB. dumperC. traceloggerD. pidin mem

2. What kernel boot parameter enables minidump collection on crash?

Recall

A. qcom_scm.download_mode=miniB. androidboot.veritymode=disabledC. nokaslrD. kgdboc=ttyMSM0,115200n8

3. Display freezes but touch input works. Which OS and tool should you check first?

Applied

A. AAOS — adb reboot dumpmode for minidumpB. QNX — pidin fd for file descriptor leaksC. AAOS — dumpsys SurfaceFlingerD. AMSS — QXDM for RF interference

4. Telematics reports no cellular data. AAOS UI and QNX cluster are normal. `adb logcat` shows no modem errors. What tool and subsystem should you use next?

Applied

A. KGDB over UART — live kernel debugger for stepping through Linux codeB. QXDM on modem diag — capture NAS/RRC state and RF metrics (RSRP/RSRQ)C. slog2info — QNX display resource manager crash dump and error messagesD. fastboot flash modem — reflash MPSS firmware image without any diagnosis

5. A QNX resource manager crashes repeatedly. `/var/dumps/` has fresh core files. What is the correct analysis path?

Applied

A. Reflash via QFIL immediately — core dumps always indicate UFS corruptionB. Parse cores with Momentics/gdb + matching symbols; confirm dumper via pidinC. Use QXDM Item View for stack traces from the modem subsystem crash logsD. Run `adb bugreport` — it captures QNX core dumps automatically

6. An AAOS native service crashes with no logcat stack trace. Tombstones exist in `/data/tombstones/`. Which cross-domain workflow gives the fastest root-cause path?

Synthesis

Connects to: Module 6 — Three Operating Systems

A. Enable KGDB on every boot permanentlyB. Pull tombstone + symbols, correlate logcat -b crash, cross-check QNX slog2C. QXDM Message View — modem SSR logsD. Force SDI via JTAG on a production-fused vehicle unit for live CPU halt

Further Reading

• GDB Documentation — Official GNU Debugger manual covering remote attach, breakpoints, and symbol loading for KGDB and QNX qconn sessions described on this page.
• Android logcat — Google’s logcat reference for buffer selection, priority filters, and live capture — the Layer 1 AAOS tool in the debug stack.
• Android dumpsys — Service snapshot commands for ActivityManager, SurfaceFlinger, audio, and car_service state inspection at Layer 2.
• Linux Kernel ftrace Documentation — Kernel tracing guide for function, function_graph, and event tracers used in AAOS kernel debug on debug BSP builds.
• Linux Kernel KGDB and KDB — Kernel debugger setup over serial, matching the kgdboc/kgdbwait boot parameters and GDB remote workflow in the AAOS stack.
• QNX Neutrino Utilities Reference — QNX documentation hub for slog2info, pidin, dumper, and tracelogger covered in the QNX debug stack.
• QNX Momentics IDE — IDE for attaching to processes via qconn, analyzing core dumps, and opening tracelogger captures in System Profiler.
• Qualcomm Developer Network — Portal for QXDM, minidump/RAMParser tooling, and modem diag resources referenced in the AMSS debug section.