register_wide_hw_breakpoint continually triggers handler callback

Question

In the Linux kernel, when a breakpoint I register with register_wide_hw_breakpoint is triggered, the callback handler endlessly runs until the breakpoint is unregistered.

Background: To test a driver for some hardware we are making, I am writing a second kernel module that emulates the hardware interface. My intent is to set a watchpoint on a memory location that in the hardware would be a control register, so that writing to this 'register' can trigger an operation by the emulator driver.

See here for a complete sample. I set the breakpoint as follows:

hw_breakpoint_init(&attr);
attr.bp_addr = kallsyms_lookup_name("test_value");
attr.bp_len = HW_BREAKPOINT_LEN_4;
attr.bp_type = HW_BREAKPOINT_W;
test_hbp = register_wide_hw_breakpoint(&attr, test_hbp_handler, NULL);

but when test_value is written to, the callback (test_hbp_handler) is triggered continually without control ever returning to the code that was about to write to test_value.

1) What should I be doing differently for this to work as expected (return execution to code that triggered breakpoint)?

2) How do I capture the value that was being written to the memory location?

In case this matters: $ uname -a Linux socfpga-cyclone5 3.10.37-ltsi-rt37-05714-ge4ee387 #1 SMP PREEMPT RT Mon Jan 5 17:51:35 UTC 2015 armv7l GNU/Linux

Answer 1

This is by design. When an ARM hardware watchpoint is hit, it generates a Data Abort exception. On ARM, Data Abort exceptions trigger before the instruction that triggers them finishes¹. This means that, in the exception handler, registers and memory locations affected by the instruction still hold their old values (or, in some cases, undefined values). As such, when the handler finishes, it must retry the aborted instruction so that the interrupted program runs as intended². If the watchpoint is still set when the handler returns, the instruction will trigger it again. This causes the loop you're seeing.

To get around this, userspace debuggers like GDB single-step over any instruction that hits a watchpoint with that watchpoint disabled before resuming execution. The underlying kernel API, however, just exposes the hardware watchpoint behavior directly. Since you're using the kernel API, it's up to your event handler to ensure that the watchpoint doesn't trigger on the retried instruction.

[The ARM watchpoint code in the kernel actually does support automatic single-step, but only under very specific conditions. Namely, it requires 1) that no event handler is registered to the watchpoint, 2) that the watchpoint is in userspace, and 3) that the watchpoint is not associated with a particular CPU. Since your use case violates at least (1) and (2), you have to find another solution.]³

Unfortunately, on ARM, there's no foolproof way to keep the watchpoint enabled without causing a loop. The breakpoint mode that GDB uses to single-step programs, "instruction mismatch," generates UNPREDICTABLE behaviour when used in kernel mode⁴. The best you can do is disable the watchpoint in your handler and then set a standard breakpoint to re-enable it on an instruction that you know will execute soon after.

For your MMIO emulation driver, watchpoints are probably not the answer. In addition to the issues just mentioned, most ARM cores have very few watchpoint registers, so the solution would not scale. I'm afraid I'm not familiar enough with ARM's memory model to suggest an alternative approach. However, Linux's existing code for emulating memory-mapped IO for virtual machines might be a good place to start.

---

¹There are two types of Data Abort exceptions, synchronous and asynchronous, and it's left to the implementation to decide which one a watchpoint generates. I'm describing the behavior of synchronous exceptions in this answer, because that's what would cause the problem you're having.

²ARMv7-A/R Architecture Reference Manual, B1.9.8, "Data Abort exception."

³Linux Kernel v4.6, arch/arm/kernel/hw_breakpoint.c, lines 634-661.

⁴ARMv7-A/R Architecture Reference Manual, C3.3.3, "UNPREDICTABLE cases when Monitor debug-mode is selected."