Major reliability improvement for logging stack traces by borrrden · Pull Request #287 · couchbase/fleece

borrrden · 2026-03-20T23:10:03Z

This took many days of effort and so it's involved a lot with various stack unwinding and address retrieval mechanisms so I'll try to provide a high level summary here. Firstly to avoid a giant spaghetti of ifdef I broke out the implementation across several files that only get compiled on certain platforms:

Backtrace+capture-darwin.cc: Apple specific logic for getting a symbol from a frame in a stack trace. This uses standard backtrace_symbols and dladdr since the implementation on Apple is much more enhanced than the glibc linux counterpart (Most likely the compiler is more friendly about it when it emits machine code).
Backtrace+capture-linux.cc: Linux specific logic for getting a symbol from a frame in a stack trace. This makes use of the GCC provided libbacktrace.a (it's present even when using clang because by default clang still uses GCC for its runtime stuff in c++). This library is very attuned to specifically getting frames on Linux even when frame pointers are omitted and handles all the gnarly unwinding stuff safely (I found when trying to implement register based stack crawling that I often got it wrong and caused an SIGSEGV while trying to do it)
Backtrace+capture-posix.cc: Other stuff common to all POSIX environments like unmangling and actual printing. In this we need to be very careful since my plan is to handle signals here as well. We can only use async signal safe functions. In particular this means no streams, no printf, and no unmangling either.
Backtrace+capture-win32.cc: Windows specific logic for getting a symbol from a frame in a stack trace. Printing is not restricted since the way that Windows faults already moves to an entirely separate context in which it is safe to do just about anything. This uses the StackWalk64 API.
Backtrace+signals-posix.cc: POSIX signal handlers! RAII style class that will register signal handlers and write out a stack trace, and then pass on control to the next handler.
Backtrace+signals-win32.cc: Win32 exception handlers! Windows doesn't really have the equivalent of POSIX signals. It has a few basic ones but that's at. Most faults result in one of the various types of exception handlers (this is an overloaded term, since Windows was using the term "exception" way before it had any meaning for c++. These "exceptions" can happen even in C. They are basically signals by another name except the way the handlers are invoked is more safe).
Backtrace.cc: This now encapsulates any logic that operates one level above all of this logically. It uses the primitives defines in the files above.

Note that since we cannot use ostreams in signal handlers, I created a new crash log in the logs directory that a crash will be written to upon a signal. The existing logs are all buried inside of ofstream objects.

1. set_unexpected is supposed to be removed in c++17 2. Need to unwind through the signal trampoline 3. iOS will not allow signal handling in app store, probably

No std, no snprintf, almost nothing is allowed inside a signal handler

It now gets caught in the abort handler

borrrden · 2026-03-20T23:17:46Z

Fleece/Support/Backtrace+signals-posix.cc

+
+        [[noreturn]] static void sig_handler(int signo, siginfo_t* info, void* context) {
+            const char* name = strsignal(signo);
+            violation_type() = name ? name : "Signal " + to_string(signo);


Ugh, how did this get back in here.....this is not going to be allowed! I need to change this to use write_to_and_stderr

borrrden · 2026-03-20T23:18:34Z

Fleece/Support/Backtrace+signals-posix.cc

+
+            crash_handler_immediate(info, context);
+
+            signal(signo, SIG_DFL);


Need to consider saving the original handler. It might not be SIG_DFL!

snej · 2026-03-23T18:09:41Z

Fleece/Support/Backtrace+capture-darwin.cc

+#include "betterassert.hh"
+
+namespace fleece {
+    Backtrace::frameInfo getFrame(const void* addr, bool stack_top) {


I'd rather this were a private static member of Backtrace than a standalone function.

snej · 2026-03-23T18:13:36Z

Fleece/Support/Backtrace+capture-posix.cc

+            "_C_A_T_C_H____T_E_S_T_",
+            "Catch::TestInvokerAsFunction::invoke() const",


Have you checked that these names are still valid after we upgraded Catch2?

I have not. Is there a method you used to determine them in the first place?

snej · 2026-03-23T18:15:53Z

Fleece/Support/Backtrace+capture-posix.cc

+        }
+    }
+
+    char* unmangle(const char* function) {


Likewise, could this be a private static member of Backtrace?

This was originally not even before my PR. Making it a private static member requires bringing the other Unmangle methods along with it. Is that the outcome you want?

snej · 2026-03-23T18:22:16Z

Fleece/Support/Backtrace.hh

+                                                                void* context = nullptr);
+
+        /// Captures a backtrace from the given starting point and returns a shared pointer to the instance.
+        [[nodiscard]] static std::shared_ptr<Backtrace> capture(void* from, unsigned maxFrames = 50,


What is from, here? I'm assuming a stack frame or PC address? This seems like too low-level a method to make public.

snej · 2026-03-23T18:22:38Z

Fleece/Support/Backtrace.hh

+        [[nodiscard]] static std::shared_ptr<Backtrace> capture(void* from, unsigned maxFrames = 50,
+                                                                void* context = nullptr);
+
+        static int raw_capture(void** buffer, int max, void* context = nullptr);


Likewise, is there a need to make this public?

It needs to be callable from the signal handler, as well as from Backtrace itself.

Fleece/Support/Backtrace.hh

borrrden added 11 commits March 6, 2026 12:20

Improve stack trace and implement crash handling

b43069a

Improvements on backtrace and POSIX signal handling

f4a9415

A few Windows fixes

e2e0044

macOS specific fixes

700e955

1. set_unexpected is supposed to be removed in c++17 2. Need to unwind through the signal trampoline 3. iOS will not allow signal handling in app store, probably

Rewrite the entire output process to be signal safe

908e0bf

No std, no snprintf, almost nothing is allowed inside a signal handler

Remove fragile code regarding ucontext

404fd86

Window enhancements and remove std::terminate handler

57b812e

It now gets caught in the abort handler

Code refactor to make more readable

abdba6b

Various fixups

5684a27

Bouncing between machines for checking that I didn't forget stuff

23703f1

Suppress compiler warning

fbe300e

borrrden requested review from jianminzhao and snej March 20, 2026 23:10

borrrden commented Mar 20, 2026

View reviewed changes

snej requested changes Mar 23, 2026

View reviewed changes

borrrden added 2 commits March 24, 2026 07:47

PR review feedback (mine and others) and formatting

f404937

Remove unused function

c68f30f

borrrden requested a review from snej March 28, 2026 00:07

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Major reliability improvement for logging stack traces#287

Major reliability improvement for logging stack traces#287
borrrden wants to merge 13 commits intomasterfrom
crash_handling

borrrden commented Mar 20, 2026 •

edited

Loading

Uh oh!

borrrden Mar 20, 2026

Uh oh!

borrrden Mar 20, 2026 •

edited

Loading

Uh oh!

snej Mar 23, 2026

Uh oh!

snej Mar 23, 2026

Uh oh!

borrrden Mar 23, 2026

Uh oh!

snej Mar 23, 2026

Uh oh!

borrrden Mar 23, 2026

Uh oh!

snej Mar 23, 2026

Uh oh!

snej Mar 23, 2026

Uh oh!

borrrden Mar 23, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		crash_handler_immediate(info, context);

		signal(signo, SIG_DFL);

		"_C_A_T_C_H____T_E_S_T_",
		"Catch::TestInvokerAsFunction::invoke() const",

Conversation

borrrden commented Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

borrrden Mar 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

borrrden commented Mar 20, 2026 •

edited

Loading

borrrden Mar 20, 2026 •

edited

Loading