HotSpot stack guard pages. Red/Yellow zones and reasons behind them

Question

Java Threads stack organization is described by the following diagram in comments. So the 1 glibc guard page seems to be the default value which is set by pthread_attr_init(pthread_attr*).

The reason I could imagine behind red/yellow zones is handling stack-overflow without receiving SIGSEGV. Also red-zone here definitely means something different from what is specified in System V AMD64 ABI which is set to 128 byte below rsp and is meant to stay untouched by interrupt/signal handlers.

The HotSpot red zone is controlled by -XX:StackRedPages option and as far as I tried it could be 0 to 3 pages in size.

What I also found was that when primordial thread creates a thread which performs the initialization steps it sets the pthread guard size to 0.

Q: What are red/yellow zones are for and why the thread performing the initialization is created without pthread guard pages?

Answer 1

On Linux one (probably of some others) usage for red and yellow_reserved zones is to make JVM HotSpot handles stack overflow. pthreads_attr_setguardsize man page has a note about setting guard size to 0:

it is the application's responsibility to handle stack overflow (perhaps by using mprotect(2) to manually define a guard area at the end of the stack that it has allocated).

and

Setting a guard size of 0 may be useful to save memory in an application that creates many threads and knows that stack overflow can never occur.

So since no threads (even a primordial one) use MAP_GROWSDOWN mapping for their stack setting the guard size to 0 at this point can probably be treated as an optimization. The "stackoverflow never occur" is guaranteed by the reserved/yellow/red pages that JVM manually maps in the function void JavaThread::create_stack_guard_pages().

As can be seen, some part at the bottom of thread's stack even with no pthread's guard pages is treated as HotSpot Guard Pages that is guarded with mprotect right as the manual page told us to do.

The actual handling of stack overflow is done via signal handler that is installed on JVM startup. The part related to stack overflow is this (a bit long):

// Handle ALL stack overflow variations here
if (sig == SIGSEGV) {
  address addr = (address) info->si_addr;
  // check if fault address is within thread stack
  if (thread->on_local_stack(addr)) {
    // stack overflow
    if (thread->in_stack_yellow_reserved_zone(addr)) {
      if (thread->thread_state() == _thread_in_Java) {
        if (thread->in_stack_reserved_zone(addr)) {
          frame fr;
          if (os::Linux::get_frame_at_stack_banging_point(thread, uc, &fr)) {
            assert(fr.is_java_frame(), "Must be a Java frame");
            frame activation =
              SharedRuntime::look_for_reserved_stack_annotated_method(thread, fr);
            if (activation.sp() != NULL) {
              thread->disable_stack_reserved_zone();
              if (activation.is_interpreted_frame()) {
                thread->set_reserved_stack_activation((address)(
                  activation.fp() + frame::interpreter_frame_initial_sp_offset));
              } else {
                thread->set_reserved_stack_activation((address)activation.unextended_sp());
              }
              return 1;
            }
          }
        }
        // Throw a stack overflow exception.  Guard pages will be reenabled
        // while unwinding the stack.
        thread->disable_stack_yellow_reserved_zone();
        stub = SharedRuntime::continuation_for_implicit_exception(thread, pc, SharedRuntime::STACK_OVERFLOW);
      } else {
        // Thread was in the vm or native code.  Return and try to finish.
        thread->disable_stack_yellow_reserved_zone();
        return 1;
      }
    } else if (thread->in_stack_red_zone(addr)) {
      // Fatal red zone violation.  Disable the guard pages and fall through
      // to handle_unexpected_exception way down below.
      thread->disable_stack_red_zone();
      tty->print_raw_cr("An irrecoverable stack overflow has occurred.");
      // This is a likely cause, but hard to verify. Let's just print
      // it as a hint.
      tty->print_raw_cr("Please check if any of your loaded .so files has "
                        "enabled executable stack (see man page execstack(8))");
    } else {
       //...

As can be noticed from the code. If the SIGSEGV is raised when on the reserved or yellow zone the guard pages are unguarded, the StackOverflowError error is dispatched into Java and the guard is re-enabled on stack unwinding.

By contrast, if the SIGSEGV is raised when in red zone it is treated as "irrecoverable stack overflow error".

For example:

public class Main {
    static void foo() throws Exception {
        foo();
    }

    public static void main(String args[]) throws Exception {
        foo();
    }
}

as can be noticed with gdb this StackOverflowError occurred in reserved/yellow zone, not in the red (so this is probably the reason it can be handled by the try-catch handler).

As a conclusion the HotSpot guard red zone has completely different meaning from AMD64 System V ABI red zone definition (simply the area that a caller can put local vars so that they are not touched by signal/interrupt handler or probably by debuggers since gdb might put its own data on the stack outside of the red zone).