bhyve: Optionally put vCPUs back in the debug state after resuming

When the gdb stub is configured to pause guest execution upon boot (i.e.,
the "w" flag is passed to -G), vCPUs end up suspended in two senses: first,
suspended by the GDB stub (marked in the vcpus_suspended set), and suspended
by the kernel (because fbsdrun_addcpu() suspends APs before spawning their
vCPU threads).  When the guest is resumed by the debugger, vCPUs are
unsuspended in both senses, but this is not correct for APs.

Hack around this problem by re-suspending vCPUs after the debugger
resumes guest execution, if they were suspended beforehand.

Reviewed by:	corvink, jhb
MFC after:	2 weeks
Sponsored by:	Innovate UK
Differential Revision:	https://reviews.freebsd.org/D46196

usr.sbin/bhyve/gdb.c

@@ -938,8 +938,28 @@ gdb_cpu_add(struct vcpu *vcpu)
 	 * executing the first instruction.
 	 */
 	if (!CPU_EMPTY(&vcpus_suspended)) {
+		cpuset_t suspended;
+		int error;
+
+		error = vm_debug_cpus(ctx, &suspended);
+		assert(error == 0);
+
 		CPU_SET(vcpuid, &vcpus_suspended);
 		_gdb_cpu_suspend(vcpu, false);
+
+		/*
+		 * In general, APs are started in a suspended mode such that
+		 * they exit with VM_EXITCODE_DEBUG until the BSP starts them.
+		 * In particular, this refers to the kernel's view of the vCPU
+		 * state rather than our own.  If the debugger resumes guest
+		 * execution, vCPUs will be unsuspended from the kernel's point
+		 * of view, so we should restore the previous state before
+		 * continuing.
+		 */
+		if (CPU_ISSET(vcpuid, &suspended)) {
+			error = vm_suspend_cpu(vcpu);
+			assert(error == 0);
+		}
 	}
 	pthread_mutex_unlock(&gdb_lock);
 }