breakpointHook: re-implement using nsenter

breakpointHook is amazing and has been a daily driver for me. Though it has been buggy several times in the past and recently it stopped working on my system, as the approach using cntr seems to be a bit sensitive to the environment and which shell is used etc.

After speaking with @mic92, it turned out there is an alternative approach using `nsenter`. After playing with that for a while, I think we should replace the current implementation of breakpointHook.

The new output when a build fails will look like this:
```
build failed in configurePhase with exit code 1
To attach, run the following command:
    sudo /nix/store/2kzihv08w9vsyi6zhs0pi2i9117v4q1v-attach/bin/attach 28715971284
```

Benefits of the new implementation over the old one:
- no need to install cntr tool on the local machine first, the copy pasted command will just work
- no need to execute `cntr exec ...` manually to enter the namespace
- interactive shell by default
- dropped into correct working directory by default
- $out available by default
- possibility to override the shell with zfs or fish, etc. by defining `debugShell` in the derivation to debug
- probably less prone to bugs as it only uses nsenter

cc @mic92
This commit is contained in:
DavHau 2025-01-02 17:50:44 +07:00 committed by Jörg Thalheim
parent 45b4efbacf
commit d7d59e137c
3 changed files with 104 additions and 0 deletions

View file

@ -0,0 +1,46 @@
#! /usr/bin/env bash
set -eu -o pipefail
id="$1"
pids="$(pgrep -f "sleep $id" || :)"
if [ -z "$pids" ]; then
echo "Error: No process found for 'sleep $id'. The build must still be running in order to attach. Also make sure it's not on a remote builder." >&2
exit 1
elif [ "$(echo "$pids" | wc -l)" -ne 1 ]; then
echo "Error: Multiple processes found matching 'sleep $id'" >&2
exit 1
fi
pid="$(echo "$pids" | head -n1)"
# helper to extract variables from the build env
getVar(){
while IFS= read -r -d $'\0' line; do
case "$line" in
*"$1="* )
echo "$line"
;;
esac
done < /proc/$pid/environ \
| cut -d "=" -f 2
}
# bash is needed to load the env vars, as we do not know the syntax of the debug shell.
# bashInteractive is used instead of bash, as we depend on it anyways, due to it being
# the default debug shell
bashInteractive="$(getVar bashInteractive)"
# the debug shell will be started as interactive shell after loading the env vars
debugShell="$(getVar debugShell)"
# to drop the user into the working directory at the point of failure
pwd="$(readlink /proc/$pid/cwd)"
# enter the namespace of the failed build
exec nsenter --mount --net --target "$pid" "$bashInteractive" -c "
set -eu -o pipefail
while IFS= read -r -d \$'\0' line; do
export \"\$line\"
done <&3
exec 3>&-
cd \"$pwd\"
exec \"$debugShell\"
" 3< /proc/$pid/environ

View file

@ -0,0 +1,20 @@
breakpointHook() {
local red='\033[0;31m'
local cyan='\033[0;36m'
local green='\033[0;32m'
local no_color='\033[0m'
# provide the user with an interactive shell for better experience
export bashInteractive="@bashInteractive@"
if [ -z "$debugShell" ]; then
export debugShell="@bashInteractive@"
fi
local id
id="$(shuf -i 999999-9999999 -n1)"
echo -e "${red}build for ${cyan}${name:-unknown}${red} failed in ${curPhase:-unknown} with exit code ${exitCode:-unknown}${no_color}"
echo -e "${green}To attach, run the following command:${no_color}"
echo -e "${green} sudo @attach@ $id${no_color}"
sleep "$id"
}
failureHooks+=(breakpointHook)

View file

@ -0,0 +1,38 @@
{
lib,
stdenv,
bash,
bashInteractive,
coreutils,
makeSetupHook,
procps,
util-linux,
writeShellScriptBin,
}:
let
attach = writeShellScriptBin "attach" ''
export PATH="${
lib.makeBinPath [
bash
coreutils
procps # needed for pgrep
util-linux # needed for nsenter
]
}"
exec bash ${./attach.sh} "$@"
'';
in
makeSetupHook {
name = "breakpoint-hook";
meta.broken = !stdenv.buildPlatform.isLinux;
substitutions = {
attach = "${attach}/bin/attach";
# The default interactive shell in case $debugShell is not set in the derivation.
# Can be overridden to zsh or fish, etc.
# This shell is also used to load the env variables before the $debugShell is started.
bashInteractive = lib.getExe bashInteractive;
};
} ./breakpoint-hook.sh