Forwarding gpg-agent over SSH
Forwarding GPG’s sockets over an SSH connection allows the client’s keys to be used on the remote machine. And it’s easy – all we need to do is forward one of GPG’s UNIX sockets to the remote machine. Well, not quite…
I’m only going to be able to explain this iteratively so you can follow my logic. A complete implementation will be found at the end of this article, if you would like to skip straight down there. You may miss important context and caveats if you do. This is being written for a Mac OS 14 (Sonoma) client and a Debian 12 (Bookworm) server, but it should work with any combination of platforms if you keep the caveats along the way in mind.
Take 1
Our first goal is just go get a socket forwarded. gpg-agent
offers an “extra” socket for forwarding purposes (I’m not entirely sure what this does – maybe just something with the Pinentry prompt) that we need to forward to the location of the expected “normal” socket. Where are these sockets? Find them with gpgconf
:
client% gpgconf -qL agent-extra-socket
/Users/jhujhiti/.gnupg/S.gpg-agent.extra
server% gpgconf -qL agent-socket
/run/user/1000/gnupg/S.gpg-agent
We can forward these UNIX sockets just like a TCP socket: ssh -R/run/user/1000/gnupg/S.gpg-agent:/Users/jhujhiti/.gnupg/S.gpg-agent.extra server
. But if you try to SSH now, you will probably see an error like Warning: remote port forwarding failed for listen path /run/user/1000/gnupg/S.gpg-agent
. This is because a socket on the remote machine already exists, likely started by an already-running gpg-agent
or by Systemd trying to do socket-activation and holding those sockets open. We can ask SSH to delete any existing socket it finds at the destination location with the option StreamLocalBindUnlink
. This option is documented in both ssh_config(5)
and sshd_config(5)
. The setting on the client appears to have no effect whatsoever, but it’s documented, so let’s set it anyway. The server-side option can be set in /etc/ssh/sshd_config
or, preferably, a wildcard-included file /etc/ssh/sshd_config.d/gpg-forwarding.conf
. Debian’s default sshd_config
includes an Include
line right at the top. I’m unsure about other distros. While we’re here, let’s also make sure that the other options we need for UNIX socket forwarding to work are enabled (they were by default for me).
AllowAgentForwarding yes
AllowStreamLocalForwarding all
StreamLocalBindUnlink yes
We can also create a static entry in our SSH client configuration to set the client options, but don’t get too fancy in there because this is not going to last long.
Host server
StreamLocalBindUnlink yes
RemoteForward /run/user/1000/gnupg/S.gpg-agent /Users/jhujhiti/.gnupg/S.gpg-agent.extra
This will work as-is. You can ssh server
now and should find that you can access your client’s agent by doing something like gpg --list-secret-keys
.
There are three major problems with this:
- This only works against the one server in our SSH client config. You could use
ssh_config
’s wildcards and compound matchers to fix this, except… - The socket locations on client and server are hard-coded. How do we know those will always be the right paths? I certainly don’t only use Linux servers where my UID is 1000, so that
/run/user/1000
is already multi-dimensionally wrong. - Multiple forwards will stomp on each other. If I connect to the same server from two clients set up like this, the second client will destroy the first client’s socket. Worse, if I connect to the same server twice from the same client and then close the second session, the first session will be broken.
We can do much better.
Take 2
This section contains links to C code formatted in the GNU style. I am not responsible for any psychic damage taken or existential crises experienced as a result of viewing it. It’s also just not very good code. You have been warned.
Let’s solve the second problem first. If you’re running GPG 2.4.4 or later everywhere, you’re in luck. They have added support for configuring the socket path with a socketdir
option: https://dev.gnupg.org/T6796#180962. Just set that to ~/.gnupg
and proceed with Take 3 (but pay attention, it will need to change to ~/.gnupg/socket
in that step). If you aren’t so fortunate, strap in.
Prior to 2.4.4, we’re stuck with the default GPG behavior. Rather than just store sockets in the configured home directory, GPG hardcodes a list of directories to try first: /run/gnupg/user/<uid>/gnupg
(maybe), /run/user/<uid>/gnupg
, /var/run/gnupg/user/<uid>/gnupg
(maybe), and /var/run/user/<uid>/gnupg
. If we make GPG angry enough by leaving a file there instead of a directory, it will fall back to the home directory. Early in your shell’s startup (I use Zsh), make sure there’s a file rather than a directory in each of these paths.
# stop gpg from trying to store sockets in eg., # /run/user/${UID}/gnupg, forcing it to fall back to the home # directory. it might be in /run or /var/run. less likely, but it # could also be /run/gnupg or /var/run/gnupg if it was built with # --enable-run-gnupg-user-socket for base in /run /var/run /run/gnupg /var/run/gnupg do # existence check to prevent us from trying to create the same # file twice if the filesystems are the same (eg., /run and # /var/run are often bind-mounted or symlinked) [ -d ${base}/user/${UID} -a ! -e ${base}/user/${UID}/gnupg ] && \ touch ${base}/user/${UID}/gnupg done
If you close all of your sessions on the remote host and leave nothing running, Systemd will clean up the /run/user/<uid>
tmpfs mount after a brief wait (I’m not sure what it is, but definitely a minute at most). But the sockets there will come back when you log back in. Something is beating the shell to it. On my Debian system, I found two causes for this.
First is Systemd socket activation. If you aren’t familiar, this is a feature that gives Systemd ownership of a socket, and attempts to use the socket will trigger an associated service to start. We need to prevent these sockets from being created. There are two options: either mask the socket units or use a drop-in unit to override the problematic path. I chose the latter because my ultimate goal is to allow the same configuration on client and server without breaking any features. Here are the sockets we’re concerned about:
server% systemctl --user list-sockets -l --no-legend | grep "/run/user/${UID}/gnupg"
/run/user/1000/gnupg/S.dirmngr dirmngr.socket dirmngr.service
/run/user/1000/gnupg/S.gpg-agent gpg-agent.socket gpg-agent.service
/run/user/1000/gnupg/S.gpg-agent.browser gpg-agent-browser.socket gpg-agent.service
/run/user/1000/gnupg/S.gpg-agent.extra gpg-agent-extra.socket gpg-agent.service
/run/user/1000/gnupg/S.gpg-agent.ssh gpg-agent-ssh.socket gpg-agent.service
Create a drop-in override for each one and kick Systemd.
server% DROPIN_DIR=${XDG_CONFIG_HOME:-~/.config}/systemd/user/
server% mkdir -p ${DROPIN_DIR}
server% cd ${DROPIN_DIR}
server% mkdir -p gpg-agent{,-browser,-extra,-ssh}.socket.d dirmngr.socket.d
server% echo "[Socket]\nListenStream=\nListenStream=${HOME}/.gnupg/S.gpg-agent" > gpg-agent.socket.d/socket-path.conf
server% echo "[Socket]\nListenStream=\nListenStream=${HOME}/.gnupg/S.gpg-agent.extra" > gpg-agent-extra.socket.d/socket-path.conf
server% echo "[Socket]\nListenStream=\nListenStream=${HOME}/.gnupg/S.gpg-agent.browser" > gpg-agent-browser.socket.d/socket-path.conf
server% echo "[Socket]\nListenStream=\nListenStream=${HOME}/.gnupg/S.gpg-agent.ssh" > gpg-agent-ssh.socket.d/socket-path.conf
server% echo "[Socket]\nListenStream=\nListenStream=${HOME}/.gnupg/S.dirmngr" > dirmngr.socket.d/socket-path.conf
server% systemctl --user daemon-reload
The other cause is trickier. I actually got so desperate here that I used SystemTap from another user account so I could watch the Systemd user daemon start up. It turns out that there is a pluggable system to configure the environment and Debian included a script with GPG. What the script intends to do isn’t important, but it runs gpgconf
as it does it, and gpgconf
helpfully notices that its hardcoded paths don’t exist. It’s able to create them because the shell hasn’t started yet. Here’s the script:
server% dpkg -L gpg-agent | grep user-environment
/usr/lib/systemd/user-environment-generators
/usr/lib/systemd/user-environment-generators/90gpg-agent
#!/bin/bash # If enable-ssh-support is present in gpg-agent.conf, export SSH_AUTH_SOCK # pointing at the gpg-agent's ssh-agent compatibility layer. # Authors: # rufo <rufo@rufoa.com> # Daniel Kahn Gillmor <dkg@fifthhorseman.net> # See https://bugs.debian.org/855868 # see gpgconf(1): $5 is the "okay" field. # see also https://dev.gnupg.org/T4866 and https://dev.gnupg.org/T4867 get_okay='BEGIN{ret=1} /^gpg-agent:/{if ($5 == "1") { ret=0; exit 0 } } END {exit ret}' if gpgconf --check-options gpg-agent | awk -F: "$get_okay" && \ [ -n "$(gpgconf --list-options gpg-agent | \ awk -F: '/^enable-ssh-support:/{ print $10 }')" ]; then echo SSH_AUTH_SOCK=$(gpgconf --list-dirs agent-ssh-socket) echo GSM_SKIP_SSH_AGENT_WORKAROUND=true fi
I would have loved to add a unit to touch /run/user/<uid>/gnupg
earlier in the login process than the shell can run, but Systemd runs these environment generators before starting any units, so there are no unit dependency tricks we can play. Our options again are to mask it or patch it. In this case, patching it to clean up after itself only if it created a directory seems hard and I don’t care if SSH_AUTH_SOCK
gets set for anyone, so let’s just mask it.
server# mkdir /etc/systemd/user-environment-generators
server# ln -s /dev/null /etc/systemd/user-environment-generators/90gpg-agent
Having wrested control of /run/user/<uid>/gnupg
away from Systemd, we can be sure gpgconf
will always return ~/.gnupg
as its socket directory. We can account for the new directory in the SSH client config, and it is here we find our first caveat: The home directory on the remote machine is unknowable by the SSH client. Assuming /home/<username>
is the best we can do. This is safe for my use. While we’re here, we can improve the configuration to use the built-in tokens for remote username and local home directory.
Host server
StreamLocalBindUnlink yes
RemoteForward /home/%r/.gnupg/S.gpg-agent %d/.gnupg/S.gpg-agent.extra
Now to tackle the real problem.
Take 3
We still need to prevent multiple clients from stepping on each other’s toes. The fundamental problem is that the remote side of the socket is always in the same place, /home/<username>/.gnupg/S.gpg-agent
after the last section. Any simultaneous clients with this forward will break something. We need to introduce some sort entropy into this filename.
After thinking about what to add to the remote filename for a while, I decided (caveat two) to limit forwarded sockets to interactive SSH sessions. This gives us two pieces of entropy: something to identify the client machine and something to identify the interactive session. Specifically, the client hostname and the client TTY. Limiting this to interactive sessions lets us put this entropy logic into our shell. One more thing: let’s clean up and normalize this entropy with a hash function.
# find a working sha1 checksum. their output is all similar enough to # work below for CHECKSUM in sha1sum shasum 'openssl sha1' do whence ${CHECKSUM%% *} >/dev/null && break done # this hash of the tty name will be used to differentiate forwarded # gpg-agent sessions on the remote host TTY_HASH=$({hostname -f ; tty ; } | $=CHECKSUM | cut -c1-12) unset CHECKSUM
Another fringe benefit of being interactive-only is that we can use a shell function to wrap ssh
. This turns out to be important, because of how environment variable expansion works in ssh_config
files. Rather than expand to nothing, the entire program will abort if it can’t expand a variable. Weirder and even worse, it tries to expand even when the variable is used in a Host
or Match
declaration that isn’t going to be matched. This is fatal for any attempts to SSH from outside of the shell. Using a shell function rather than a wrapper script in, say, ~/bin/ssh
, ensures we don’t run into weird issues with Sudo or other things that spawn their own shell and/or filter environment variables. You’ll notice that TTY_HASH
isn’t even exported in the block above.
Here’s the shell function. There are a few Zshisms in here. Sorry, not sorry.
# wrap interactive ssh sessions so we can forward gpg-agent sockets # through. sadly this is necessary because we need to do environment # variable expansion in the socket path on the remote side and there # are contexts where the variable will not be set (any ssh command run # outside the shell). ssh will error if it fails to do a variable # substitution even in cases where the substitution happens in a match # stanza that is not going to be used. function ssh { # first we need to figure out what the hostname and username ssh # wants to use are. this isn't trivial so we'll let ssh process # its config and tell us. TODO: this might need considerations for # SSH's hostname canonicalization local ssh_params_raw ssh_params_raw=$(command ssh -G "$@") || return $? local -A ssh_params ssh_params=(${(@f)$(grep -Ei '^(user|hostname) ' <<< "$ssh_params_raw")}) || return $? local -a ssh_args # now we can match against them if [ "$ssh_params[user]" = 'jhujhiti' -a "${ssh_params[hostname]%%.adjectivism.org}" != "$ssh_params[hostname]" ] then local GPG_SOCKET_PREFIX GPG_SOCKET_PREFIX="${TTY_HASH}-" ssh_args+=( -oStreamLocalBindUnlink=yes # assumption: all machines we ssh *to* have /home/username # homedirs -R"/home/%r/.gnupg/socket/${GPG_SOCKET_PREFIX}S.gpg-agent:%d/.gnupg/socket/S.gpg-agent.extra" -o"SetEnv=GPG_SOCKET_PREFIX=${GPG_SOCKET_PREFIX}" ) fi command ssh "$ssh_args[@]" "$@" }
This will replace the ~/.ssh/config
changes from before. And it will expand on the single-server Host
clause in the earlier SSH configurations to match any attempts to connect to a *.adjectivism.org
machine as my normal username. The username is hardcoded here to make sure it works as intended if this shell runs on a machine that isn’t mine where my username might be different – it’s the remote username I care about, not the local one.
Accounting for these changes on the server side first requires a new SSHD option to be set:
AcceptEnv GPG_SOCKET_PREFIX
After allowing the variable to propagate via the SSH session, we need to use it when figuring out which gpg-agent
socket we’re going to use in a given session. GPG gives us this strange libassuan
that allows variable expansion in socket paths. We will replace the usual socket files (that are in ~/.gnupg
now, thanks to Take 2) with these weird little files that point to the real socket location we want to use. Do this on both client and server for consistency.
both% cd ~/.gnupg
both% mkdir socket
both% rm -f S.gpg-agent{,.extra,.ssh,.browser} S.dirmngr
both% echo "%Assuan%\nsocket=${HOME}/.gnupg/socket/${GPG_SOCKET_PREFIX}S.gpg-agent" > S.gpg-agent
both% echo "%Assuan%\nsocket=${HOME}/.gnupg/socket/${GPG_SOCKET_PREFIX}S.gpg-agent.ssh" > S.gpg-agent.ssh
both% echo "%Assuan%\nsocket=${HOME}/.gnupg/socket/${GPG_SOCKET_PREFIX}S.gpg-agent.extra" > S.gpg-agent.extra
both% echo "%Assuan%\nsocket=${HOME}/.gnupg/socket/${GPG_SOCKET_PREFIX}S.gpg-agent.browser" > S.gpg-agent.browser
both% echo "%Assuan%\nsocket=${HOME}/.gnupg/socket/${GPG_SOCKET_PREFIX}S.dirmngr" > S.dirmngr
The null-expansion of GPG_SOCKET_PREFIX
here has the nice property that it returns a plain socket named eg., ~/.gnupg/socket/S.gpg-agent
so that these files are also safe to use on a client machine where gpg-agent
is started outside of this shell.
Final state
Here’s the full collection of configurations and setup commands.
AllowAgentForwarding yes
AllowStreamLocalForwarding all
StreamLocalBindUnlink yes
AcceptEnv GPG_SOCKET_PREFIX
server% DROPIN_DIR=${XDG_CONFIG_HOME:-~/.config}/systemd/user/
server% mkdir -p ${DROPIN_DIR}
server% cd ${DROPIN_DIR}
server% mkdir -p gpg-agent{,-browser,-extra,-ssh}.socket.d dirmngr.socket.d
server% echo "[Socket]\nListenStream=\nListenStream=${HOME}/.gnupg/S.gpg-agent" > gpg-agent.socket.d/socket-path.conf
server% echo "[Socket]\nListenStream=\nListenStream=${HOME}/.gnupg/S.gpg-agent.extra" > gpg-agent-extra.socket.d/socket-path.conf
server% echo "[Socket]\nListenStream=\nListenStream=${HOME}/.gnupg/S.gpg-agent.browser" > gpg-agent-browser.socket.d/socket-path.conf
server% echo "[Socket]\nListenStream=\nListenStream=${HOME}/.gnupg/S.gpg-agent.ssh" > gpg-agent-ssh.socket.d/socket-path.conf
server% echo "[Socket]\nListenStream=\nListenStream=${HOME}/.gnupg/S.S.dirmngr" > dirmngr.socket.d/socket-path.conf
server% systemctl --user daemon-reload
server# mkdir /etc/systemd/user-environment-generators
server# ln -s /dev/null /etc/systemd/user-environment-generators/90gpg-agent
# stop gpg from trying to store sockets in eg., # /run/user/${UID}/gnupg, forcing it to fall back to the home # directory. it might be in /run or /var/run. less likely, but it # could also be /run/gnupg or /var/run/gnupg if it was built with # --enable-run-gnupg-user-socket for base in /run /var/run /run/gnupg /var/run/gnupg do # existence check to prevent us from trying to create the same # file twice if the filesystems are the same (eg., /run and # /var/run are often bind-mounted or symlinked) [ -d ${base}/user/${UID} -a ! -e ${base}/user/${UID}/gnupg ] && \ touch ${base}/user/${UID}/gnupg done
# find a working sha1 checksum. their output is all similar enough to # work below for CHECKSUM in sha1sum shasum 'openssl sha1' do whence ${CHECKSUM%% *} >/dev/null && break done # this hash of the tty name will be used to differentiate forwarded # gpg-agent sessions on the remote host TTY_HASH=$({hostname -f ; tty ; } | $=CHECKSUM | cut -c1-12) unset CHECKSUM # wrap interactive ssh sessions so we can forward gpg-agent sockets # through. sadly this is necessary because we need to do environment # variable expansion in the socket path on the remote side and there # are contexts where the variable will not be set (any ssh command run # outside the shell). ssh will error if it fails to do a variable # substitution even in cases where the substitution happens in a match # stanza that is not going to be used. function ssh { # first we need to figure out what the hostname and username ssh # wants to use are. this isn't trivial so we'll let ssh process # its config and tell us. TODO: this might need considerations for # SSH's hostname canonicalization local ssh_params_raw ssh_params_raw=$(command ssh -G "$@") || return $? local -A ssh_params ssh_params=(${(@f)$(grep -Ei '^(user|hostname) ' <<< "$ssh_params_raw")}) || return $? local -a ssh_args # now we can match against them if [ "$ssh_params[user]" = 'jhujhiti' -a "${ssh_params[hostname]%%.adjectivism.org}" != "$ssh_params[hostname]" ] then local GPG_SOCKET_PREFIX GPG_SOCKET_PREFIX="${TTY_HASH}-" ssh_args+=( -oStreamLocalBindUnlink=yes # assumption: all machines we ssh *to* have /home/username # homedirs -R"/home/%r/.gnupg/socket/${GPG_SOCKET_PREFIX}S.gpg-agent:%d/.gnupg/socket/S.gpg-agent.extra" -o"SetEnv=GPG_SOCKET_PREFIX=${GPG_SOCKET_PREFIX}" ) fi command ssh "$ssh_args[@]" "$@" }
both% cd ~/.gnupg
both% mkdir socket
both% rm -f S.gpg-agent{,.extra,.ssh,.browser} S.dirmngr
both% echo "%Assuan%\nsocket=${HOME}/.gnupg/socket/${GPG_SOCKET_PREFIX}S.gpg-agent" > S.gpg-agent
both% echo "%Assuan%\nsocket=${HOME}/.gnupg/socket/${GPG_SOCKET_PREFIX}S.gpg-agent.ssh" > S.gpg-agent.ssh
both% echo "%Assuan%\nsocket=${HOME}/.gnupg/socket/${GPG_SOCKET_PREFIX}S.gpg-agent.extra" > S.gpg-agent.extra
both% echo "%Assuan%\nsocket=${HOME}/.gnupg/socket/${GPG_SOCKET_PREFIX}S.gpg-agent.browser" > S.gpg-agent.browser
both% echo "%Assuan%\nsocket=${HOME}/.gnupg/socket/${GPG_SOCKET_PREFIX}S.dirmngr" > S.dirmngr
All of these are safe to put on both client and server. As a bonus, here is part of the Makefile
I distribute with my dotfiles repository that will set up this GPG stuff.
.SECONDEXPANSION: # we need the sockets in a known location for forwarding over ssh ifneq (,$(shell which gpgconf)) GPG_SOCKETDIR:=$(shell gpgconf -q --list-dir socketdir) # it would be nice if we could rely on the shell to trick gpg into not # storing sockets in /run/user/UID/gnupg, but it won't work during the # initial dotfiles setup, since the idea is to run this Makefile in # one shot and then open a new shell. it almost seems like we could # make the socketdir target depend on .zshrc and then execute the # gpgconf inside a shell, but that might be a bit too clever. the # shell might not be set yet, and running zsh explicitly will fail if # it's not even installed yet. so, let's make this dummy file here as # well. # only bother with this if gpgconf says it wants to use /run or # /var/run (ie., if it's on an affected platform that hasn't been # fixed by the shell already). ifneq (,$(filter /run/% /var/run/%,$(GPG_SOCKETDIR))) $(GPG_SOCKETDIR): # gpg won't use the directory if it can't create it, so we don't need # to care if the touch fails # FIXME: we really need to run this multiple times until we no longer # get these directories back from gpgconf... rm -rf $(GPG_SOCKETDIR) touch $(GPG_SOCKETDIR) || echo '' ../.gnupg/socket: $(GPG_SOCKETDIR) endif ../.gnupg/socket: | ../.gnupg install -m 0700 -d $@ gpg: ../.gnupg/socket GPG_SOCKET_TYPES:=gpg-agent gpg-agent.ssh gpg-agent.browser gpg-agent.extra dirmngr GPG_SOCKETS:=$(addprefix ../.gnupg/S.,$(GPG_SOCKET_TYPES)) gpg: $(GPG_SOCKETS) $(GPG_SOCKETS): | ../.gnupg/socket rm -f $@ echo "%Assuan%\nsocket=\$${HOME}/.gnupg/socket/\$${GPG_SOCKET_PREFIX}$(@F)" > $@ # these redirect files might be left over sockets (real sockets) from # gpg-agent. we can conditionally force a remake by marking them phony # and letting the above rm take care of the dangling socket (or # soon-to-be dangling on the other side...) .PHONY: $(shell find $(GPG_SOCKETS) -type s 2>/dev/null) ifneq (,$(shell which systemctl)) # systemd socket units are named like the actual sockets but with - # instead of . (gpg-agent-ssh.socket vs gpg-agent.ssh) SYSTEMD_GPG_SOCKET_UNITS:=$(addsuffix .socket,$(subst .,-,$(GPG_SOCKET_TYPES))) SYSTEMD_GPG_SOCKET_OVERRIDES:=$(addprefix ../.config/systemd/user/,$(addsuffix .d/socket-path.conf,$(SYSTEMD_GPG_SOCKET_UNITS))) # map socket override back to gpg's filename ../.config/systemd/user/gpg-agent.socket.d/socket-path.conf: GPG_SOCKET:=S.gpg-agent ../.config/systemd/user/gpg-agent-ssh.socket.d/socket-path.conf: GPG_SOCKET:=S.gpg-agent.ssh ../.config/systemd/user/gpg-agent-browser.socket.d/socket-path.conf: GPG_SOCKET:=S.gpg-agent.browser ../.config/systemd/user/gpg-agent-extra.socket.d/socket-path.conf: GPG_SOCKET:=S.gpg-agent.extra ../.config/systemd/user/dirmngr.socket.d/socket-path.conf: GPG_SOCKET:=S.dirmngr # we need an absolute path so these can't be symlinked from files # committed to git $(SYSTEMD_GPG_SOCKET_OVERRIDES): | $$(@D)/ ../.gnupg/socket echo "[Socket]\nListenStream=\nListenStream=$(abspath ../.gnupg/socket)/$(GPG_SOCKET)" > $@ $(dir $(SYSTEMD_GPG_SOCKET_OVERRIDES)): | ../.config/systemd/user mkdir $@ ../.config/systemd/user/.gpg-override-stamp: $(SYSTEMD_GPG_SOCKET_OVERRIDES) systemctl --user daemon-reload touch $@ gpg: ../.config/systemd/user/.gpg-override-stamp endif endif