SSH: What happens behind the scenes
Let's tear apart all the layers of SSH
Command Palette
Search for a command to run...
Comments
Join the discussionNo comments yet. Be the first to comment.
Search for a command to run...
Let's tear apart all the layers of SSH
No comments yet. Be the first to comment.
Every time you type ssh user@host, you trigger a cryptographic ceremony that involves at least 6 distinct protocol phases, 3 layers of encryption negotiation and a minimum of 8 TCP round trips i.e., before a single byte of your shell session is transmitted. Let's tear it apart.
Not just secure telnet. SSH is three protocols stacked on top of each other, all defined in RFC 4251–4254
| Layer | RFC | Purpose |
|---|---|---|
| SSH Transport Layer | RFC 4253 | Encryption, key exchange, integrity |
| SSH Authentication Layer | RFC 4252 | User identity verification |
| SSH Connection Layer | RFC 4254 | Channels, multiplexing, port forwarding |
All of this rides on TCP port 22 (one persistent TCP connection)
Every SSH message after key exchange looks like this
┌──────────────────────────────
│ uint32 packet_length (4 bytes) │
│ byte padding_length (1 byte) │
│ byte[] payload (variable) │
│ byte[] random_padding (padding_length) │
│ byte[] MAC (variable, optional) │
└──────────────────────────────
packet_length = length of everything AFTER the length field itself
Padding is added to make packet_length + padding_length + 1 a multiple of cipher block size (or 8)
MAC (Message Authentication Code) covers: sequence number + packet content — not encrypted, computed over plaintext then appended to ciphertext
NOTE: SSH is NOT TLS. It has its own key exchange, its own record format, its own MAC scheme.
Before SSH does anything, TCP happens:
Client → SYN → Server
Client ← SYN-ACK ← Server
Client → ACK → Server
3 packets. Connection established on port 22.
Observable
tcpdump -i eth0 'tcp port 22' -nn
First thing both sides do — send a plaintext identification string
Server → "SSH-2.0-OpenSSH_9.3\r\n"
Client → "SSH-2.0-OpenSSH_9.3\r\n"
This is cleartext — fully visible in Wireshark
Format: SSH-protoversion-softwareversion [comments]
Client and server both record each other's version — used later in key exchange hash
If versions are incompatible, connection drops immediately
Observable
# Capture the banner
nc -v 192.168.1.100 22
# or
ssh -vvv user@host 2>&1 | head -20
Both sides send a SSH_MSG_KEXINIT packet simultaneously (no waiting). This is the "menu"
byte SSH_MSG_KEXINIT (20)
byte[16] cookie (random bytes)
name-list kex_algorithms
name-list server_host_key_algorithms
name-list encryption_algorithms_client_to_server
name-list encryption_algorithms_server_to_client
name-list mac_algorithms_client_to_server
name-list mac_algorithms_server_to_client
name-list compression_algorithms_client_to_server
name-list compression_algorithms_server_to_client
...
Negotiation rule: client's preference list wins — first algorithm in the client's list that the server also supports is chosen.
Typical modern negotiation result:
| What | Chosen Algorithm |
|---|---|
| Key Exchange | curve25519-sha256 |
| Host Key | ssh-ed25519 |
| Encryption | [email protected] or aes256-gcm |
| MAC | (implicit with AEAD cipher) |
| Compression | none (usually) |
Observable
ssh -vvv user@host 2>&1 | grep -E 'kex|cipher|mac|compress'
# Shows:
# kex: algorithm: curve25519-sha256
# kex: host key algorithm: ssh-ed25519
# kex: server->client cipher: chacha20-poly1305...
This is the most important phase. Using Diffie-Hellman or ECDH to establish a shared secret without transmitting it.
With curve25519-sha256 (most modern):
Step 1 — Client generates ephemeral keypair:
client_private_key = random scalar (32 bytes)
client_public_key = scalar × G (Curve25519 base point)
Step 2 — Client sends SSH_MSG_KEX_ECDH_INIT:
byte SSH_MSG_KEX_ECDH_INIT (30)
string Q_C ← client's ephemeral public key
Step 3 — Server computes shared secret + signs everything:
Server has:
Its own ephemeral keypair (server_priv, server_pub)
Client's public key Q_C
Server computes:
K (shared_secret) = server_priv × Q_C
= client_priv × Q_S (ECDH magic — same result both sides)
Server builds exchange hash H:
H = SHA-256(
V_C || V_S || ← version strings from Phase 1
I_C || I_S || ← KEXINIT payloads from Phase 2
K_S || ← server's HOST key (public, long-term)
Q_C || ← client's ephemeral pubkey
Q_S || ← server's ephemeral pubkey
K ← shared secret
)
Server sends SSH_MSG_KEX_ECDH_REPLY:
string K_S ← server host public key
string Q_S ← server ephemeral public key
string sig_H ← signature of H using K_S (host key)
Step 4 — Client verifies:
1. Compute K = client_priv × Q_S (same shared secret!)
2. Recompute H
3. Verify sig_H using K_S
4. Check K_S against known_hosts (TOFU or CA)
Critical insight: The shared secret K is never transmitted. Both sides compute it independently. Even if someone recorded all packets, they cannot compute K without either private key. This is Forward Secrecy.
Step 5 — Key Derivation (RFC 4253)
From K and H, six keys are derived using the hash:
IV client→server = HASH(K || H || "A" || session_id)
IV server→client = HASH(K || H || "B" || session_id)
Key client→server = HASH(K || H || "C" || session_id)
Key server→client = HASH(K || H || "D" || session_id)
MAC client→server = HASH(K || H || "E" || session_id)
MAC server→client = HASH(K || H || "F" || session_id)
From this point forward: all packets are encrypted.
Step 6 — SSH_MSG_NEWKEYS: Both sides send this to say "switching to new keys now." One packet each.
Now inside encrypted tunnel. Client must prove identity.
Step 1 — Service request:
SSH_MSG_SERVICE_REQUEST → "ssh-userauth"
SSH_MSG_SERVICE_ACCEPT ← server confirms
Step 2 — Auth methods (tried in order):
Method A: publickey (most common)
1. Client sends probe:
SSH_MSG_USERAUTH_REQUEST {
username, "ssh-connection", "publickey",
FALSE, ← just checking if key is acceptable
"ssh-ed25519", public_key_blob
}
2. Server responds SSH_MSG_USERAUTH_PK_OK if key is in authorized_keys
3. Client sends actual auth:
SSH_MSG_USERAUTH_REQUEST {
..same..,
TRUE, ← now actually signing
signature of (session_id || request_data) using private key
}
4. Server verifies signature → SSH_MSG_USERAUTH_SUCCESS
Where is authorized_keys checked?
~/.ssh/authorized_keys (default)
# or configured via AuthorizedKeysFile in sshd_config
# or AuthorizedKeysCommand for LDAP/Vault lookups
Method B: password
SSH_MSG_USERAUTH_REQUEST {
username, "ssh-connection", "password",
FALSE,
plaintext_password ← encrypted by transport layer!
}
Method C: keyboard-interactive (used for MFA/PAM)
Multi-round challenge-response. Used by Google Authenticator PAM, Duo, etc.
Observable
ssh -vvv user@host 2>&1 | grep -E 'auth|Authentications'
# debug1: Authentications that can continue: publickey,password
# debug1: Next authentication method: publickey
# debug1: Offering public key: /home/user/.ssh/id_ed25519
Authentication passed. Now the connection protocol opens logical channels over the single TCP connection.
Opening a channel
Client → SSH_MSG_CHANNEL_OPEN {
channel_type: "session"
sender_channel: 0
initial_window_size: 2097152 (2MB)
maximum_packet_size: 32768
}
Server → SSH_MSG_CHANNEL_OPEN_CONFIRMATION {
recipient_channel: 0
sender_channel: 0
initial_window_size: ...
maximum_packet_size: ...
}
Requesting a PTY (pseudo-terminal):
SSH_MSG_CHANNEL_REQUEST {
channel: 0
request_type: "pty-req"
want_reply: TRUE
TERM: "xterm-256color"
width_chars, height_chars, width_px, height_px
terminal_modes: (speed, echo, etc.)
}
Requesting a shell:
SSH_MSG_CHANNEL_REQUEST {
channel: 0
request_type: "shell"
want_reply: TRUE
}
Data flow:
Client → SSH_MSG_CHANNEL_DATA { channel: 0, data: "ls -la\n" }
Server → SSH_MSG_CHANNEL_DATA { channel: 0, data: "total 48\n..." }
Flow control — window adjust:
When receiver's buffer fills:
SSH_MSG_CHANNEL_WINDOW_ADJUST { channel: 0, bytes_to_add: 2097152 }
Key insight: Multiple channels can run simultaneously — shell + port forwards + X11 — all multiplexed over one TCP connection with independent flow control windows
User types exit → shell sends EOF
SSH_MSG_CHANNEL_EOF (no more data from this side)
SSH_MSG_CHANNEL_CLOSE (I'm done with this channel)
SSH_MSG_DISCONNECT {
reason_code: SSH_DISCONNECT_BY_APPLICATION
description: "logout"
}
TCP FIN/ACK
ssh -vvv — Built-in debugssh -vvv -o 'LogLevel=DEBUG3' user@host 2>&1 | less
Shows every protocol message, algorithm chosen, key used.
# Capture SSH handshake (cleartext phase visible)
tshark -i eth0 -f 'tcp port 22' -w /tmp/ssh.pcap
# Analyze
tshark -r /tmp/ssh.pcap -Y 'ssh' -T fields \
-e frame.number -e ssh.message_code -e ssh.kex.algorithms
In Wireshark:
Filter: ssh
You'll see: Protocol: SSHv2, message types like Key Exchange Init, Elliptic Curve Diffie-Hellman Key Exchange Init/Reply, New Keys
After New Keys: everything shows as Encrypted packet
strace — System call level# Trace the ssh client
strace -e trace=network,read,write -f ssh user@host 2>&1 | head -100
# On server side, trace sshd for a specific connection
strace -p $(pgrep -f "sshd: user") -e trace=read,write
# Trace all connect() calls to port 22
bpftrace -e '
tracepoint:syscalls:sys_enter_connect /
((struct sockaddr_in *)args->uservaddr)->sin_port == 5632
/ {
printf("SSH connect: pid=%d comm=%s\n", pid, comm);
}'
# Watch SSH data writes (channel data)
bpftrace -e '
kprobe:tcp_sendmsg /comm == "ssh"/ {
printf("ssh write: pid=%d bytes=%d\n", pid, arg2);
}'
/proc and kernel crypto — See what's happening inside# See active SSH connections
ss -tnp | grep :22
# Check which crypto is loaded
cat /proc/crypto | grep -A5 'curve25519\|chacha20'
# OpenSSL timing of key exchange (benchmark)
openssl speed curve25519 ed25519
sshd debug mode — Server-side visibility# Run sshd in debug mode on alternate port (non-disruptive)
/usr/sbin/sshd -d -p 2222
# Full verbose
/usr/sbin/sshd -ddd -p 2222
Complete timeline of one SSH connection
t=0ms TCP SYN
t=1ms TCP SYN-ACK
t=2ms TCP ACK ──────────────────── TCP established
t=3ms Server → SSH banner (cleartext)
t=4ms Client → SSH banner (cleartext)
t=5ms Client → SSH_MSG_KEXINIT
t=5ms Server → SSH_MSG_KEXINIT (simultaneous)
t=6ms Client → SSH_MSG_KEX_ECDH_INIT (client ephemeral pubkey)
t=8ms Server → SSH_MSG_KEX_ECDH_REPLY (host key + sig + server pubkey)
t=8ms [Both sides derive 6 symmetric keys]
t=9ms Client → SSH_MSG_NEWKEYS
t=9ms Server → SSH_MSG_NEWKEYS
──────────────────────── EVERYTHING ENCRYPTED FROM HERE
t=10ms Client → SSH_MSG_SERVICE_REQUEST ("ssh-userauth")
t=11ms Server → SSH_MSG_SERVICE_ACCEPT
t=12ms Client → SSH_MSG_USERAUTH_REQUEST (publickey probe)
t=13ms Server → SSH_MSG_USERAUTH_PK_OK
t=14ms Client → SSH_MSG_USERAUTH_REQUEST (signed)
t=15ms Server → SSH_MSG_USERAUTH_SUCCESS
t=16ms Client → SSH_MSG_CHANNEL_OPEN ("session")
t=17ms Server → SSH_MSG_CHANNEL_OPEN_CONFIRMATION
t=18ms Client → SSH_MSG_CHANNEL_REQUEST ("pty-req")
t=19ms Server → SSH_MSG_CHANNEL_SUCCESS
t=20ms Client → SSH_MSG_CHANNEL_REQUEST ("shell")
t=21ms Server → SSH_MSG_CHANNEL_SUCCESS
t=22ms ──────────────────────── SHELL PROMPT APPEARS
~22ms, ~20+ TCP segments, 3 protocol layers, 1 shared secret never transmitted.