# SSH: What happens behind the scenes Every time you type `ssh user@host`, you trigger a cryptographic ceremony that involves at least 6 distinct protocol phases, 3 layers of encryption negotiation and a minimum of 8 TCP round trips i.e., before a single byte of your shell session is transmitted. Let's tear it apart. ## What SSH actually is? Not just ***secure telnet***. SSH is three protocols stacked on top of each other, all defined in [RFC 4251–4254](https://datatracker.ietf.org/doc/html/rfc4251) | Layer | RFC | Purpose | | --- | --- | --- | | SSH Transport Layer | RFC 4253 | Encryption, key exchange, integrity | | SSH Authentication Layer | RFC 4252 | User identity verification | | SSH Connection Layer | RFC 4254 | Channels, multiplexing, port forwarding | All of this rides on TCP port 22 (one persistent TCP connection) ### The binary packet format ([RFC 4253](https://datatracker.ietf.org/doc/html/rfc4253)) Every SSH message after key exchange looks like this ```shell ┌────────────────────────────── │ uint32 packet_length (4 bytes) │ │ byte padding_length (1 byte) │ │ byte[] payload (variable) │ │ byte[] random_padding (padding_length) │ │ byte[] MAC (variable, optional) │ └────────────────────────────── ``` * `packet_length` = length of everything AFTER the length field itself * Padding is added to make `packet_length + padding_length + 1` a multiple of cipher block size (or 8) * MAC (Message Authentication Code) covers: sequence number + packet content — **not encrypted**, computed over plaintext then appended to ciphertext NOTE: SSH is NOT TLS. It has its own key exchange, its own record format, its own MAC scheme. ## THE FULL CONNECTION LIFECYCLE ### Phase 0: TCP Handshake Before SSH does anything, TCP happens: ```shell Client → SYN → Server Client ← SYN-ACK ← Server Client → ACK → Server ``` 3 packets. Connection established on port 22. Observable ```shell tcpdump -i eth0 'tcp port 22' -nn ``` ### Phase 1: Version Exchange (RFC 4253) First thing both sides do — send a plaintext identification string ```shell Server → "SSH-2.0-OpenSSH_9.3\r\n" Client → "SSH-2.0-OpenSSH_9.3\r\n" ``` * This is **cleartext** — fully visible in Wireshark * Format: `SSH-protoversion-softwareversion [comments]` * Client and server both record each other's version — used later in key exchange hash * If versions are incompatible, connection drops immediately Observable ```shell # Capture the banner nc -v 192.168.1.100 22 # or ssh -vvv user@host 2>&1 | head -20 ``` ### Phase 2: Algorithm Negotiation — KEXINIT (RFC 4253) Both sides send a `SSH_MSG_KEXINIT` packet **simultaneously** (no waiting). This is the "menu" ```shell byte SSH_MSG_KEXINIT (20) byte[16] cookie (random bytes) name-list kex_algorithms name-list server_host_key_algorithms name-list encryption_algorithms_client_to_server name-list encryption_algorithms_server_to_client name-list mac_algorithms_client_to_server name-list mac_algorithms_server_to_client name-list compression_algorithms_client_to_server name-list compression_algorithms_server_to_client ... ``` **Negotiation rule**: client's preference list wins — first algorithm in the client's list that the server also supports is chosen. **Typical modern negotiation result:** | What | Chosen Algorithm | | --- | --- | | Key Exchange | `curve25519-sha256` | | Host Key | `ssh-ed25519` | | Encryption | [`chacha20-poly1305@openssh.com`](mailto:chacha20-poly1305@openssh.com) or `aes256-gcm` | | MAC | (implicit with AEAD cipher) | | Compression | `none` (usually) | Observable ```shell ssh -vvv user@host 2>&1 | grep -E 'kex|cipher|mac|compress' # Shows: # kex: algorithm: curve25519-sha256 # kex: host key algorithm: ssh-ed25519 # kex: server->client cipher: chacha20-poly1305... ``` ### Phase 3: Key Exchange — The Cryptographic Heart (RFC 4253) This is the most important phase. Using **Diffie-Hellman** or **ECDH** to establish a shared secret without transmitting it. **With** `curve25519-sha256` **(most modern):** **Step 1 — Client generates ephemeral keypair:** ```shell client_private_key = random scalar (32 bytes) client_public_key = scalar × G (Curve25519 base point) ``` **Step 2 — Client sends** `SSH_MSG_KEX_ECDH_INIT`**:** ```shell byte SSH_MSG_KEX_ECDH_INIT (30) string Q_C ← client's ephemeral public key ``` **Step 3 — Server computes shared secret + signs everything:** Server has: * Its own ephemeral keypair `(server_priv, server_pub)` * Client's public key `Q_C` Server computes: ```shell K (shared_secret) = server_priv × Q_C = client_priv × Q_S (ECDH magic — same result both sides) ``` Server builds **exchange hash H**: ```shell H = SHA-256( V_C || V_S || ← version strings from Phase 1 I_C || I_S || ← KEXINIT payloads from Phase 2 K_S || ← server's HOST key (public, long-term) Q_C || ← client's ephemeral pubkey Q_S || ← server's ephemeral pubkey K ← shared secret ) ``` Server sends `SSH_MSG_KEX_ECDH_REPLY`: ```shell string K_S ← server host public key string Q_S ← server ephemeral public key string sig_H ← signature of H using K_S (host key) ``` **Step 4 — Client verifies:** ```shell 1. Compute K = client_priv × Q_S (same shared secret!) 2. Recompute H 3. Verify sig_H using K_S 4. Check K_S against known_hosts (TOFU or CA) ``` **Critical insight**: The shared secret `K` is **never transmitted**. Both sides compute it independently. Even if someone recorded all packets, they cannot compute K without either private key. This is **Forward Secrecy**. **Step 5 — Key Derivation (RFC 4253)** From `K` and `H`, six keys are derived using the hash: ```shell IV client→server = HASH(K || H || "A" || session_id) IV server→client = HASH(K || H || "B" || session_id) Key client→server = HASH(K || H || "C" || session_id) Key server→client = HASH(K || H || "D" || session_id) MAC client→server = HASH(K || H || "E" || session_id) MAC server→client = HASH(K || H || "F" || session_id) ``` From this point forward: **all packets are encrypted**. **Step 6 —** `SSH_MSG_NEWKEYS`**: Both sides send this to say "switching to new keys now." One packet each.** ### Phase 4: SSH Authentication Layer (RFC 4252) Now inside encrypted tunnel. Client must prove identity. **Step 1 — Service request:** ```shell SSH_MSG_SERVICE_REQUEST → "ssh-userauth" SSH_MSG_SERVICE_ACCEPT ← server confirms ``` **Step 2 — Auth methods (tried in order):** **Method A:** `publickey` **(most common)** ```shell 1. Client sends probe: SSH_MSG_USERAUTH_REQUEST { username, "ssh-connection", "publickey", FALSE, ← just checking if key is acceptable "ssh-ed25519", public_key_blob } 2. Server responds SSH_MSG_USERAUTH_PK_OK if key is in authorized_keys 3. Client sends actual auth: SSH_MSG_USERAUTH_REQUEST { ..same.., TRUE, ← now actually signing signature of (session_id || request_data) using private key } 4. Server verifies signature → SSH_MSG_USERAUTH_SUCCESS ``` Where is authorized\_keys checked? ```shell ~/.ssh/authorized_keys (default) # or configured via AuthorizedKeysFile in sshd_config # or AuthorizedKeysCommand for LDAP/Vault lookups ``` **Method B:** `password` ```shell SSH_MSG_USERAUTH_REQUEST { username, "ssh-connection", "password", FALSE, plaintext_password ← encrypted by transport layer! } ``` **Method C:** `keyboard-interactive` **(used for MFA/PAM)** Multi-round challenge-response. Used by Google Authenticator PAM, Duo, etc. Observable ```shell ssh -vvv user@host 2>&1 | grep -E 'auth|Authentications' # debug1: Authentications that can continue: publickey,password # debug1: Next authentication method: publickey # debug1: Offering public key: /home/user/.ssh/id_ed25519 ``` ### Phase 5: SSH Connection Layer — Channel Multiplexing (RFC 4254) Authentication passed. Now the **connection protocol** opens logical channels over the single TCP connection. **Opening a channel** ```shell Client → SSH_MSG_CHANNEL_OPEN { channel_type: "session" sender_channel: 0 initial_window_size: 2097152 (2MB) maximum_packet_size: 32768 } Server → SSH_MSG_CHANNEL_OPEN_CONFIRMATION { recipient_channel: 0 sender_channel: 0 initial_window_size: ... maximum_packet_size: ... } ``` **Requesting a PTY (pseudo-terminal):** ```shell SSH_MSG_CHANNEL_REQUEST { channel: 0 request_type: "pty-req" want_reply: TRUE TERM: "xterm-256color" width_chars, height_chars, width_px, height_px terminal_modes: (speed, echo, etc.) } ``` **Requesting a shell:** ```shell SSH_MSG_CHANNEL_REQUEST { channel: 0 request_type: "shell" want_reply: TRUE } ``` **Data flow:** ```shell Client → SSH_MSG_CHANNEL_DATA { channel: 0, data: "ls -la\n" } Server → SSH_MSG_CHANNEL_DATA { channel: 0, data: "total 48\n..." } ``` **Flow control — window adjust:** ```shell When receiver's buffer fills: SSH_MSG_CHANNEL_WINDOW_ADJUST { channel: 0, bytes_to_add: 2097152 } ``` **Key insight:** Multiple channels can run simultaneously — shell + port forwards + X11 — all multiplexed over one TCP connection with independent flow control windows ### Phase 6: Session Teardown ```shell User types exit → shell sends EOF SSH_MSG_CHANNEL_EOF (no more data from this side) SSH_MSG_CHANNEL_CLOSE (I'm done with this channel) SSH_MSG_DISCONNECT { reason_code: SSH_DISCONNECT_BY_APPLICATION description: "logout" } TCP FIN/ACK ``` ## OBSERVABILITY DEEP DIVE ### Tool 1: `ssh -vvv` — Built-in debug ```shell ssh -vvv -o 'LogLevel=DEBUG3' user@host 2>&1 | less ``` Shows every protocol message, algorithm chosen, key used. ### Tool 2: Wireshark / tshark — Packet-level ```shell # Capture SSH handshake (cleartext phase visible) tshark -i eth0 -f 'tcp port 22' -w /tmp/ssh.pcap # Analyze tshark -r /tmp/ssh.pcap -Y 'ssh' -T fields \ -e frame.number -e ssh.message_code -e ssh.kex.algorithms ``` In Wireshark: * Filter: `ssh` * You'll see: `Protocol: SSHv2`, message types like `Key Exchange Init`, `Elliptic Curve Diffie-Hellman Key Exchange Init/Reply`, `New Keys` * After `New Keys`: everything shows as `Encrypted packet` ### Tool 3: `strace` — System call level ```shell # Trace the ssh client strace -e trace=network,read,write -f ssh user@host 2>&1 | head -100 # On server side, trace sshd for a specific connection strace -p $(pgrep -f "sshd: user") -e trace=read,write ``` ### Tool 4: eBPF / bpftrace — Zero overhead production observability ```shell # Trace all connect() calls to port 22 bpftrace -e ' tracepoint:syscalls:sys_enter_connect / ((struct sockaddr_in *)args->uservaddr)->sin_port == 5632 / { printf("SSH connect: pid=%d comm=%s\n", pid, comm); }' # Watch SSH data writes (channel data) bpftrace -e ' kprobe:tcp_sendmsg /comm == "ssh"/ { printf("ssh write: pid=%d bytes=%d\n", pid, arg2); }' ``` ### Tool 5: `/proc` and kernel crypto — See what's happening inside ```shell # See active SSH connections ss -tnp | grep :22 # Check which crypto is loaded cat /proc/crypto | grep -A5 'curve25519\|chacha20' # OpenSSL timing of key exchange (benchmark) openssl speed curve25519 ed25519 ``` ### Tool 6: `sshd` debug mode — Server-side visibility ```shell # Run sshd in debug mode on alternate port (non-disruptive) /usr/sbin/sshd -d -p 2222 # Full verbose /usr/sbin/sshd -ddd -p 2222 ``` ## THE FULL PICTURE Complete timeline of one SSH connection ```shell t=0ms TCP SYN t=1ms TCP SYN-ACK t=2ms TCP ACK ──────────────────── TCP established t=3ms Server → SSH banner (cleartext) t=4ms Client → SSH banner (cleartext) t=5ms Client → SSH_MSG_KEXINIT t=5ms Server → SSH_MSG_KEXINIT (simultaneous) t=6ms Client → SSH_MSG_KEX_ECDH_INIT (client ephemeral pubkey) t=8ms Server → SSH_MSG_KEX_ECDH_REPLY (host key + sig + server pubkey) t=8ms [Both sides derive 6 symmetric keys] t=9ms Client → SSH_MSG_NEWKEYS t=9ms Server → SSH_MSG_NEWKEYS ──────────────────────── EVERYTHING ENCRYPTED FROM HERE t=10ms Client → SSH_MSG_SERVICE_REQUEST ("ssh-userauth") t=11ms Server → SSH_MSG_SERVICE_ACCEPT t=12ms Client → SSH_MSG_USERAUTH_REQUEST (publickey probe) t=13ms Server → SSH_MSG_USERAUTH_PK_OK t=14ms Client → SSH_MSG_USERAUTH_REQUEST (signed) t=15ms Server → SSH_MSG_USERAUTH_SUCCESS t=16ms Client → SSH_MSG_CHANNEL_OPEN ("session") t=17ms Server → SSH_MSG_CHANNEL_OPEN_CONFIRMATION t=18ms Client → SSH_MSG_CHANNEL_REQUEST ("pty-req") t=19ms Server → SSH_MSG_CHANNEL_SUCCESS t=20ms Client → SSH_MSG_CHANNEL_REQUEST ("shell") t=21ms Server → SSH_MSG_CHANNEL_SUCCESS t=22ms ──────────────────────── SHELL PROMPT APPEARS ``` ~22ms, ~20+ TCP segments, 3 protocol layers, 1 shared secret never transmitted.