When the Handshake Fails: Diagnosing Schannel from Its Own Evidence

A failed TLS handshake on Windows rarely explains itself. The application says the connection was forcibly closed, or that a secure channel could not be established, and the person on call starts guessing: a certificate, a firewall, an update, the other team's server. Yet Schannel, the Windows component that carries out Transport Layer Security (TLS) for the operating system and the programs built on it, leaves precise evidence behind every failure. It records numbered events, it receives and sends numbered alerts, and it returns specific status codes to the program that called it. This article is a method for reading that evidence in the right order, including one measured surprise: two quite different root causes can produce byte-for-byte identical errors on the client, and only a second, independent TLS implementation tells them apart.

Three vocabularies for one failure

The same failed handshake is described three times, in three different languages, by three different layers. The application speaks first and least precisely: a database driver reports operating system error 10054, "an existing connection was forcibly closed by the remote host"; an HTTP client reports that the secure connection could not be established; Windows curl reports error 35. None of these name a cause.

Underneath, the program received a status code from the Security Support Provider Interface (SSPI), the Windows API through which programs call Schannel. These codes are specific and hexadecimal, such as SEC_E_UNTRUSTED_ROOT (0x80090325) or SEC_E_ALGORITHM_MISMATCH (0x80090331). Good applications log them; many swallow them.

The third vocabulary is the TLS protocol itself. When one side gives up, it sends the other a fatal alert, a small message carrying a number defined by the TLS standard, RFC 8446: 40 for handshake_failure, 48 for unknown_ca, 70 for protocol_version, 116 for certificate_required, and so on. Schannel writes the alert number into the System event log. Microsoft publishes a mapping between alert numbers and SSPI codes, and the first discipline of diagnosis is to translate every failure into one row containing all three before forming any theory. The mapping has a trap worth knowing: several different alerts, and any alert Schannel does not specifically recognise, all collapse into the same code, SEC_E_ILLEGAL_MESSAGE (0x80090326). Seeing that code tells you that the peer objected, not why.

Direction first: which side raised the alarm

Before asking what went wrong, ask where it was noticed. Schannel uses two different events for alerts. Event 36888, "a fatal alert was generated", means this machine detected the problem and told the peer. Event 36887, "a fatal alert was received", means the peer detected the problem and told this machine. Microsoft's description of 36888 is explicit that the computer logging it is the one that found the error condition.

That distinction saves hours, because the reason is logged on the side that generated the alert. A client that shows only 36887 with alert 40 has learned nothing except that the server was unhappy; the server's own log is where an event such as 36874, "none of the cipher suites supported by the client application are supported by the server", or 36870, a fatal error accessing the server credential's private key, explains why. Reading the client harder will not reveal what only the server knows.

Microsoft also cautions that 36887 does not always mean an application problem: it can appear when something sends plain HTTP to an HTTPS port or when a scanner probes the server. A burst of received alerts with no user complaint is often noise; a received alert that matches a user-visible failure, by time and target, is a lead. Timestamps are the join key, so the first action in any incident is to reproduce the failure once and write down the exact time and target name.

On current builds the received-alert event also names the program involved. In the October 2026 tests described below, each 36887 event on the client ended with the SSPI client process name and process ID, for example curl with its PID. That small detail matters on a busy host, where dozens of programs share Schannel: it lets you separate the one failing caller from background services that happen to fail at the same minute, and it tells you whose configuration, and whose code, to examine next.

One alert, two causes: a measured example

To see how little the client side can know, it helps to break a handshake on purpose in two different ways and compare the evidence. In October 2026, on Windows 11 version 26H2 with Schannel's default settings, Windows curl.exe (which uses Schannel) was pointed at small local test servers built with OpenSSL 3.5.9, an independent TLS implementation shipped with Git for Windows. No machine settings were changed.

In the first case the test server accepted only one key exchange group, the hybrid post-quantum group X25519MLKEM768, which Schannel does not offer by default. In the second case the server accepted only one TLS 1.2 cipher suite that could not be used with its certificate, so no suite could ever be agreed. These are unrelated root causes: one is a missing key exchange group on the client, the other is a broken server configuration. On the Schannel client they looked identical. Both produced curl's SEC_E_ILLEGAL_MESSAGE (0x80090326) and both logged event 36887 with alert 40. Only the server logs differed, reporting "no suitable key share" in one case and "no shared cipher" in the other.

A third case, a server that accepted only TLS 1.1, produced a different and more informative row: SEC_E_UNSUPPORTED_FUNCTION (0x80090302) and alert 70, protocol_version. On current Windows, where TLS 1.1 is disabled by default, that combination points straight at a version mismatch. The general lesson is that alert 40 means "no acceptable parameters" without saying which parameter, so a client-side investigation that stops at alert 40 is guessing.

The reason is built into the protocol. RFC 8446 defines handshake_failure as the sender being unable to negotiate an acceptable set of security parameters given the options available, and it reserves a more specific alert, insufficient_security, only for the case where a server requires parameters stronger than the client supports. A server that finds no common group and a server that finds no common cipher suite have both failed to negotiate acceptable parameters, so both may legitimately send the same number. The alert is a verdict, not an explanation, and the explanation stays on the side that wrote it.

A root-cause map built from Schannel's events

Once the generating side is known, its own events usually name the cause. Microsoft's reference for Schannel events lists them by number, and a small set covers most real incidents. Reading them as a map, from symptom to check, turns a list of numbers into a procedure.

Two causes deserve a note because they are not cryptographic at all. Event 36870 on a server typically means the service could not load the private key of its certificate, which is a permission or key-storage problem rather than a protocol one; Microsoft's LDAP over TLS guidance checks it with certutil -verifykeys. And when several valid certificates sit in the local computer store, Microsoft notes that Schannel may simply pick the first valid one it finds, which may not be the one intended. Size can matter too: Schannel limits how large a fragmented handshake message it accepts, by default 0x8000 bytes for a client and 0x4000 bytes for a server without client authentication, so an unusually large certificate chain can fail for reasons that have nothing to do with trust.

An order of checks that avoids rabbit holes

Most wasted time in TLS incidents comes from checking the right things in the wrong order, for example spending an hour on a proxy before noticing that the server's certificate expired. A fixed order keeps the investigation honest because each step either finds the cause or removes a whole class of causes.

Start by reproducing the failure once, with a timestamp and the exact target name. Next, find the side that generated the alert. Then compare what both ends allow: protocol versions, cipher suites and key exchange groups, reading both the local list and any policy list, since a policy list replaces the local one. Only then move to the certificate on the presenting side: is it present, can the service identity use its private key, does it carry the right extended key usage (Server Authentication or Client Authentication), does its name match, and are there several candidates? After that, build the chain on the other side, where trust is actually decided; Microsoft's LDAP guidance does this by exporting the certificate and running certutil -v -urlfetch -verify on the peer. Client certificate problems come next, and network devices such as proxies or inspection appliances come last, once both endpoints have been cleared. When the endpoints agree that all is well and the handshake still fails, a packet capture of the Client Hello and Server Hello is the tie-breaker.

Two details in the overlap step catch experienced engineers. The first is that the list you read back with a cmdlet may not be the list in force: on a machine that receives Group Policy or device management policy, the policy's priority list replaces the local one, and the database troubleshooting guidance from Microsoft points to the local list under the Cryptography configuration key as only one of the places to compare. The second is that overlap problems can depend on the pairing of Windows versions. Microsoft describes handshake failures between different Windows versions that both offered TLS_DHE cipher suites, resolved by removing those suites, which is a reminder to record the operating system build of both ends in the evidence row.

Splitting client from server with a second TLS stack

The measured example points to the most useful trick in Schannel diagnosis: ask the same question with a TLS implementation that does not share Schannel's settings. OpenSSL's s_client tool connects to a server and, with the -brief option, prints the protocol, cipher suite and, for TLS 1.3, the negotiated group; options such as -tls1_2, -tls1_3, -groups and -servername let you pin exactly what is offered. Its counterpart s_server stands up a test server with exactly the parameters you choose.

Two probes answer the decisive question. Probe the failing server from OpenSSL: if OpenSSL also fails with the same alert, the server's configuration or certificate is the cause, and no client change will help. If OpenSSL succeeds while the Schannel client fails, the server can do the job and the client's policy is the difference: a missing group, a disabled version, a suite removed by policy. The reverse probe confirms it: point the Schannel client at an OpenSSL test server pinned to one version, suite or group, and watch which pins it can and cannot meet. Neither probe changes any machine setting, which makes the method safe to use on production clients.

openssl s_client -connect host.example:443 -servername host.example -tls1_3 -brief
openssl s_client -connect host.example:443 -servername host.example -tls1_2 -brief
openssl s_client -connect host.example:443 -groups X25519MLKEM768 -tls1_3 -brief

Collecting more evidence without making a mess

Sometimes the default error-only logging is not enough and success or warning events are needed to compare a working connection with a failing one. Schannel's logging level is the EventLogging value under HKLM\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL, with levels from 0 to 7; Microsoft notes that a reboot is required after changing it. Raise it only for the window you need, on the machines involved, and set it back afterwards, because success logging is voluminous.

Collection itself should be read-only and precise. Microsoft's own troubleshooting guidance uses the wevtutil tool with an XPath filter to pull one Schannel event ID from the last day, which is easy to adapt to any ID in the root-cause map. The structured fields of each event, such as the alert number, the target name and the calling process, are what should travel into a ticket, not a screenshot of the Event Viewer.

wevtutil qe system /c:1 /rd:true /f:text /q:"Event[System[Provider[@Name='Schannel'] and EventID=36887 and TimeCreated[timediff(@SystemTime) <= 86400000]]]"

AI agents: diagnosis without dangerous fixes

Agent hosts make TLS failures both more frequent and harder to see. An AI agent built on a Windows runtime that uses Schannel calls many endpoints it discovered at run time, and agent frameworks often report a failed handshake as a generic tool error and simply retry. The retries multiply the 36887 events, bury the first failure, and hide the one detail that mattered: which target, which alert. The first improvement is plain engineering: have the agent host record the SSPI status code, the alert number when available and the target name for every failed connection, so a human or a monitoring rule can read the row instead of the noise.

The second concerns what an agent is allowed to do about it. Every read-only step in this method is safe to automate: an agent can query events with wevtutil, list cipher suites and groups, verify a certificate with certutil, and run OpenSSL probes, then propose a cause and a fix. What it must not do is apply the fixes that make errors vanish by removing protection. Turning off certificate validation, whether through a validation callback that always returns success or through Schannel's manual-validation flag without performing equivalent checks itself, hides the 36882 and 36884 class of failures by accepting any server. Re-enabling an old protocol for the whole machine to satisfy one endpoint widens exposure for every program on it. A practical rule for agent tooling is that diagnostics are automated and changes to trust or protocol policy go through a human review with the evidence row attached.

Glossary

SSPI

Security Support Provider Interface, the Windows API that programs use to ask Schannel to secure a connection.

Fatal alert

A short TLS message with a standard number that one side sends before abandoning the handshake.

SEC_E_ILLEGAL_MESSAGE

The SSPI status 0x80090326, which Schannel returns for several different alerts, including handshake_failure.

Key exchange group

The mathematical setting, such as curve25519 or a hybrid post-quantum group, used to agree a shared secret.

Extended key usage

A certificate field listing what the certificate may be used for, such as Server Authentication or Client Authentication.

OpenSSL s_client

A command-line TLS client from the OpenSSL project that reports what a server negotiates.

Second TLS stack

An independent TLS implementation used to repeat a test without sharing the first implementation's settings.

Further reading

1. Schannel error codes for TLS and SSL alerts (Microsoft Learn)
2. An existing connection was forcibly closed (OS error 10054) (Microsoft Learn)
3. RFC 8446: The Transport Layer Security (TLS) Protocol Version 1.3 (IETF)
4. openssl-s_client, OpenSSL 3.5 manual (OpenSSL Project)