Martin's Blog

Finding the NetScaler SAML Crash in nsaaad

NetScaler’s SAML problem led me into nsaaad, the authentication daemon. In build 14.1-73.30, a routine identified by embedded source strings as canonicalize_data puts namespace-prefix pointers into a sixteen-entry stack array. It accepts sixteen entries, but the code that consumes them expects a NULL terminator. There is no room for that terminator when the array is full.

The next pointer read comes from the saved stack canary. The diagnostic code treats it as a string pointer, and the controlled test ended in strlen with SIGBUS, a core, and one Pitboss restart of nsaaad.

That test started at the daemon’s internal message boundary. The disposable appliance had no AAA licence, so I could not reproduce delivery through a network-facing Gateway or AAA SAML virtual server. The static path reaches the same operation, but the external route remains untested locally. The observed impact is one daemon crash and restart; the run did not establish code execution, restart exhaustion, or an appliance reboot.

Update, 4 October 2026: Citrix’s CTX697174 identifies CVE-2026-88779 and lists fixed builds. The reverse engineering and controlled run described here took place on 3 October. The bulletin records its initial publication on 3 October PST.

Where CVE-2026-88779 fits

CTX697174 describes a memory overflow causing denial of service on customer-managed appliances configured as a SAML service provider or identity provider. Citrix rates it High, with CWE-119 and a CVSS v4.0 score of 8.7. The published vector gives confidentiality and integrity impact as None:

CVSS:4.0/AV:N/AC:L/AT:N/PR:N/UI:N/VC:N/VI:N/VA:H/SC:N/SI:N/SA:N

The bulletin’s configuration prerequisites are a SAML action or SAML IdP profile. This local analysis supplies a candidate mechanism: both parser paths converge on the daemon’s canonicalization routine, and the internal test reproduces a process-level denial of service.

Citrix has not identified canonicalize_data, PrefixList, or this stack layout as the root cause in that bulletin. I have not compared the fixed 14.1-73.41 binary. The evidence supports the mapping to CVE-2026-88779, but a fixed-build comparison would independently confirm or disprove it.

There is also a useful distinction in the terminology. Citrix calls its issue a memory overflow. The local failure demonstrated here is an out-of-bounds pointer read: the diagnostic loop reads the saved canary without overwriting it.

The builds I inspected

The release archives were retained unchanged:

build-14.1-73.30_nc_64.tgz
  SHA-256 acd260b4382a33e8c05ba604b5b9a80bb2c38c0bf161d8ba653aa4b8118c22d6

build-14.1-73.37_nc_64.tgz
  SHA-256 e0151fd7c44c6d1fa99872a1e01912e92a0df1b611cf9c3c91fc854701293a06

The relevant extracted binaries were:

14.1-73.30 /netscaler/nsaaad
  size       666,488 bytes
  SHA-256    07407c37cfbf9ee4027769549c222f9aa6207fff93746ebdc3ea40f60f6d944e

14.1-73.37 /netscaler/nsaaad
  size       666,488 bytes
  SHA-256    bcf23e59136ad4fa3805dc7f74c57ac64e4bbc4e058f9e35e4a4f569a8da380c

Both nsaaad files with .ns_sig removed
  SHA-256    4764127b4d8dc25681e3c67caf1446be956d23af70e3c8cd890bc909b4d64c84

14.1-73.30 /netscaler/nsppe
  SHA-256    6ce73d12c24d2c23e77fbc8234091dd069b610b658455a7dc18b087ee8890f85

Only 70 raw bytes differ between the two nsaaad files, all inside the vendor’s .ns_sig section. With that section removed, the files are byte-identical. The bundled libxml2.so.2.13.9 files are also identical after removing .ns_sig, with SHA-256 d3a6adfc39bd7d248cf1c868b738b037625ffcc81889eb8fc0cb362c7813a866.

So the move from 73.30 to 73.37 did not change this routine or the bundled libxml2 implementation. Only 73.30 was run in the controlled test.

I used Binary Ninja 6.0.10601 to reconstruct the call path. Its C-like output is an approximation: inferred names, types, signatures, and structured control flow need checking. The stack layout and loop conditions below were checked against the retained x86-64 disassembly for the exact 73.30 binary.

Following the PrefixList

SAML XML signatures can contain an InclusiveNamespaces PrefixList. The service-provider and identity-provider paths read it in separate packet-engine parsers, then converge before entering nsaaad:

SAMLResponse (service provider) or signed AuthnRequest (identity provider)
        |
        v
nsppe SAML parser                              [static analysis]
  SP:  sub_a54a20 -> sub_a140e0
  IdP: sub_a002a0
        |
        | PrefixList pointer and byte length
        v
sub_7dfe10 -> internal AAAD canonicalization request
        |
---------------- internal boundary ----------------
        |
        v
nsaaad -> canonicalize_data at 0x429750         [runtime test begins here]
        |
        v
unterminated pointer array -> diagnostic string formatting -> SIGBUS

The SP parser at 0xa54a20 and the IdP parser at 0xa002a0 both enforce a byte-length limit of less than 0x201, or 513 bytes. Neither enforces a maximum number of space-separated entries. The common request builder at 0x7dfe10 carries the parsed PrefixList into the internal canonicalization request, which nsaaad dispatches to canonicalize_data.

Canonicalization is part of XML signature verification. In the reviewed control flow, the request is submitted before signature acceptance completes. That supports an unauthenticated path to the operation, subject to the earlier parsers accepting the XML structure. It remains a static reachability conclusion: the local test began after those packet-engine parsers.

The sixteen-entry error

The relevant behaviour reduces to this explanatory C. It is a sketch of the reconstructed control flow, not recovered vendor source:

char *prefixes[16] = { 0 };
size_t count = 0;

while (prefix_bytes_remaining > 0) {
    char *copy = copy_next_space_delimited_entry();

    if (count == 16) {           /* reached only when parsing entry 17 */
        free(copy);
        return -1;
    }

    prefixes[count++] = copy;    /* entries 1 through 16 are accepted */
}

for (size_t i = 0; prefixes[i] != NULL; i++)
    log_debug("namespace prefix %zu is %s", i + 1, prefixes[i]);

xmlC14NDocDumpMemory(doc, ..., prefixes, ...);

Fifteen entries leave prefixes[15] as NULL, which stops the diagnostic loop. A seventeenth entry reaches the rejection branch. Exactly sixteen entries fill the array and end the input before that branch runs again.

The important question is what sits immediately after the array. In this build, it is the saved stack canary.

The prologue saves the canary at rbp-0x30, and the pointer array begins at rbp-0xb0. These are instruction excerpts with the byte columns and some intervening instructions omitted; comments explain the relevant operands:

429779  mov    0x27a0e0(%rip),%rax      # 6a3860 <__stack_chk_guard@FBSD_1.0>
429780  mov    %rax,-0x30(%rbp)        # saved canary
...
4297b4  movaps %xmm0,-0xb0(%rbp)      # start of cleared pointer array

The distance is 0x80 bytes, exactly sixteen 64-bit pointers. The parser’s check and store are:

429839  cmp    $0x10,%rcx
42983d  je     429a57                 # reject entry 17
429843  mov    %r13,-0xb0(%rbp,%rcx,8)
42984b  add    $0x1,%rcx

The diagnostic loop has no independent count. It keeps loading pointers until one is zero:

4298a0  mov    -0xb0(%rbp,%rbx,8),%r10
4298a8  add    $0x1,%rbx
4298ac  test   %r10,%r10
4298af  je     429924
...
4298d5  push   %r10                   # %s argument
4298d7  push   %rbx
4298d8  call   4125a0 <dprintf_pipe@@Base>

After sixteen real entries, the next indexed load is -0xb0 + 16 * 8, or rbp-0x30. The observed canary was non-zero, so it passed the loop’s NULL check and became the next string argument.

If that diagnostic call did not fault, the same unterminated array would be passed to xmlC14NDocDumpMemory. The observed failure happened in the earlier diagnostic call.

What happened in the lab

The controlled run took place on 3 October against a disposable, host-only 14.1-73.30 appliance. Its running /netscaler/nsaaad hash matched the 73.30 binary above. The restored guest clock was still set to 27 July, which explains the earlier dates in the appliance logs.

The lab could not enable AAA: the appliance returned Feature(s) not licensed, and authentication virtual-server creation returned Feature not licensed [AAA]. I did not bypass the licence check. The prepared external SAML test was therefore not executed against an HTTP endpoint.

Instead, the controlled run sent canonicalization requests directly to the daemon over guest loopback. The three cases differed only in the number of prefix entries and ran in this order:

Prefix entriesObserved responseDaemon result
1547 bytesPID 717 stayed alive; the log reached “successfully canonicalized”; no new core
1740 bytesPID 717 stayed alive; no new core
16EOF, zero bytesPID 717 died with SIGBUS; a core was created; Pitboss started PID 2394

The first two cases provide useful controls. A valid shorter list completed canonicalization, and the longer list returned a response without killing the daemon, consistent with the static rejection branch. The full array was the case that failed. Its log printed prefixes 1 through 16 and stopped before the success message.

Pitboss recorded these message excerpts, with timestamps and unrelated fields omitted:

proc nsaaad (717) SIGNALED
proc nsaaad (717) EXITED with status 0x8a
Restarting process nsaaad old pid (717)
New pid (2394) for (nsaaad) restarts (1)

The core was /var/core/2/nsaaad-717, 25,837,568 bytes, with SHA-256 c4b55a8dd1016d980ffe9bc7241a1d610e0954abb739efdf0101a780579ded5f. It was analysed on the appliance and not copied off it, since a process core can contain credentials and unrelated secrets.

GDB reported SIGBUS at strlen+31. The relevant fault-frame excerpt was:

Program terminated with signal SIGBUS, Bus error.
#0  0x00000008048877cf in strlen () from /lib/libc.so.7
...
rdi            0x94c4ee49e4769e33  -7726789059527926221
rcx            0x94c4ee49e4769e30  -7726789059527926224
r12            0x94c4ee49e4769e33  -7726789059527926221
=> 0x8048877cf <strlen+31>: mov    (%rcx),%rax

The original string argument in rdi was 0x94c4ee49e4769e33, also present in r12. The faulting load used rcx, which held that address aligned down to 0x94c4ee49e4769e30. The core therefore shows both the canary-derived string argument and the aligned address used by the failing memory access.

Frame 4 returned to 0x4298dd, immediately after the prefix diagnostic call. Its sixteen populated pointer slots occupied rbp-0xb0 through rbp-0x38. The next slot, rbp-0x30, contained 0x94c4ee49e4769e33: the saved canary and the same value passed to strlen. The backtrace ran through dprintf_pipe and vasprintf_l before reaching strlen.

This ties the process death to the out-of-bounds sentinel walk. It does not depend on guessing from the signal alone.

The wait status is consistent with the core too. FreeBSD’s wait-status definitions use 0x80 as the core flag and the low seven bits for the signal. For 0x8a, that leaves signal 10, which its signal definitions identify as SIGBUS.

The noncanonical pointer is consistent with an amd64 general-protection fault. The capture did not record the kernel trap or si_code, so I cannot claim a directly observed BUS_OBJERR classification. SIGBUS, the core, the registers, and the daemon restart were observed.

After the run, the snapshot was restored. Checks showed AAA disabled, no authentication virtual server, no SAML action or IdP profile, and no retained nsaaad core. The appliance was then powered off.

The username record belongs to CVE-2026-88771

The command-bearing username record considered alongside this crash belongs to the separate CVE-2026-88771 investigation. I described the login-log command-injection path in that post: attacker-controlled login text reaches a later log-processing shell path. It is not evidence for this SAML canonicalization failure.

The distinction matters when reading incident logs. A command-bearing username can indicate an exploitation attempt; confirming execution requires evidence that the command ran. A nsaaad crash shows a process failure; by itself, it does not establish the cause or code execution. Putting the two records next to each other does not establish a shared vulnerability or an exploit chain.

What remains unproven

The internal test and core establish the daemon’s failure on 73.30. The binary comparison establishes that 73.37 still contains the same executable content. Static analysis connects both SAML roles to that operation.

Two gaps remain: delivery through a licensed, SAML-configured Gateway or AAA virtual server, and comparison with the fixed 73.41 build. Neither was completed here. The first would confirm the external path locally; the second would test the proposed CVE mapping against the shipped change.

I observed one nsaaad death and one restart. There is no local result for restart exhaustion, appliance reboot, remote code execution, or persistence.

Upgrading and reading the logs

Citrix lists these fixed builds in CTX697174:

Product lineFixed build
NetScaler ADC and NetScaler Gateway 14.114.1-73.41 or later
NetScaler ADC and NetScaler Gateway 13.113.1-64.28 or later
NetScaler ADC 14.1-FIPS14.1-73.41 FIPS or later
NetScaler ADC 13.1-FIPS and 13.1-NDcPP13.1-37.282 or later

The September 14.1-73.37 and 13.1-64.23 builds are earlier than those fixes. For deployments meeting the SAML prerequisite, that September upgrade is not sufficient for CVE-2026-88779.

For exposure assessment, compare the running and saved configuration for SAML actions and IdP profiles. For incident review, correlate nsaaad deaths, 0x8a statuses, cores, and Pitboss restarts with SAML authentication activity. Preserve the relevant logs, build information, and core when operationally possible.

A matching SAML configuration establishes exposure under the vendor’s guidance, not exploitation. A 0x8a core is also not a unique signature of this bug. The useful evidence in this run was the combination: the boundary controls, the failing diagnostic call, and the canary-derived string pointer in the core.

#NetScaler #SAML #CVE-2026-88779 #Nsaaad #Denial of Service #Reverse Engineering #Incident-Response