13. Investigating a Suspected RCS Deployment

The first impulse on finding an RCS login page is often to try the default administrator password. That is precisely the wrong first action.

Logging in is not passive. RCS creates and invalidates session state, writes audit records, updates recent activity, and may send a “used on another machine” logout push to an operator already connected as the same user. A browser session can reveal that investigators are present. Restoring a backup, opening a task workflow, deleting a suspicious record, or compacting MongoDB changes far more.

The correct opening is preservation. RCS may hold years of communications, locations, credentials, photographs, files, and relationship data about people who did not consent to collection. It may also be a compromised system whose operators, logs, binaries, and access controls can no longer be trusted. The investigation has to preserve technical evidence while minimizing further exposure and harm.

This chapter provides a defensive workflow for three discovery contexts: an RCS backend, collector or infrastructure node, and suspected endpoint. It does not authorize access to third-party systems, provide implant operating instructions, or assume that every match belongs to HackingTeam. The detection confidence model from Chapter 12 remains the entry gate.

Define the incident before touching it

“RCS found” can describe several different incidents:

• an organization discovers an operator database in its own environment;
• a provider identifies a collector or anonymizer on infrastructure it controls;
• a responder finds a suspected RCS component on a person’s device;
• leaked or seized storage contains a dormant installation;
• an authorized surveillance owner suspects its RCS platform was itself compromised; or
• a researcher finds a public service that resembles RCS but has no authority to interact with it.

The legal authority, containment options, victim risk, and evidence sources differ sharply. Before interaction, record who authorized the work, which systems and accounts are in scope, what data may be collected, who may see it, and which actions are prohibited. A public-facing candidate is not an invitation to test credentials, enumerate targets, or download evidence.

Establish a restricted evidence repository at the beginning. Ordinary malware tickets and team chat are inappropriate places for victim communications, photographs, location histories, credentials, target names, or customer identities. Use identifiers in working notes and keep the mapping under separate control.

The initial incident statement should separate three propositions:

1. Platform identification: how strongly the artifacts identify RCS.
2. Platform compromise: whether somebody gained unauthorized control of the RCS operator system.
3. Target compromise: which people or devices may have been monitored.

These are independent questions. A backend can exist without a known endpoint sample; a recovered implant does not identify its operator; a platform compromise can expose victim data without changing any endpoint.

The first decision tree should remain deliberately conservative:

candidate RCS artifact or service
        │
        ├── no authority over system ──> preserve passive/public evidence only
        │                               document lead; seek lawful authority
        │
        └── authorized control
                │
                ├── backend/database ──> volatile state → disks/volumes → BSON
                │
                ├── collector/edge ───> connections → config/logs → peer map
                │
                └── endpoint ─────────> safeguarding decision → memory/disk
                                        → inert artifact analysis

Each branch begins with preservation, but the harm calculus differs. A backend may contain data about many people and credentials for other components. An edge node may be the only observable point in a larger topology and may still hold buffered evidence. An endpoint belongs to a person whose immediate safety, accounts, contacts, and legal position may be affected by both surveillance and the response.

Record the initial classification as a hypothesis. A machine named “collector” can be a research copy; a target-scoped database can be a backup; an endpoint artifact can be inactive. The classification should change when evidence changes, without erasing the reasoning that led to the earlier state.

Preserve volatile state before restart

If the host is under authorized control, record its time, timezone, network identity, and clock offset. Capture processes, services, listeners, connections, routes, firewall policy, logged-on users, containers, mounts, open files, and available application status. Acquire memory when lawful and proportionate. Record tool versions, hashes, timestamps, failures, and known side effects.

Do not restart rcs-db before volatile capture. Its WebSocket manager and session lookup path keep important sockets and caches in memory; a restart drops live connections and changes what can be learned about connected operators. Human session documents remain in MongoDB, while startup explicitly clears server-component sessions. Restart also changes service, process, and log timelines. Similarly, do not use the console simply to make the current state easier to read.

Preserve passive network telemetry and the currently presented TLS chain. Record the 443/444 database pair, internal worker and MongoDB listeners, collector and controller ports, peer addresses, and any archive, connector, translation, or anonymizer destinations visible from authorized local state. Do not contact those peers to “see what answers.” A remote system may be outside scope, and a probe can alter evidence or endanger monitored people.

Containment is a risk decision, not an automatic cable pull. Continued connectivity may permit evidence deletion or ongoing surveillance; abrupt isolation may trigger adversary action, prevent capture of volatile state, or interrupt an authorized safety operation. The response lead should document the decision, the victim impact considered, and the exact containment change.

A first-response sequence

The order below is a dependency sequence, not a universal clock. Teams should adapt it to safety, law, and local forensic procedure without reversing the principle that volatile and easily destroyed evidence comes first.

Chain of custody needs to include derived artifacts, not only disk images. Record the source image and command for MongoDB dumps, collection inventories, certificate fingerprints, memory-derived process lists, decoded log exports, and timelines. Hash outputs where the format is stable and preserve tool logs when it is not. Timezone conversion, redaction, and deduplication are analytical changes and should be reproducible.

The response log should also record failed actions. An unreadable shard, absent volume, rejected query, or certificate that cannot be parsed is part of the evidence boundary. Quietly substituting a later backup or regenerated container can make the case appear complete while erasing the original gap.

Acquire the whole system

Prefer bit-for-bit images of disks and volumes before logical export. On an original Windows backend, preserve the entire C:\RCS\ tree rather than the files that already look interesting. On Linux or a reconstruction, preserve the complete application root, configuration, container definitions, image digests, writable layers, and named volumes.

At minimum the acquisition should include:

• configuration and certificate directories;
• database router, shard, configuration-server, and journal files;
• logs and error logs;
• backups and attached or remote backup media;
• worker staging and failed-decode material;
• service, registry, firewall, scheduled-task, and installer state; and
• dependency versions and application source or packaged code.

The directories named cores, exploits, or similar remain inert evidence. Do not build, repair, or execute their contents. Hash files on acquisition and analyze copies in an environment that cannot reach production, historical RCS infrastructure, or ordinary networks.

Container acquisition needs more than copying a bind-mounted directory. The reconstruction demonstrates how database data, configuration, and logs can live in separate named volumes while the image supplies code. Preserve the compose file, environment, resolved mounts, image identifiers, network configuration, container metadata, and every relevant volume. Recreating a container from its image before preserving the writable state can destroy the very differences under investigation.

Preserve MongoDB as BSON and as topology

RCS depends on a sharded MongoDB layout. A useful acquisition preserves data, indexes, collection options, shard keys, and the configuration database. JSON exports alone are inadequate: they can erase BSON types, ObjectIds, dates, and binary distinctions that matter to both operation and provenance.

Inventory before opening content. Record databases, collection names, counts, indexes, collection statistics, shard membership, and shard keys. Then preserve the global collections and every target-scoped family:

• evidence.<target_id>;
• aggregate.<target_id>;
• grid.<target_id>.files; and
• grid.<target_id>.chunks.

Also preserve the worker’s grid.evidence staging bucket, application users, groups, sessions, signatures, items, entities, alerts, audit, status, backup, collector, connector, injector, queue, and topology collections. Some are volatile or capped. Their absence in a later logical export may not mean they were absent at discovery.

The companion query catalogue intentionally projects metadata rather than secrets or evidence bodies. User queries omit password hashes. Session queries omit cookies. Signature queries return scopes without values. Item queries omit agent log and configuration keys. Evidence queries begin with counts, types, identifiers, and times. The objective is to establish the system’s shape before exposing its most sensitive content.

For compatibility evidence, all 24 catalogue blocks were evaluated in the MongoDB 2.6 shell against a network-disabled disposable synthetic clone. Eight targeted assertions covered version, BSON type matching, aggregation, access anomalies, sessions, GridFS, and agent resolution; the rcs and config database hashes were unchanged before and after. This validates the documented syntax and fixture paths, not every malformed value or operational side effect that a real acquisition may contain.

Run those queries on a forensic clone whenever possible. Even reads can affect surrounding access logs, cache state, or filesystem access times. Record the shell version and exact commands. Never run a “helpful” repair, reshard, compact, restore, or statistics recalculation against original evidence.

Reconstruct the investigation graph

The items collection stores operations, targets, factories, and deployed agents in one polymorphic collection. The _kind field selects the type, and the path array records ancestry. User and group membership is denormalized through user_ids and group_ids.

Build an operation → target → agent map from a clone while omitting agent cryptographic keys from routine output. Record open, closed, deleted, and uninstalled flags separately. Factories and agents can share an ident, while the deployed instance adds an instance value and is commonly named from the factory with a numeric suffix.

Check for:

• items with missing or empty user_ids;
• paths pointing to objects that no longer exist;
• descendants whose users differ from their operation;
• objects referenced by groups but not visible through expected membership;
• target-scoped collections without a matching target;
• evidence whose aid does not resolve to an agent; and
• the same object appearing in inconsistent operation paths.

These are leads, not automatic proof of intrusion. RCS rebuilds access control through asynchronous group callbacks. A crashed task, incomplete cascade, version migration, direct database edit, lab fixture, or product bug can leave similar residue. Chapter 10 shows that attackers can also exploit those gaps. The timeline and surrounding audit, session, and log evidence determine which explanation is strongest.

Preserve empty-access objects even if the console cannot display them. The lab found that group membership controls ordinary listing and deletion, so an operation with no groups can become invisible to normal operator workflows while remaining in MongoDB. Direct collection inventory is more complete than the application UI.

Reconstruct the evidence timeline

Evidence has at least two primary clocks. da records acquisition time from the evidence header; dr records receipt by the backend. GridFS adds upload time. Worker staging adds its metadata time. Alerts, audit documents, sessions, component status, filesystem metadata, and backup directories add more.

Do not collapse those clocks into one “event time.” A large difference between acquisition and receipt may describe delayed synchronization, clock error, import, replay, migration, or tampering. Compare time distributions per agent and type. Record the host and database clock offsets before interpreting seconds as precise chronology.

Begin with metadata counts and type histograms. Compare raw counts and binary lengths with cached statistics on agents, targets, and operations, but do not repair discrepancies. In this snapshot, wrong BSON types in a shard key can produce apparent update success without persistence; deletion callbacks can fail; target deletion drops entire collections; and connectors can remove evidence after export. A mismatch is evidence about system history.

The aid field should normally be a string. A BSON ObjectId or other type can indicate direct insertion, a migration, a fixture, or manipulation. It is not by itself proof that evidence is false. Likewise, an evidence document can reference GridFS content through _grid and _grid_size; investigators must verify the corresponding file and chunks without assuming that metadata deletion removed binary content or that orphaned chunks are irrelevant.

Worker failure artifacts deserve separate attention. Raw encrypted blobs are normally removed from staging after processing, while decoded failure material can survive under decoding_failed. That material may capture a parsing error, version mismatch, corruption, or hostile input that never became visible to an analyst.

Determine whether the platform was compromised

Apply Chapter 10’s findings to the deployed version and topology rather than copying its severity table unchanged. Establish the exact code, patches, configuration, listener exposure, operating system, and roles held by each suspected account.

The first questions are high value:

• Did any low-privilege operator acquire new privs?
• Were unknown or unusual task types submitted?
• Did restore, archive, move, filesystem, backup, or build tasks occur outside expected maintenance?
• Were files read through temporary or GridFS endpoints?
• Did one user access evidence, agents, or groups outside their investigation?
• Did disabled or demoted users retain active sessions?
• Did component credentials or signatures change?
• Did an archive node bootstrap unexpectedly or communicate through an untrusted path?
• Was MongoDB reachable without application authentication?
• Are backend source, certificates, service commands, or configuration files different from the frozen installation baseline?

RCS audit is useful but not authoritative. The logger catches write failures without rolling back the business action, bulk deletion records criteria rather than every evidence identifier, and some callback-driven deletion lacks a direct operator audit event. Audit documents share the database trust boundary and are not cryptographically chained. Compare them with operating system, network, backup, filesystem, and database history.

Sessions require similar care. They contain a snapshot of privilege at login, so current user roles do not necessarily describe the authority of an older active session. Correlate session time and address with password, enabled-state, and privilege changes. Do not expose cookie values in the working timeline.

The absence of an obvious code-execution artifact does not rule out platform compromise. The task evaluation path runs inside a legitimate Ruby service; self-promotion uses an ordinary user update; cross-investigation access uses valid REST routes. Behavioral and data-integrity evidence may be more important than a new executable.

Classify the compromise claim by effect

One incident can occupy several rows, and confidence can differ between them:

The standard is not that every alternative be eliminated absolutely. The report should show which evidence supports the unauthorized explanation, which normal mechanisms could produce a similar artifact, and why the selected conclusion is stronger. When the platform is itself compromised, its audit, account, evidence, and configuration records are mutually corroborating sources—not independent ground truth.

Scope the effect precisely. Reading one foreign evidence object establishes that access path, not control of every target. Worker exhaustion establishes an availability failure, not disclosure. A broad component credential may create potential access without evidence that every route was used. Conversely, verified backend-process or unauthenticated database control should trigger a platform-wide integrity review because both authorities can alter the records used to scope their own impact.

Backend branch

For a backend discovery, prioritize MongoDB, session memory, task and audit state, application source, keys, certificates, and backup history. Map every service that can authenticate as the broad server principal. Determine whether the host is a central database, shard worker, archive, all-in-one system, or research reconstruction.

Do not use the default administrator account to enumerate the system. Besides changing state, the default may have been rotated, trapped, or reused by an operator. If application-level access becomes necessary after preservation, use a separately approved method, create a full interaction log, and assume the RCS audit record is only one copy of that log.

Backups are part of both recovery and exposure. Inventory jobs and on-disk archives before opening them. Metadata backups omit evidence; full and scoped backups can preserve data deleted from the live database. Incremental backups depend on sequence. Restore can replace signatures and optionally drop collections, so it belongs only in a disposable recovery environment.

Collector and infrastructure branch

A collector or anonymizer can reveal the database address, certificates, frontend trust, public files, topology, and synchronization activity. Preserve configuration, logs, current connections, certificate material, controller state, and uploaded/downloadable files before stopping the service.

Look for the private X-Auth-Frontend header in authorized passive telemetry, but redact its value: the source treats it as a broad component credential. Correlate decoy 444 or closed responses with collector logs, database traffic, heartbeats, and topology. Do not equate an nginx 444 alone with RCS.

Map peers from local evidence only until authority covers them. An anonymizer, remote archive, translation service, connector destination, or network injector may be operated by a different entity or hold separate victim data. The response plan should treat each as a new scope decision.

Endpoint branch

On a suspected endpoint, victim safety and preservation govern isolation. A network cut may prevent further collection; it may also trigger deletion, change adversary behavior, or interfere with a safety plan. Acquire memory, disk or mobile backup, logs, current network state, and relevant provider records using established forensic methods.

Keep suspected binaries inert. Hash and copy them into an isolated analysis environment; do not launch, rebuild, “fix,” or connect them to a collector. Use family identification, configuration recovery, persistence, injection, and C2 evidence to establish what the artifact is. The existence of an RCS binary does not reveal which exploit delivered it, whether collection succeeded, who operated it, or whether the backend found elsewhere belongs to the same deployment.

Endpoint and backend timelines can corroborate one another without importing all victim content into the investigation. Agent ident and instance, synchronization times, evidence types and counts, network endpoints, and file hashes may be enough to establish linkage. Minimize content review and record why any sensitive record had to be opened.

Attribution is a separate conclusion

RCS is a commercial platform. Product identity does not uniquely identify a government customer, operator, reseller, victim, or HackingTeam employee. A licence, operation name, IP address, language, timezone, or contact field can be stale, fabricated, migrated, or changed after compromise.

Report attribution in layers:

• the artifact or service is consistent with a specified RCS version;
• independent evidence links particular components;
• records indicate an account or organization had access;
• activity can be associated with a person or authority; and
• external evidence supports responsibility for targeting or collection.

Each step needs its own evidence and uncertainty. Do not turn customer negotiations from the leak into proof of deployment, or a target record into proof that a device was successfully infected.

Victim notification and data minimization

The people represented in RCS are not merely forensic objects. Notification can help them secure accounts, preserve legal rights, understand risk, and obtain support. It can also create danger if performed without safeguarding, legal review, and knowledge of the surveillance context.

Build a notification process with civil-society and victim-support expertise. Separate the minimum facts needed for outreach from raw evidence content. Explain what is confirmed: a target record, deployed agent, synchronization, specific evidence types, or actual collected content. Explain what remains unknown. Provide secure communication options and avoid exposing contacts or associates through the notification itself.

Detection and case-management systems should store metadata and restricted references rather than copies of private communications. Analysts should not open a photograph, chat, or location trail merely because the database makes it easy.

Remediation after preservation

Once evidence is secure and the response authority approves change, rebuild the backend from trusted media rather than attempting to clean a system with critical code-execution and privilege-escalation paths. Rotate operator, component, archive, database, TLS, signing, backup, and infrastructure credentials from a clean environment. Replace first-client bootstrap and shared server credentials with scoped identities. Require database authentication and restrict every listener.

Do not restore an old full backup wholesale into the clean system until it has been examined for malicious users, privileges, signatures, tasks, connectors, source changes, and forged or cross-operation records. Import only what the recovery plan has validated. Retain the original images and analysis clone under the applicable evidence and privacy schedule.

For affected endpoints, eradication and account recovery depend on the platform and threat model. Treat credentials and session tokens observed by the surveillance operation as exposed. Coordinate device rebuild, credential rotation, provider actions, and monitoring so that remediation does not alert an adversary before people are safe.

A defensible conclusion

The investigation is complete enough to act when it can answer five questions with explicit uncertainty:

1. What identifies this as RCS, and at what confidence?
2. Which components, versions, accounts, and data stores were involved?
3. Was the operator platform itself compromised?
4. Which people or devices may have been affected, and what data categories were collected or exposed?
5. Which containment, rebuild, rotation, notification, and monitoring actions are authorized and safe?

The original media, volatile capture, logical exports, query output, analyst notes, and derived timelines should have verifiable chain of custody. Every interaction with the system should be reproducible from the log. The final report should distinguish source expectation, observed fact, lab behavior, and inference.

RCS was designed to turn private life into a searchable analyst workflow. A responsible investigation should not repeat that logic by treating unrestricted access as permission. Preserve broadly, inspect narrowly, explain uncertainty, and keep the people behind the records at the center of the response.

Sources and evidence

• SOURCE: rcs-db item, user, group, session, evidence, GridFS, alert, audit, backup, status, connector, task, and authentication models/controllers at commit 6cff59d28634d718cac9fdd17cb629fd59a3cf3f.
• LAB: docker-lab/README.md, IOCs.md, compose volumes and listener topology, console live-test cleanup behavior, and synthetic evidence storage.
• ASSESSMENT: Chapter 10’s 27 validated findings and explicit deferred questions; apply them only after deployed-version and reachability checks.
• PROCEDURE: research/hackingteam-rcs/incident-response/IR-CHECKLIST.md and MONGODB-QUERIES.md; MONGODB26-VALIDATION.md records the synthetic MongoDB 2.6 runtime replay and no-write equality check.
• LIMITATION: no live incident, no third-party interaction, no implant or exploit execution, and no claim that generic acquisition steps replace an organization’s forensic, legal, or safeguarding procedures.