Appendix H: API and Storage Map

This appendix maps the examined RCS database API to its MongoDB storage. It is designed for source review, defensive reconstruction, and incident response—not as an operator tutorial. Request bodies, credential values, implant configuration, build parameters, and exploit-delivery instructions are deliberately omitted. The map describes the frozen original rcs-db source; lab-only proxying and safety controls are identified separately.

The central caution is that neither the URL structure nor the collection layout is an authorization boundary. The dispatcher authenticates a session, then individual controller actions apply symbolic privileges. Some actions also filter documents by denormalized user membership, while others do not. Chapter 10 assesses those inconsistencies. A responder should use this map to find evidence on an authorized forensic clone, not to infer that a reachable route is safe to call.

Service and transport boundary

The original database process exposes two adjacent TLS services. Its default HTTPS REST listener is TCP 443, and the WebSocket listener is the next port, 444. The worker normally receives synthetic or collected evidence on TLS 442. MongoDB is split among a mongos router on 27017, a shard server on 27018, and a configuration server on 27019 in the reconstructed topology.

The lab maps only the database listeners to host loopback:

The original HTTP response object sets Server: nginx even though the Ruby EventMachine application generated the response. JSON is the default content type; selected listings use gzip, and file/GridFS actions stream binary content. Unknown root requests receive nginx-like HTML. These are protocol facts and detection leads, not proof that nginx fronts the service (rcs-db/lib/rcs-db/rest_response.rb:40-144; rcs-db/lib/rcs-db/rest.rb:328-340).

How a request becomes an action

The parser accepts paths in three broad forms:

/<controller>
/<controller>/<action-or-id>
/<controller>/<action>/<id>

If the first path segment after the controller names a public method on that controller, it becomes the action. Otherwise the dispatcher maps the HTTP verb to index, show, create, update, or destroy:

This is only a routing description. Named actions do not inherit a universal verb constraint, and individual methods interpret their own parameters. The controller name is capitalized and resolved dynamically from the registered controller classes (rcs-db/lib/rcs-db/parser.rb:24-27,73-121; rcs-db/lib/rcs-db/rest.rb:118-183,232-243).

Parameters can come from a CGI query string, JSON body, multipart body, or a Ruby Marshal body. A body advertised as ordinary URL-encoded form data is not actually parsed as a form body: outside multipart and Marshal branches, the parser attempts JSON and otherwise treats the body as binary. Parsing happens before controller session validation, a sequencing detail relevant to the security review (rcs-db/lib/rcs-db/parser.rb:30-63,82-121; rcs-db/lib/rcs-db/events.rb:122-163).

Most calls authenticate with a case-insensitive session=<UUID> cookie. The controller retrieves the corresponding sessions document and then evaluates action-specific privilege checks. Responses normally serialize objects to JSON. The API has no original browser CORS requirement; the AIR client did not need one. The web reconstruction therefore uses a same-origin nginx proxy rather than changing the backend.

Authentication and authorization layers

Four layers recur across the surface:

1. Session authentication. Every action requires a valid session unless its controller lists it in bypass_auth.
2. Licence checks. A small number of actions require a named entitlement before session handling.
3. Role and sub-privilege checks. require_auth_level tests the symbolic array captured in the session at login. Sequential checks can require both a base role such as VIEW and a sub-privilege such as VIEW_DELETE.
4. Object membership. Some queries constrain items or entities by user_ids, derived largely from groups. This layer is action-specific and is not consistently applied.

The ordinary roles are ADMIN, SYS, TECH, and VIEW, each with narrower privileges such as user administration, backend management, configuration, file execution, evidence editing/deletion/export, profiles, and alerts (db_objects/user.rb:9-36). A component can instead receive a server session. Session privileges are a login-time snapshot, so later user changes do not automatically revise an already issued session.

The explicitly session-bypassed surface is small but important:

“Bypassed” means the normal cookie gate is skipped, not that the action has no security assumptions. The synchronization actions implement their own shared signature comparison, while setup has first-write bootstrap behavior (rest/sync.rb:9-152). The worker’s separate evidence receiver does not use this controller and stages a non-empty body without an equivalent application authentication check (lib/rcs-worker/worker_controller.rb:7-34).

REST controller map

The following table is an inventory, not a list of supported client recipes. “Nominal access” summarizes explicit role checks; object-level membership and licence requirements can add constraints—or, as Chapter 10 documents, may be missing on a particular path.

Several names are easy to misread. The shard prefix is singular /shard. Backup jobs and produced archives are separate controllers. The generic evidence index excludes filesystem, info, command, and ip, which have special actions. The examined controller provides ips, but the later console library requests sync_history; that route is absent. The library also expects an evidence/update_multi action not found in this snapshot.

The task surface has its own version defect: task creation keys the in-memory map by user name, while list indexes it with the user object and then :name. Consequently GET /task fails on a fresh examined session even though individual task polling can work (tasks.rb:368-399). This is a fidelity seam, not an invitation to repair the original behavior in the historical account.

WebSocket push map

The WebSocket channel is state notification, not an alternate REST API. A client first sends an auth message carrying the existing session cookie. A valid session receives auth: granted and server time; the socket is stored by cookie. The server and client exchange ping and pong, and a received pong refreshes the session’s last-contact time. The default timeout is 900 seconds from that time (rcs-db/lib/rcs-db/websocket.rb:23-105; rcs-db/lib/rcs-db/sessions.rb:99-129).

Observed/source-backed push families include logout, message, monitor/status, operation and agent changes, evidence/dashboard updates, alerts, and entity changes. Recipient selection generally uses user IDs rather than the exact old socket that caused an event. That design explains the same-user-login race described in Chapters 5 and 6: a queued logout for a user can reach a newly connected socket belonging to that user.

Malformed-message process impact remains an explicit security follow-up. The source parses JSON before its message switch and dereferences session state in the pong path, but this project has not promoted those traces into a process- termination claim without scoped validation.

Database topology

The source defines two Mongoid sessions:

MongoDB configuration database
└── shard membership and collection shard-key metadata

main application database (lab name: rcs, through mongos)
├── global control and investigation collections
├── evidence.<target_id>
├── aggregate.<target_id>
├── grid.<target_id>.files
└── grid.<target_id>.chunks

worker staging database (default name: rcs-worker, direct shard session)
├── grid.evidence.files
└── grid.evidence.chunks

The application session uses MONGOID_DATABASE through MONGOID_HOST and MONGOID_PORT. The worker session defaults to database rcs-worker and a host/port default of 127.0.0.1:27018; the lab points it at the internal shard server. Acquisition must keep the two namespaces separate: grid.evidence staging is not in the main rcs namespace (config/mongoid.yaml:1-14).

On startup, the database enables sharding, creates global indexes, shards the audit collection, creates default filters and queues, clears server sessions, and prepares an ensured metadata backup (rcs-db/lib/rcs-db/db.rb:51-103). The MongoDB configuration database is therefore part of the evidence needed to interpret the logical application collections.

Global collection catalogue

Mongoid creates some relationship keys and default collection names in addition to explicitly declared fields. Preserve actual BSON, indexes, and collection options rather than rebuilding a schema solely from this table.

Target-scoped evidence families

Creating a target prepares three related storage families:

TargetScoped creates a duplicate model class whose collection name is the lowercase prefix plus target ID (target_scoped.rb:13-49). The target ID in the namespace organizes and shards records, but does not independently verify that the caller belongs to that target.

The evidence type list includes address books, applications, calendars, calls, camera images, chats, clipboards, device records, files, keylogs, messages, microphone recordings, money, mouse data, passwords, positions, prints, screenshots, and URLs. Four additional pseudo-view types—filesystem, info, command, and IP—are excluded from normal statistics. Binary content is stored in GridFS rather than inline when the processing path supplies a grid reference (db_objects/evidence.rb:16-48).

The worker stages incoming encrypted blobs in rcs-worker as grid.evidence.files/chunks, with filename set to the agent UID and a creation timestamp in metadata. Per-instance processing fetches at most 100 files oldest-first, normally deletes the raw GridFS entry after processing, and can leave decoded failure bytes in a worker filesystem directory (instance_worker.rb:32-75,187-218). Those filesystem failures are outside MongoDB and belong in an acquisition plan.

Queue and log collections

Notification queues are capped collections created with a default maximum of 100,000 entries and 50 MB; push_queue overrides that with 1,000 entries and 100 KB. Each uses a flag where zero is queued and one is processed. Because the dispatcher flips the flag when claiming an item, processed entries may remain in a capped queue until rotation.

Collectors and injectors receive dynamic capped collections named logs.<component_id>. Each stores integer time, string type, and description, with a maximum of 2,000 records or 1 MB. Deleting the component drops its log collection; the log-clearing endpoint also drops rather than empties it (db_objects/log.rb:6-46; collector/injector model callbacks). Collection absence after a deletion is therefore expected behavior, not proof that no logs ever existed.

Deletion and preservation relationships

The highest-risk storage actions cross collection boundaries:

• deleting one evidence document attempts to delete its referenced GridFS object and adjusts embedded statistics;
• bulk deletion records filter criteria rather than a durable list of every affected evidence ID;
• deleting an agent removes matching evidence from its parent target family;
• deleting a target drops its entire evidence, aggregate, and target GridFS collections;
• deleting an operation cascades through targets;
• connector queue completion can destroy evidence after all required exports;
• collector or injector deletion drops its capped component log; and
• backup restore can replace signatures and, depending on mode, drop collections before restoring them.

Forensic acquisition must consequently preserve relationships before invoking the application. At minimum, record collection names, options, counts, indexes, shard metadata, BSON types, GridFS file/chunk joins, filesystem archives, worker staging, decoded failures, configuration, and logs. JSON-only exports lose type distinctions such as string versus ObjectId aid values.

The read-only companion catalogue at research/hackingteam-rcs/incident-response/MONGODB-QUERIES.md begins with metadata and projections that omit password hashes, session cookies, signature values, agent keys, and evidence bodies. All 24 blocks pass against a network-disabled disposable MongoDB 2.6 synthetic clone, with eight targeted assertions and unchanged pre/post database hashes. This does not make shell access passive or authorize use against original evidence.

Known fidelity seams

This map applies to the examined source snapshot, not an abstract “RCS 9” API:

• component version markers disagree across rcs-db, common library, and console sources;
• the later console requests actions absent from the older controller, including evidence sync history and multi-update;
• task listing contains a source defect even though per-task polling works;
• the research web console adds /api/ and /wss/ proxy prefixes that do not exist on the original backend;
• mock fixtures and direct Moped seeds reproduce stored shapes without proving worker or implant provenance; and
• lab port mappings and internal-only worker/MongoDB exposure are defensive orchestration choices.

When a field, route, or collection differs in an acquired deployment, preserve the difference. It may reflect another version, customer configuration, migration, patch, corruption, defensive reconstruction, or compromise. The source map is a comparison baseline, not a repair specification.

Sources and evidence

• SOURCE: lib/rcs-db/parser.rb, rest.rb, rest_response.rb, websocket.rb, sessions.rb, target_scoped.rb, grid.rb, db.rb, db_layer.rb, rest/*.rb, and db_objects/*.rb in the frozen original rcs-db snapshot; worker staging from lib/rcs-worker/worker_controller.rb and instance_worker.rb.
• LAB: docker-lab/docker-compose.yml for loopback mappings, internal service boundaries, database names, and MongoDB roles; live console tests described in Chapter 6; and the recorded but not yet publication-traced synthetic worker exercise described in Chapter 7.
• INFERENCE: forensic prioritization follows from the source-defined relationships and deletion paths; it does not establish what a historical operator retained or exposed.
• LIMIT: no implant, exploit, collector, anonymizer, injector, build pipeline, or operational connector was executed for this appendix.