Finding Evil with Skeptic SIFT

Recently, SANS hosted a hackathon to build autonomous AI agents that could run on the SANS SIFT Workstation. The main goal was to improve the existing Protocol SIFT, a framework that helps AI agents use the forensics tools included in the SIFT workstation.

My goal

The hackathon ran from the 15th of April until the 19th of June. Due to my own time constraints, I only had a day to dedicate to this. Therefore, my goal was not to truly compete against teams that had been working for months on solutions (I did not even submit my project). Instead, I wanted to see what would be possible to do within a day.

Current state of Protocol SIFT

Protocol SIFT adds skills to your AI agent such that it's able to better understand and use the tooling that comes within SIFT. For example, this is the first part of the windows-artifacts skill that gets installed with Protocol SIFT:

sansforensics@siftworkstation: ~/.claude/skills/windows-artifacts
$ head -n 25 SKILL.md 
# Skill: Windows Artifacts (EZ Tools / Autoruns / Event Logs)

## Overview
Use this skill for Windows host-based artifact analysis on the SIFT workstation.
Covers Eric Zimmerman (EZ) Tools, ASEP/persistence analysis, Windows event log parsing,
and execution/access evidence artifacts.

---

## EZ Tools — Running on Linux (SIFT)

All EZ Tools are installed at `/opt/zimmermantools/`. On Linux they run via the
.NET runtime using the `.dll` file — **not** the `.exe` (which is a Windows PE binary).

```bash
# Root-level tools:
dotnet /opt/zimmermantools/<ToolName>.dll [options]

# Subdirectory tools:
dotnet /opt/zimmermantools/<Subdir>/<ToolName>.dll [options]
```

> **GUI tools** (TimelineExplorer, RegistryExplorer, MFTExplorer, ShellBagsExplorer)
> are Windows PE applications. Run via `wine` on SIFT, or use the Windows analysis VM.

Note that these are mostly instructions on how to use tooling, and allow the agent to create the appropriate shell commands to execute. This offers great flexibility while still providing the agent with all required information.

However, it does leave the door open for more hallucination and less reproducibility. Therefore, I decided to create a small PoC to address these issues.

Skeptic SIFT

The idea is to give Protocol SIFT more constraints and add an audit trail to enable better reproducibility. To achieve this, I added an MCP server that Claude can interface with. This MCP server exposes multiple tools an agent can call. I kept a limited scope for now, but there are tools for memory analysis (Volatility plugins), disk analysis (The Sleuth Kit), as well as some 'meta' tools, which help the agent stay on track.

The meta tools are probably the biggest addition and include the following:

Tool	Purpose
case_status	Active image, its SHA-256, case ID, ledger location, what's been run.
list_capabilities	The whitelist, grouped by phase, with guidance — the agent's "menu".
audit_log	The provenance ledger: every run, its full command, exit code, timing, output hash.
verify_chain	Recompute the hash chain; detect any tamper/deletion/reorder.
verify_finding(seq)	Re-run a recorded command; confirm the output still hashes identically.
read_artifact(sha)	Fetch the complete byte-for-byte output behind any result.

As the agent does its analysis, it will keep a ledger. The tools mentioned above will help it keep track of this ledger, as well as verify it. With this ledger, we can add a healthy dose of skepticism: verify_finding(seq) will re-run a command and verify the output is the same (to avoid hallucination). Outside of the ledger, the MCP server will also give guidance to the agent to be a bit more skeptical.

Results

Below is an overview of some of the results I got. I ran Skeptic SIFT on two forensics cases, testing both disk and memory forensics abilities:

Case	Evidence	What it found
s4a web server	Windows Server 2008 memory + raw NTFS disk image	DVWA file-upload compromise from 192.168.56.102; phpshell.php, phpshell2.php, c99.php; loaded kernel driver ad_driver.sys; local hacker account RID 1006; exposed Apache/FTP/MySQL services.
Rocba-Memory	Windows 10 memory image	Active RDP attack: 57 carved TCP/3389 connection objects from three external IPs, two established sessions at the same timestamp, suspicious D:\Tools\MRC.exe, and DKOM-hidden svchost.exe PID 7900 created 97 seconds after RDP success.

Skepticism: corrections it made

Below are some of the explicit self-corrections the agent made during analysis.

Note: this is a single agent running. Another strategy would be to run multiple agents and have the results compared by a coordinator, who will make the final call on what is evil and what is not.

s4a web server

• Rejected the tempting "RWX == injected" shortcut: six malfind regions had no MZ header and matched benign runtime/zero-fill patterns.
• Treated four modscan-only null/garbage rows as pool-tag false positives, not hidden drivers.
• Noted no active C2 at capture and no console history, so the attack narrative rests on web logs, disk artifacts, memory modules, and SAM evidence.

Rocba-Memory

• Diffed 11 psscan-only PIDs and kept only PID 7900 as malicious; the others were exited churn or Windows Search borderline cases needing disk corroboration.
• Explained all 16 malfind rows as non-PE JIT/trampoline/zero-fill artifacts.
• Marked hashdump, lsadump, cmdscan, and consoles 0-row results as limits, not absence of compromise.

Interested to read more or try it out for yourself? Check out the project on GitHub. This repo includes example ledgers, as well as the reports for the s4a and Rocba cases.