Wednesday, February 6, 2019

Remote LIVE Memory Analysis with The Memory Process File System v2.0

This blog entry aims to give an introduction to The Memory Process File System and show how easy it is to do high-performant memory analysis even from live remote systems over the network.

This and much more is presented in my BlueHatIL 2019 talk on February 6th.

Connect to a remote system over the network over a kerberos secured connection. Acquire only the live memory you require to do your analysis/forensics - even over medium latency/bandwidth connections.

An easy to understand file system user interface combined with continuous background refreshes, made possible by the multi-threaded analysis core, provides an interesting new different way of performing incident response by live memory analysis.

Analyzing and Dumping remote live memory with the Memory Process File System.
The image above shows the user staring MemProcFS.exe with a connection to the remote computer and with the DumpIt live memory acquisition method. it is then possible to analyze live memory simply by clicking around in the file system. Dumping the physical memory is done by copying the pmem file in the root folder.

The Memory Process File System was released for PCILeech in March 2018, supporting 64-bit Windows, and was used to find the Total Meltdown / CVE-2018-1038 page table permission bit vulnerability in the Windows 7 kernel. People have also used it to cheat in games - primarily cs:go using it via the PCILeech API.

The Memory Process File System was released as a stand-alone project focusing exclusively on memory analysis in November 2018. The initial release included both APIs and Plugins for C/C++ and a Python. Support was added soon thereafter for 32-bit memory models and Windows support was expanded as far back as Windows XP.

What is new?
Version 2.0 of The Memory Process File System marks a major release that was released in conjunction with the BlueHatIL 2019 talk Practical Uses for Hardware-assisted Memory Visualization.

New functionality includes:
  • A new separate physical memory acquisition library - the LeechCore.
  • Live memory acquisition with DumpIt or WinPMEM.
  • Remote memory capture via a remotely running LeechService.
  • Support from Microsoft Crash Dumps and Hyper-V save files.
  • Full multi-threaded support in the memory analysis library.
  • Major performance optimizations.
The combination live memory capture via Comae DumpIt, or WinPMEM, and secure remote access may be interesting both for convenience and incident-response. It even works remarkably well over medium latency- and bandwidth connections.

The LeechCore library
The LeechCore library, focusing exclusively on memory acquisition, is released as a standalone open source project as a part of The Memory Process File System v2 release. The LeechCore library abstracts memory acquisition from analysis and makes things more modular and easier to re-use. The library supports multiple memory acquisition methods - such as:
  • Hardware: USB3380, PCILeech FPGA and iLO
  • Live memory: Comae DumpIt and WinPMEM
  • Dump files: raw memory dump files, full crash dump files and Hyper-V save files.
The LeechCore library also allows for transparently connecting to a remote LeechService running on a remote system over a compressed mutually authenticated RPC connection secured by Kerberos. Once connected any of the supported memory acquisition methods may be used.

The LeechService
The LeechService may be installed as a service with the command LeechSvc.exe install. Make sure all necessary dependencies are in the folder of leechsvc.exe - i.e. leechcore.dll and att_winpmem_64.sys (if using winpmem). The LeechService will write an entry, containing the kerberos SPN to the application event log once started provided that the computer is a part of an Active Directory domain.
The LeechService is installed and started with the Kerberos SPN: book-test$@AD.FRIZK.NET
Now connect to the remote LeechService with The Memory Process File System - provided that the port 28473 is open in the firewall. The connecting user must be an administrator on the system being analyzed. An event will also be logged for each successful connection. In the example below winpmem is used.
Securely connected to the remote system - acquiring and analyzing live memory.
It's also possible to start the LeechService in interactive mode. If starting it in interactive mode it can be started with DumpIt to provide more stable memory acquisition. It may also be started in insecure no-security mode - which may be useful if the computer is not joined to an Active Directory domain.
Using DumpIt to start the LeechSvc in interactive insecure mode.
If started in insecure mode everyone with access to port 28473 will be able to connect and capture live memory. No logs will be written. The insecure mode is not available in service mode. It is only recommended in secure environments in which the target computer is not domain joined. Please also note that it is also possible to start the LeechService in interactive secure mode.

To connect to the example system from a remote system specify:
MemProcFS.exe -device dumpit -remote rpc://insecure:<address_of_remote_system>

How do I try it out?
Yes! - both the Memory Process File System and the LeechService is 100% open source.
  1. Download The Memory Process File System from Github - pre-built binaries are found in the files folder. Also, follow the instructions to install the open source Dokany file system.
  2. Download the LeechService from Github - pre-built binaries with no external dependencies are found in the files folder. Please also note that you may have to download Comae DumpIt or WinPMEM (download and copy .sys driver file to directory of MemProcFS.exe) to acquire live memory.

The Future
Please do keep in mind that this is a hobby project. Since I'm not working professionally with this future updates may take time and are also not guaranteed.

The Memory Process File System and the LeechCore is already somewhat mature with its focus on fast, efficient, multi-threaded live memory acquisition and analysis even though current functionality is somewhat limited.

The plan for the near future is to add additional core functionality - such as page hashing and PFN database support. Page hashing will allow for more efficient remote memory acquisition and better forensics capabilities. PFN database support will strengthen virtual memory support in general.

Also, additional and more efficient analysis methods - primarily in the form of new plugins will also be added in the medium future.

Support for additional operating systems, such as Linux and macOS is a long-term goal. It shall however be noted that the LeechCore library is already supported on Linux.

Tuesday, March 27, 2018

Total Meltdown?

Did you think Meltdown was bad? Unprivileged applications being able to read kernel memory at speeds possibly as high as megabytes per second was not a good thing.

Meet the Windows 7 Meltdown patch from January. It stopped Meltdown but opened up a vulnerability way worse ... It allowed any process to read the complete memory contents at gigabytes per second, oh - it was possible to write to arbitrary memory as well.

No fancy exploits were needed. Windows 7 already did the hard work of mapping in the required memory into every running process. Exploitation was just a matter of read and write to already mapped in-process virtual memory. No fancy APIs or syscalls required - just standard read and write!

Accessing memory at over 4GB/s, dumping to disk is slower due to disk transfer speeds.

How is this possible?
In short - the User/Supervisor permission bit was set to User in the PML4 self-referencing entry. This made the page tables available to user mode code in every process. The page tables should normally only be accessible by the kernel itself.

The PML4 is the base of the 4-level in-memory page table hierarchy that the CPU Memory Management Unit (MMU) uses to translate the virtual addresses of a process into physical memory addresses in RAM. For more in-depth information about paging please have a look at Getting Physical: Extreme abuse of Intel based Paging Systems - Part 1 and Part 2.

PML4 self-referencing entry at offset 0xF68 with value 0x0000000062100867.

Windows have a special entry in this topmost PML4 page table that references itself, a self-referencing entry. In Windows 7 the PML4 self-referencing is fixed at the position 0x1ED, offset 0xF68 (it is randomized in Windows 10). This means that the PML4 will always be mapped at the address: 0xFFFFF6FB7DBED000 in virtual memory. This is normally a memory address only made available to the kernel (Supervisor). Since the permission bit was erroneously set to User this meant the PML4 was mapped into every process and made available to code executing in user-mode.

"kernel address" memory addresses mapped in every process as user-mode read/write pages.

Once read/write access has been gained to the page tables it will be trivially easy to gain access to the complete physical memory, unless it is additionally protected by Extended Page Tables (EPTs) used for Virtualization. All one has to do is to write their own Page Table Entries (PTEs) into the page tables to access arbitrary physical memory.

The last '7' in the PML4e 0x0000000062100867 (from above example) indicates that bits 0, 1, 2 are set, which means it's Present, Writable and User-mode accessible as per the description in the Intel Manual.

Excerpt from the Intel Manual, if bit 2 is set to '1' user-mode access are permitted.

Can I try this out myself?
Yes absolutely. The technique has been added as a memory acquisition device to the PCILeech direct memory access attack toolkit. Just download PCILeech and execute it with device type: -device totalmeltdown on a vulnerable Windows 7 system.

Dump memory to file with the command: pcileech.exe dump -out memorydump.raw -device totalmeltdown -v -force .

If you have the Dokany file system driver installed you should be able to mount the running processes as files and folders in the Memory Process File System - with the virtual memory of the kernel and the processes as read/write.

To mount the processes issue the command: pcileech.exe mount -device totalmeltdown .

Please remember to re-install your security updates if you temporarily uninstall the latest one in order to test this vulnerability.

A vulnerable system is "exploited" and the running processes are mounted with PCILeech.
Process memory maps and PML4 are accessed.

Is my system vulnerable?
Only Windows 7 x64 systems patched with the 2018-01 or 2018-02 patches are vulnerable. If your system isn't patched since December 2017 or if it's patched with the 2018-03 2018-03-29 patches or later it will be secure.

Other Windows versions - such as Windows 10 or 8.1 are completely secure with regards to this issue and have never been affected by it.

I discovered this vulnerability just after it had been patched in the 2018-03 Patch Tuesday. I have not been able to correlate the vulnerability to known CVEs or other known issues.

Windows 2008R2 was vulnerable as well.
OOB security update released to fully resolve the vulnerability on 2018-03-29. CVE-2018-1038. Apply immediately if affected!

2018-03-xx--25: Issue identified in Windows 7 x64. Issue seemed to be patched already. PoC coded. Contacted MSRC with technical description asking if OK to publish a blog entry or if I should hold off publication.
2018-03-26: Green light given by MSRC for me to publish blog entry.
2018-03-27: Published blog entry and PoC.
2018-03-28: Found out that the March patches only partially resolved the vulnerability. Contacted MSRC again.
2018-03-29: OOB security update released by Microsoft. CVE-2018-1038. Apply immediately if affected!

Huge Thank You to everyone at Microsoft that worked hard to resolve this issue. It is super impressive to be able to be able to roll out a complex kernel update in little over a day. It was never my intention to release a fairly potent kernel 0-day publicly. I hope the above timeline explains how this could happen.

Monday, March 12, 2018

Introducing the Memory Process File System for PCILeech

The Memory Process File System for PCILeech is an easy and convenient way to quickly look into memory dumps. The processes in a memory dump and their virtual memory should be mapped as files and folders in less than a second.

Click around the processes, look at their memory maps and corresponding virtual memory! Oh - if you run it in live mode with a supported PCILeech FPGA device you'll be able to write to memory as well! Super convenient if your target system employs software based anti-forensic or anti-cheating functionality since this is all handled in hardware on the target!

Using the Memory Process File System for PCILeech to explore Windows processes in a memory dump file.

Cool! So how do I try it out?
Check out and download PCILeech on Github. PCILeech is open source and totally free of charge! You'll find both source code and pre-compiled binaries on Github.

How does it work?
It works by parsing in-memory page tables which are used by the CPU to translate the virtual memory of a process into physical memory. Since page tables are hardware dependent this means that the Memory Process File System is able to map processes from all 64-bit operating systems running on x64 hardware, if the base of the process page table called PageDirectoryBase or PML4 or CR3 is known.

If parts of the page tables, or other process memory, doesn't exist in the memory, it may have been swapped out to the swap on disk, analysis of that part of memory will fail.

If analyzing Windows memory dumps additional analysis will be performed to identify running processes, modules and which memory regions they belong to.

What about Functionality and Limitations?
- The Memory Process File System is currently only supported when running PCILeech on Windows.
- x64 64-bit target operating systems only, no 32-bit, no ARM.
- Read-only mode on memory dump files, read-write mode if PCILeech FPGA is used on a live system.
- Automatic process identification only in Windows memory dumps.
- Automatic identification of EPROCESS, PEB and DLL addresses in Windows memory dumps.
- May fail on memory dumps taken from Virtual Machines, such as VirtualBox.
- May fail for various other reasons as well.

Looking at the Linux kernel by manually specifying a valid CR3 value (PageDirectoryBase/PML4).

How Forensically Secure is it?
Memory swapped out to disk will also be invisible to the Memory Process File System - there is no guarantee that everything will be mapped.

There is no way for user-mode malware to hide from the Memory Process File System though. Since the Memory Process File System analyzes page tables, which are used by the CPU and thus must remain in memory somewhat unobfuscated.

The Memory Process File System however employs some tricks to filter out what looks like invalid page tables. This means that malware running with kernel privileges may hide from the Memory Process File System at the moment

Acquiring Memory Dumps
If you have access to a PCILeech supported FPGA device this is the preferred acquisition method. Insert the PCILeech FPGA device in your target computer. Then identify the size of the physical memory address space. This is likely to be slightly larger than the amount of RAM in the computer. Do this by running the command "pcileech.exe probe". This will give an output similar to the one below. Here the maximum address 0x21E5FFFFF is important.
Enumerating target system memory with a PCILeech FPGA device (PCIeScreamer in image).
Now we are able to dump the memory. Dump using "pcileech.exe dump -v -force -max 0x21E5FFFFF -out my_dump.raw". The -force option tells PCILeech not to abort when it encounters unreadable memory, such as memory holes between 3-4 GB or memory protected with Vt-d.

DumpIt is a software one-click dumper that is easy to run. The older MoonSols DumpIt may however create corrupt memory dumps on more recent Windows 10 computers, hence it's recommended to use the more recent up-to-date Comae DumpIt or WinPMEM on Windows 10.

Comae DumpIt may be run in an elevated command prompt with the /t raw switch to create raw memory dumps. "DumpIt.exe /t raw dump.raw"

WinPMEM is a software memory dumping utility which is part of the Google Rekall memory forensics project on Github. You'll find a signed WinPMEM to download here. Execute, from an administrator elevated command prompt, "winpmem-2.1.post4.exe --format raw -o dump-pmem.raw.aff4". WinPMEM will dump to an .aff4 volume. Open this volume in 7-Zip and unpack the memory dump named PhysicalMemory.

Reading and Writing to Memory
Reading and writing to virtual memory can be done in two ways. Either open the vmem file in the process directory, which contains the whole process virtual memory, or open a file in the vmemd directory which contains individual virtual memory mappings. The memory map is found in the map file.
The memory map, vmem memory file and vmemd directory for the Notepad process.
If using PCILeech FPGA with a hardware based target the virtual memory files will be writable. Please note that writes go the the corresponding physical memory which may be shared between multiple processes! As an example, alterations written to ntdll.dll in one process is likely to be written into the ntdll.dll of all processes.
Looking at the notepad.exe virtual memory in hexedit in Ubuntu WSL.
When reading to, or writing to a kernel address, for example 0xFFFFF801'10000000 please open the vmem file in your favorite hex editor and navigate to the address as seen when cutting away the first four F's. That is 0xF801'1000000 in the example from above. The topmost 16-bits are what is called sign-extended so this won't matter. It's still the same address. This is done since Windows don't support file sizes above 63-bits (file size becomes negative).
Looking at notepads EPROCESS struct in kernel memory. Note that we look at it in the System process (PID 4) and that the leading four (4) FFFF are skipped from the address.

Special Files
The virt2phys file is a special always writable file. The first 64-bit value contains the virtual address and the second 64-bit value is the corresponding physical address. Whenever a new virtual address gets written the physical address will update.
notepads 0x7FF660160000 virtual address maps to the physical address 0x14C40D000.

The win-process file contains the address of the EPROCESS kernel struct that belongs to the process. Since the EPROCESS struct resides in kernel memory one must look for this one in the System (PID 4) process virtual memory.

The win-peb file contains the address of the PEB (Process Environment Block), which resides in process memory. The win-entry contains the process entry point, win-modules the modules (.dlls) in the PEB Ldr list.

The pml4 file contains the physical address of the PML4 a.k.a. the CR3 register a.k.a. the Page Directory Base.

The Future
The future of the Memory Process File System will depend on how useful it is. If useful I see a lot of possibilities, somewhat limited by time available to implement them. The Memory Process File System is not meant to become a serious forensic file system, that space will continue to be reserved by industry standard tools, such as the excellent Volatility. The Memory Process File System should nevertheless provide convenient easy-to use access if performing a quick analysis, or be helpful when patching live processes via DMA.

Comments, suggestions and ideas are very welcome. PCILeech in both source and binary distributions is found on Github.  Also check out Twitter for updated information and announcements.

Updated reference to Comae DumpIt.
The Memory Process File System is now released as a stand-alone separate project here.