Course Overview

CS 6V81 is a graduate level, research oriented, system security course. Our focus is digital forensics and data reverse engineering, which tackles the problem of what information is stored in a computer system and how this information can be extracted and used. There are a wide range of applications of data reverse engineering, including digital forensics, crash analysis, game hacking, kernel rootkit defense, and malware analysis. The overall goal of this course is to introduce students to the current techniques used in both research and practice.

In particular, we will cover the underline technical details (including the most recent techniques) of digital forensics and data reverse engineering, discuss various security applications, analyze potential limitations of existing systems, and propose/develop more secure systems. In the first a few lectures, the instructor will introduce the techniques, foundations, and applications of digital forensics and data reverse engineering. After that, in each class students will read current and seminal research papers from the reading materials. Students are encouraged to prepare a short summary/review of the paper of their choosing and submit it to the discussion board in UT Dallas elearning system. Students will lead and prepare for presentations explaining to others.

Students will also need to perform research, and will pick a semester-long research topic of their choosing. In addition, this course will have one hands-on challenge.

Course Schedule

Date Topic To Be Covered Presenter Slides
08/26 A Course Overview Instructor [pdf] [handout]
08/26 B OS (Memory Data Management), File System (Disk Data Management) Instructor [pdf] [handout]
Techniques, Tools, and Applications
09/02 A Data Structure Reverse Engineering: REWARDS [1], TIE [2], and HOWARD [3] Instructor [pdf] [handout]
09/02 B Kernel Rootkit Defense I: SigGraph [10], KOP [11], Crash tool[5], Memory graph [4] Instructor [pdf] [handout]
09/09 A Vulnerability Analysis I: Exploit Hardening [Q], Surgically Returning to Randomized lib(c) [58], Pointer corruption [7], Non-control data attack[8], and Data flow integrity [9] Mitchell Adair [pdf] [handout]
09/09 B Vulnerability Analysis II: Crash dump analysis using Bitblaze [6] Kevin Weaver [pdf] [handout]
09/16 A Protocol Reverse Engineering: Discoverer [12], Polyglot [13], AutoFormat [14], Protocol Analysis [15] ... Yangchun Fu [pdf] [handout]
09/16 B Digital Forensics I: Research problem and roadmap [16], the next 10 years [17] Kevin Weaver [pdf] [handout]
09/23 A Digital Forensics II: Metadata extraction [18][19][20] Junyuan Zeng [pdf] [handout]
09/23 B Digital Forensics III: File carving [21][22][23] Kevin Hulin [pdf] [handout]
09/30 A Digital Forensics IV: Cell phone forensics [24][25][26][27][28] Donald Talkington [pdf] [handout]
09/30 B Digital Forensics V: Network Forensics [29] (Wireshark/TCPdump, IP prefix hijacking[30], OS Fingerprinting[31]) Scott Hand [pdf] [handout]
10/07 A Digital Forensics VI: Crypto key discovery [32], and extraction [33] Tom Hill [pdf] [handout]
10/07 B Digital Forensics VII: Windows Memory Analysis[34][35][36][37][38] Camron Quitugua [pdf] [handout]
10/14 A Digital Forensics VIII: Legal and ethical issues [39] Donald Talkington [pdf] [handout]
10/14 B Game Hacking I: Differentiating human and bots [59] Nathan McDaniel [pdf] [handout]
10/21 Working on your project (instructor on leave for a conference)
10/28 A Malicious Code Analysis I: Using data structure as program signature (Laika[40]), Data structure layout randomization ([DSLR]) Scott Hand [pdf] [handout]
10/28 B Malicious Code Analysis II: String analysis for PHP [41, x86 binary [42] Mitchell Adair [pdf] [handout]
11/04 A Program Analysis I: Value-invariant discovery (Daikon[43][44]), Robust signatures using value-invariant [52] Junyuan Zeng [pdf] [handout]
11/04 B Program Analysis II: Server-side verification of client behavior in online games [60] Nathan McDaniel [pdf] [handout]
11/11 A Kernel Rootkit Defense II: data centric approach [46] Camron Quitugua [pdf] [handout]
11/11 B Using Reverse Engineering Practices to Improve Systems-of-Systems Understanding Tom Hill [pdf] [handout]
11/18 A Game Hacking II: Preventing map hacks [50] Kevin Hulin [pdf] [handout]
11/18 B Cloud Computing: Information leakage in 3rd party cloud [51] Yangchun Fu [pdf] [handout]
11/25 Thanksgiving Hollidays. No Class
Hands on Challenge: Participating UCSB iCTF
12/02 A UCSB iCTF (10AM - 7PM) ECSS 4.619
12/02 B UCSB iCTF (10AM - 7PM) ECSS 4.619
Project Presentation
12/09 A Project Presentation Adair, Fu, Hand, Hill, Hulin, McDaniel
12/09 B Project Presentation Talkington, Quitugua, Weaver, Zeng

Reading List

We do not have a text book, but we have the following reading list. Students are requied to read all of these papers (at least of their choosing) and encouraged to submit their summary on each paper.

[1] Z. Lin, X. Zhang, and D. Xu. "Automatic Reverse Engineering of Data Structures from Binary Execution," In NDSS 2010
[2] J. Lee, T. Avgerinos, and D. Brumley. "TIE: Principled Reverse Engineering of Types in Binary Programs," In NDSS 2011
[3] A. Slowinska, T. Stancescu, and H. Bos. "Howard: A Dynamic Excavator for Reverse Engineering Data Structures," In NDSS 2011
[4] T. Zimmermann and A. Zeller. "Visualizing memory graphs," In Revised Lectures on Software Visualization, International Seminar, 2002
[5] Mission Critical Linux -- Crash Core Analysis Suite
[6] Charlie Miller, Juan Caballero, Noah M. Johnson, Min Gyung Kang, Stephen McCamant, Pongsin Poosankam, and Dawn Song. "Crash Analysis with Bitblaze," In Blackhat 2010
[7] Shuo Chen, Jun Xu, Nithin Nakka, Zbigniew Kalbarczyk, and Ravishankar K. Iyer, "Defeating Memory Corruption Attacks via Pointer Taintedness Detection," In DSN 2005
[8] Shuo Chen, Jun Xu, Emre C. Sezer, Prachi Gauriar, and Ravishankar K. Iyer, "Non-Control-Data Attacks Are Realistic Threats," In USENIX Security 2005
[9] Miguel Castro, Manuel Costa, and Tim Harris. "Securing software by enforcing data-flow integrity," In OSDI 2006
[10] Z. Lin, X. Zhang, D.Xu, and X. Jiang. "SigGraph: Brute Force Scanning of Kernel Data Structure Instances Using Graph-based Signatures," In NDSS 2011
[11] M. Carbone, W. Cui, L. Lu, W. Lee, M. Peinado, and X. Jiang. "Mapping kernel objects to enable systematic integrity checking," In CCS 2009
[12] W. Cui, J. Kannan, and H. J. Wang. "Discoverer: Automatic protocol reverse engineering from network traces," In USENIX Security 2007
[13] J. Caballero and D. Song. "Polyglot: Automatic extraction of protocol format using dynamic binary analysis," In CCS 2007
[14] Z. Lin, X. Jiang, D. Xu, and X. Zhang. "Automatic protocol format reverse engineering through context-aware monitored execution," In NDSS 2008
[15] G. Wondracek, P. M. Comparetti, C. Kruegel, and E. Kirda. "Automatic Network Protocol Analysis," In NDSS 2008
[16] "A Road Map for Digital Forensic Research," Report From the First Digital Forensic Research Workshop (DFRWS). 2001
[17] Simson L. Garfinkel. "Digital forensics research: The next 10 years". In DFRWS 2010
[18] Metadata Extraction Tool,
[19] Hachoir
[20] Sleuthkit
[21] Mikus, Nicholas A. "An analysis of disc carving techniques," MS Thesis. Naval Postgraduate School, 2006
[22] Golden G. Richard and Vassil Roussev. "Scalpel: A Frugal, High Performance File Carver," In DFRWS 2005
[23] Anandabrata Pal and Nasir Memon. "The Evolution of File Carving," IEEE Signal Processing Magazine, Vol26(2) March 2009
[24] Vrizlynn L.L. Thing, Kian-Yong Ng, Ee-Chien Chang. "Live memory forensics of mobile phones," In DFRWS 2010
[25] Guidelines on Cell Phone Forensics (NIST SP 800-101), May 2007
[26] Cell Phone Forensic Tools: An Overview and Analysis (NISTIR 7250)
[27] PDA Forensic Tools: An Overview and Analysis (NISTIR 7100)
[28] R Ahmed. "Mobile forensics: an overview, tools, future trends and challenges from law enforcement perspective". 6th International Conference on E-Governance. 2008
[30] Hitesh Ballani, Paul Francis, and Xinyang Zhang. "A study of prefix hijacking and interception in the internet," In SIGCOMM 2007
[31] Techniques in OS-Fingerprinting,
[32] J. Halderman, S. Schoen, N. Heninger, W. Clarkson, W. Paul, J. Calandrino, A. Feldman, J. Appelbaum, E. Felton. "Lest We Remember: Cold Boot Attacks on Encryption Keys," In USENIX Security 2008
[33] Carsten Maartmann-Moe, Steffen E. Thorkildsen, Andre Arnes. "The persistence of memory: Forensic identification and extraction of cryptographic keys," In DFRWS 2009
[34] Andreas Schuster. "Searching for Processes and Threads in Microsoft Windows Memory Dumps," In DFRWS 2006
[35] Ali Reza Arasteh and Mourad Debbabi. "Forensic Memory Analysis: From Stack and Code to Execution History," In DFRWS 2007
[36] Mariusz Burdach. "Finding Digital Evidence In Physical Memory," In Black Hat Federal 2008
[37] Brendan Dolan-Gavitt. "Forensic Analysis of the Windows Registry in Memory," In DFRWS 2008
[38] Andreas Schuster. "The impact of Microsoft Windows pool allocation strategies on memory forensics," In DFRWS 2008
[39] Aaron J. Burstein, "Toward a Culture of Cybersecurity Research," UC Berkeley Public Law Research Paper No. 1113014, 2008
[40] A. Cozzie, F. Stratton, H. Xue, and S. T. Kin. "Digging for Data Structures," In OSDI 2008
[41]Fang Yu, Muath Alkhalaf, Tevfik Bultan. "Stranger: An Automata-based String Analysis Tool for PHP." [Tool paper]. In TACAS 2010
[42] Mihai Christodorescu, Nicholas Kidd, and Wen-Han Goh. "String analysis for x86 binaries," In PASTE 2005
[43] Michael D. Ernst, Jeff H. Perkins, Philip J. Guo, Stephen McCamant, Carlos Pacheco, Matthew S. Tschantz, and Chen Xiao. "The Daikon system for dynamic detection of likely invariants," Science of Computer Programming, 2007
[44] Michael D. Ernst, Jake Cockrell, William G. Griswold, and David Notkin. "Dynamically discovering likely program invariants to support program evolution," IEEE Transactions on Software Engineering, 27(2) 2001
[45] Y. Jhi, X. Wang, X. Jia, S. Zhu, P. Liu, and D. Wu. "Value-Based Program Characterization and Its Application to Software Plagiarism Detection," In ICSE 2011
[46] Junghwan Rhee, Ryan Riley, Dongyan Xu, and Xuxian Jiang. "Kernel malware analysis with un-tampered and temporal views of dynamic kernel memory. In RAID 2010
[47] A. Baliga, V. Ganapathy, and L. Iftode. "Automatic inferenceand enforcement of kernel data structure invariants," In ACSAC 2008
[48] An architecture for specification-based detection of semantic integrity violations in kernel dynamic data. In USENIX Security 2006
[49] Copilot - a coprocessor-based kernel runtime integrity monitor. In USENIX Security 2004
[50] E. Bursztein, J. Lagarenne, M. Hamburg, D. Boneh. "OpenConflict: Preventing Real Time Map Hacks in Online Games," In IEEE S&P (Oakland) 2011
[51] Thomas Ristenpart, Eran Tromer, Hovav Shacham, Stefan Savage. "Hey, You, Get Off of My Cloud! Exploring Information Leakage in Third-Party Compute Clouds," In CCS 2009
[52] B. Dolan-Gavitt, A. Srivastava, P. Traynor, and J. Giffin. "Robust signatures for kernel data structures," In CCS 2009
[53] Zhi Wang, Xuxian Jiang, Weidong Cui, Xinyuan Wang. "Countering Persistent Kernel Rootkits Through Systematic Hook Discovery". In RAID 2008
[54] Zhi Wang, Xuxian Jiang, Weidong Cui, and Peng Ning. "Countering Kernel Rootkits with Lightweight Hook Protection". In CCS 2009
[55] Heng Yin, Pongsin Poosankam, Steve Hanna, and Dawn song. "HookScout: Proactive and Binary-Centric Hook Detection". In DIMVA 2010
[56] Zhiqiang Lin, Ryan Riley, and Dongyan Xu. "Polymorphing Software by Randomizing Data Structure Layout". In DIMVA 2009
[57] Edward J. Schwartz, Thanassis Avgerinos, and David Brumley. "Q: Exploit Hardening Made Easy ". In USENIX Security 2011
[58] Giampaolo Fresi Roglia, Lorenzo Martignoni, Roberto Paleari, Danilo Bruschi "Surgically Returning to Randomized lib(c)". In ACSAC 2009
[59] Steven Gianvecchio, Zhenyu Wu, Mengjun Xie, and Haining Wang, "Battle of Botcraft: Fighting Bots in Online Games with Human Observational Proofs". In ACM CCS 2009
[60] D. Bethea, R. A. Cochran and M. K. Reiter, "Server-side verification of client behavior in online games". In NDSS 2010

Office Hours

Thursday 3-5PM


"Data Structures", "Compilers", "Operating System", "System Security", or permission of the instructor.

Note for undergraduate students who may be interested in taking this class, please be aware that the class is designed for graduate students, you are encouraged to attend the first lecture and then talk to the instructor.

Course Projects

Course Policy

Grading Policy

Late Policy

No late submission. Otherwise, it will be penalized or may not be graded.

Collaboration Policy

Students are encouraged to collaborate, particularly on the course project. But we will limit the team member to at most two students. For paper review, studentes are encourage to talk to each other as well, but each student must turn in his or her summary.

Cheating Policy

We will strictly follow the university policy on cheating and plagiarism which is available here. Please avoid. There are also several examples of Scholastic Dishonesty If you have any questions regarding this issue, please contact the instructor.