Course Overview

CS 6301 is a graduate level, research oriented, systems and software security class.

The goal of this course is to understand the low-level system details with the real system implementations from compiler, linker, loader, to OS kernel and computer architectures, examine the weakest link in each system component, explore the left bits and bytes after all these transformations, and study the state-of-the-art offenses and defenses.

The learning outcome is students shall be able to understand how an attack is launched (e.g., how an expoit is created), and how to do the defense (e.g., developing OS patches, analyzing the binary code, and detecting intrusions)

In particular, we will cover

This class will have heavy-hands on projects. The students will be able to get the experience on how to build real systems with virtual machine monitors, and process instrumentations.

Course Schedule

Please login your elearning website and download the lecture notes.

Week Topic To Be Covered
0 Course Overview
Understanding Binary Code Analysis
1 Binary Code/Data Representation
2 Program Representation
3 Dynamic Binary Instrumentation (PIN, Valgrind, Qemu)
4 Principles of Program Analysis
5 Design and Implementation of a Data Flow Analysis (taint analysis)
Understanding the OS Kernel
6 Understanding the OS Architecture
7 Process Management
8 Virtual Memory
9 File System
Beyond OS Kernel
10 Executable File Format: ELF, PE, Library Interposition
11 Revealing Internals of Compiler (gcc), Linker, and Loader
12 Virtual Machine Introspection (QEMU/VirtualBox/Xen/KVM)
Other Techniques, Tools, and Applications
13 Symbolic Execution and Whitebox Fuzzing
14 Exploits: Buffer Overflows, Heap Overflow, Integer Overflow
15 Robust Exploits: ROP shellcode, Heap Spray
16 Fighting for Malware: Unpack, Disassemble, Decompile

Office Hours

Wednesday, Friday 4PM - 6PM


Solid programming/development skills (Assembly, C, C++, Unix) are required for this class. "Operating System", "Compilers", and "Computer Security", are the least prerequisites for this class. In particular, for UTD student

Note for undergraduate students who may be interested in taking this class, please be aware that the class is designed for graduate students, you are encouraged to attend the first lecture and then talk to the instructor.

Course Projects

Course Policy

Grading Policy

Late Policy

No late submission.

Collaboration Policy

Students are encouraged to collaborate, particularly on the course project. But we will limit the team member to at most two students.

Cheating Policy

We will strictly follow the university policy on cheating and plagiarism which is available here. Please avoid. There are also several examples of Scholastic Dishonesty If you have any questions regarding this issue, please contact the instructor.