Course Overview

CS 7301 is a graduate level (PhD in particular), research oriented, systems and security, seminar class.

The goal of this course is to read, understand, and present the recent advances (which has not been systemized into text book yet) in operating systems security. We will select the most recent papers from both operating systems venues including SOSP, OSDI, USENIX ATC, EuroSys, ASPLOS, and security venues including IEEE S&P, ACM CCS, USENIX Security, and NDSS.

In particular, we will cover

The learning outcome is students will be able to understand the recent advances in operating systems security, the relevent security problems, and how these approaches/solutions are proposed.

Tentative Schedule

Please login your elearning website and download the lecture notes.

Week Topic To Be Covered
0 Course Overview
1 Overivew of the landscape of modern computing, and OS Kernel Key Components
Understanding Kernel Level Attacks
2 Kernel level attack vectors: memory exploits, code based attacks
3 Kernel level attack vectors: rootkits, data-based, DKOM
Out-of-VM Techniques
4 Virtual Machine Introspection
5 Bridging the Semantic-Gap: SBCFI, VMwatcher, Virtuoso, VMST
6 Hardware-based monitoring: Copilot, Vigilare, mGuard, KI-Mon
7 Overshadow, CloudVisor
8 VM Management, forking, replication, deduplication
In-VM Techniques l
9 In-VM monitoring, Access Control
10 Control Flow Integrity (for COTS Binaries), CCFIR, STIR
11 Binary Rewriting, In-lined Reference Monitors
12 Vulnerability Analysis: KINT, KGuard
13 Kernel Data Structure Analysis: KOP, MAS, SigGraph, Value Invariant
14 Kernel ASLR, ROP
15 The papers of our interest
16 The papers of our interest

Reading List

We do not have a text book, but we have the following reading list. Students are required to read, and present one or two of the following papers. Note that the order these papers is of no particular meaning but just for you to track which paper you will be presenting.

Office Hours

Wednesday, Friday 4PM-6PM


Solid programming/development skills (Assembly, C, C++, Unix) are required for this class. "Operating System", "Compilers", and "Computer Security", are the least prerequisites for this class. In particular, for UTD student

Note for undergraduate students who may be interested in taking this class, please be aware that the class is designed for graduate students, you are encouraged to attend the first lecture and then talk to the instructor.

Course Projects

Course Policy

Grading Policy

Late Policy

No late submission.

Collaboration Policy

Students are encouraged to collaborate, particularly on the course project. But we will limit the team member to at most two students.

Cheating Policy

We will strictly follow the university policy on cheating and plagiarism which is available here. Please avoid. There are also several examples of Scholastic Dishonesty If you have any questions regarding this issue, please contact the instructor.