CS 6301-006: Language-based Security

Course Information

Title: CS 6301-006: Language-based Security
Course Registration Number: 89299
Times: MW 1:00-2:15
Location: ECSN 2.112
Instructor: Dr. Kevin Hamlen (hamlen AT utdallas)
Instructor's Office Hours: MW 2:15-3:15, ECSS 3.704

Course Summary

This course will introduce and survey the emerging field of Language-based Security, in which techniques from compilers and programming language theory are leveraged to address issues in computer security. Topics include:

  1. Certifying Compilers
  2. In-lined Reference Monitors
  3. Software Fault Isolation
  4. Address Space Randomization
  5. Binary Obfuscation
  6. Web Scripting Security
  7. Information Flow Control

The aim of the course is to allow each student to develop a solid understanding of at least one of these topics, along with a more general familiarity with the range of research in the field. In-course discussion will highlight opportunities for cutting-edge research in each area. If you do research involving software security, this course will provide you with an array of powerful tools for addressing software security issues. If you do research involving programming languages or compilers, this course will show you how to take techniques that you already know and apply them to a new and important problem domain. If your career involves management or development of high-assurance software systems, this course will provide a comparative analysis of traditional versus language-based techniques.

The course is open to Ph.D. students and Masters students. Interested undergraduates should see the instructor for permission to take the course.

Suggested (non-mandatory) prerequisite: CS 6371 Advanced Programming Languages (or taken concurrently)


Homework (30%): For the first 11 weeks of the course, students will complete selected programming exercises from the online text Software Foundations.

Quizzes (30%): Most classes will begin with a short quiz testing the students comprension of an assigned reading for the day. Questions will typically be multiple choice or short answer. The easier questions will be designed to test whether the student has read the material at all, and the harder ones will test deeper understanding of more subtle points.

Class Participation (10%): Students are expected to come to class having read the assigned paper(s), and prepared with questions, critiques, and discussion topics. Regular attendance and class participation will count 10% towards their grades in the course.

Project (30%): Students will work individually or in a small team for the last 5 weeks of the course to complete a small project using the Coq theorem-prover. A typical project will involve implementing and formally verifying a standard algorithm of the student's choosing. Students will present their projects in class, with the presentation counting toward their project grade.


In our study of the Coq theorem proving system, we will be using the following online textbook:

For those who wish to explore Coq in greater depth (e.g., for developing their projects), the following book by the Coq developers is highly recommended:

Additionally, the following two texts available through the UTD library may be useful for general background on type theory and computer security, respectively:

Tentative Course Schedule

Date Topic Assigned Reading(s) Coq Exercises
Program-Proof Co-development
Lecture 1:
Mon 8/26
Introduction to Language-based Security
Lecture Slides
Lecture Notes


  1. nandb (*)
  2. andb3 (*)
  3. factorial (*)
  4. blt_nat (**)
  5. plus_id_exercise (*)
  6. mult_S_1 (**)
  7. zero_nbeq_plus_1 (*)
Lecture 2:
Wed 8/28
Functional Programming with Coq
Lecture Notes
Install Coq and complete the first exercise (nandb).
No Class:
Mon 9/2
No class: Labor Day
Lecture 3:
Wed 9/4
Programming Proofs in Coq
Lecture Notes
Software Foundations: "Basics" chapter
Lecture 4:
Mon 9/9
Polymorphism and Higher-order Functions
Lecture Notes
P. Hudak. Conception, Evolution, and Application of Functional Programming Languages. ACM Computing Surveys, 21(3):359-411, May 1989.
  • Required sections: Abstract, Introduction, 2.1, 2.3, and 2.4
Language-based Security Foundations
Lecture 5:
Wed 9/11
The Science of Security
Lecture Slides
J. Bau and J. C. Mitchell. Security Modeling and Analysis. IEEE Security & Privacy 9(3):18-25, 2011.


  1. and_true_elim2 (**)
  2. basic_induction (**)
  3. double_plus (**)
  4. destruct_induction (*)
  5. evenb_n__oddb_Sn (**)
Lecture 6:
Mon 9/16
Type Theory and Security
Lecture Slides
F. B. Schneider, G. Morrisett, and R. Harper. A Language-based Approach to Security. Informatics: 10 Years Back, 10 Years Ahead, Lecture Notes in Computer Science, 2000.
Lecture 7:
Wed 9/18
Formally Verified Compilation
Lecture Slides
X. Leroy. Formal Verification of a Realistic Compiler. Communications of the ACM, 52(7):107-115, 2009.
Lecture 8:
Mon 9/23
Introduction to Model-checking M. Müller-Olm, D. Schmidt, and B. Steffen. Model-Checking: A Tutorial Introduction. In Proc. 6th Int. Sym. Static Analysis (SAS), pp. 330-354, September 1999.
  • Required sections: 1–4.2 and 5
  • I will not quiz you on the parts about fixed point theory in Section 3.3, but do read the paragraph on Computational Tree Logic.
Lecture 9:
Wed 9/25
Model-checking Device Drivers T. Ball, V. Levin, and S.K. Rajamani. A Decade of Software Model Checking with SLAM. Communications of the ACM, 54(7):68-76, 2011.
In-lined Reference Monitoring
Lecture 10:
Mon 9/30
Theory of IRMs
Lecture Slides
F. B. Schneider. Enforceable Security Policies. ACM Transactions on Information and System Security, 3(1):30-50, 2000.


  1. snd_fst_is_swap (*)
  2. list_funs (**)
  3. list_exercises (***)
Lecture 11:
Wed 10/2
Aspect-oriented Programming and IRMs
Lecture Slides
M. Jones and K. W. Hamlen. Disambiguating Aspect-oriented Security Policies. In Proc. 9th Int. Conf. Aspect-Oriented Software Development (AOSD), pp. 193-204, March 2010.
Lecture 12:
Mon 10/7
Type-based Certification of IRMs
Lecture Slides
K. W. Hamlen, G. Morrisett, and F. B. Schneider. Certified In-lined Reference Monitoring on .NET. In Proc. 1st ACM SIGPLAN Workshop on Programming Languages and Analysis for Security (PLAS), pp. 7-16, June 2006.


  1. poly_exercises (**)
  2. split (**)
  3. map_rev (***)
Lecture 13:
Wed 10/9
Model-checking IRMs K. W. Hamlen, M. M. Jones, and M. Sridhar. Aspect-oriented Runtime Monitor Certification. In Proc. 18th Int. Conf. Tools and Algorithms for the Construction and Analysis of Systems (TACAS), pp. 126-140, March-April 2012.
Software Fault Isolation
Lecture 14:
Mon 10/14
Return-oriented Programming
Lecture Slides
E. J. Schwartz, T. Avgerinos, and D. Brumley. Q: Exploit Hardening Made Easy. In Proc. 20th USENIX Security Symposium, 2011.


  1. silly_ex (**)
  2. sillyex1 (*)
  3. beq_nat_true (**)
Lecture 15:
Wed 10/16
Binary Randomization
Lecture Slides
H. Shacham, M. Page, B. Pfaff, E.-J. Goh, N. Modadugu, and D. Boneh. On the Effectiveness of Address-Space Randomization. In Proc. ACM Conf. Computer and Communications Security (CCS), pp. 298-307, 2004.
Lecture 16:
Mon 10/21
Google Native Client B. Yee, D. Sehr, G. Dardyk, J. B. Chen, R. Muth, T. Ormandy, S. Okasaka, N. Narula, and N. Fullagar. Native Client: A Sandbox for Protable, Untrusted x86 Native Code. Communications of the ACM, 53(1), 2010.


  1. b_timesm (*)
  2. gorgeous_sum (**)
  3. double_even (*)
Lecture 17:
Wed 10/23
Randomization & SFI for COTS Software R. Wartell, V. Mohan, K. W. Hamlen, and Z. Lin. Securing Untrusted Code via Compiler-Agnostic Binary Rewriting. In Proc. 28th Annual Computer Security Applications Conf. (ACSAC), pp. 299-308, December 2012.
Lecture 18:
Mon 10/28
Certifying the Certifier
Lecture Slides
G. Morrisett, G. Tan, J. Tassarotti, J.-B. Tristan, and E. Gan. RockSalt: Better, Faster, Stronger SFI for the x86. In Proc. 33rd ACM SIGPLAN Conf. Programming Languages Design and Implementation (PLDI), pp. 395-404, June 2012.


  1. and_assoc (*)
  2. or_distributes_over_and_2 (**)
  3. contrapositive (**)
  4. dist_not_exists (*)
Lecture 19:
Wed 10/30
Control-flow Integrity
Lecture Slides
M. Abadi, M. Budiu, Ú. Erlingsson, and J. Ligatti. Control-Flow Integrity: Principles, Implementations, and Applications. In Proc. ACM Conf. Computer and Communications Security, pp. 340-353, 2005.
ROP & Malware
Lecture 20:
Mon 11/4
Frankenstein V. Mohan and K. W. Hamlen. Frankenstein: Stitching Malware from Benign Binaries. In Proc. USENIX Workshop on Offensive Technologies, pp. 77-84, 2012.
Web Security
Lecture 21:
Wed 11/6
Lecture Slides
Z. Li, K. Zhang, Y. Xie, F. Yu, and X. Wang. Knowing Your Enemy: Understanding and Detecting Malicious Web Advertising. In Proc. ACM Computer and Communications Security, pp. 674-686, 2012.
Lecture 22:
Mon 11/11
JavaScript Security
Lecture Slides
M. Johns and S. Lekies. Tamper-Resistant LikeJacking Protection In Proc. 16th Int. Sym. on Research in Attacks, Intrusions and Defenses (RAID), pp. 265-285, 2013.


Information Flow
Lecture 23:
Wed 11/13
Introduction to Type-based Information Flow Control
Lecture Slides
A. Sabelfeld and A. C. Myers. Language-Based Information-Flow Security. IEEE J. Selected Areas in Communications, 21(1):5-19, 2003.
Lecture 24:
Mon 11/18
Info Flow for Java
Lecture Slides
A. C. Myers. JFlow: Practical Mostly-Static Information Flow Control. In Proc. 26th ACM Sym. Principles of Programming Languages (POPL), pp. 228-241, 1999.
Lecture 25:
Wed 11/20
Dynamic vs. Static IFC A. Sabelfeld and A. Russo. From Dynamic to Static and Back: Riding the Roller Coaster of Information-Flow Control Research. In Ershov Memorial Conf., 2009.
No Class:
Mon 11/25
No class: Fall break
No Class:
Wed 11/27
No class: Fall break
Mon 12/2
Course Wrap-up
Project Presentations

Presentation Schedule

  1. Hamlen
  2. Hoffman and Sanderlin
  3. Araujo, Wang, and Xu
Wed 12/4
Project Presentations

Presentation Schedule

  1. Chatterjee, Ghosh, and Karanjai
  2. Irwin, Krishnamurthy, and Roy
  3. Kuni, Pradhan, and Strohbeck
No Class:
Mon 12/9
Class Cancelled
No Class:
Wed 12/11
Class Cancelled