CS 6301-002: Language-based Security

Course Information

Title: CS/SE 6301-002: Language-based Security
Course Registration Number: 82024/82025
Times: MW 1:00–2:15
Location: ECSN 2.120
Instructor: Dr. Kevin Hamlen (hamlen AT utdallas)
Instructor's Office Hours: MW 2:15–3:15, ECSS 3.704

Course Summary

This course will introduce and survey the field of Language-based Software Security, in which techniques from compilers and programming language theory are leveraged to address issues in computer security. Topics include:

  1. Certifying Compilers
  2. In-lined Reference Monitors
  3. Software Fault Isolation
  4. Address Space Randomization
  5. Formal Methods
  6. Web Scripting Security
  7. Information Flow Control
  8. Cyber-deceptive Software Engineering

The aim of the course is to allow each student to develop a solid understanding of at least one of these topics, along with a more general familiarity with the range of research in the field. In-course discussion will highlight opportunities for cutting-edge research in each area. If you do research involving software security, this course will provide you with an array of powerful tools for addressing software security issues. If you do research involving programming languages or compilers, this course will show you how to take techniques that you already know and apply them to a new and important problem domain. If your career involves management or development of high-assurance software systems, this course will provide a comparative analysis of traditional versus language-based techniques.

The course is open to Ph.D. students and Masters students. Interested undergraduates should see the instructor for permission to take the course.

Suggested complementary course: Interested students may wish to consider taking CS 6371 Advanced Programming Languages before/after this course or concurrently, since it presents material that supplements and enhances several of the above topics.


Homework (30%): For the first 10 weeks of the course, students will complete a series of programming exercises assigned through eLearning. Background material helpful for completing the exercises can be found in the online textbook Software Foundations.

Quizzes (30%): Most classes will begin with a short quiz testing the students comprension of an assigned reading for the day. Questions will typically be multiple choice or short answer. The easier questions will be designed to test whether the student has read the material, and the harder ones will test deeper understanding of more subtle points.

Class Participation (10%): Students are expected to come to class having read the assigned paper(s), and prepared with questions, critiques, and discussion topics. Regular attendance and class participation will count 10% towards their grades in the course.

Project (30%): Students will work individually or in a small team for the last 6 weeks of the course to complete a small project. A typical project will involve implementing and/or formally verifying a language-based security mechanism covered in the class or related to class material. Students conducting research are encourage to choose projects that synergize with their research activities. Students will present their projects in class, with the presentation counting toward their project grade.


In our study of the Coq theorem proving system, we will be using the following online textbook:

For those who wish to explore Coq in greater depth (e.g., for developing projects), the following book by the Coq developers is highly recommended:

Additionally, the following two texts available through the UTD library may be useful for general background on type theory and computer security, respectively:

On December 2, 2016, a panel of industry, government, and academic cybersecurity experts issued a report urging the former and current U.S. presidents to undertake an array of security education and prevention steps recognized as imperative for the future security of the nation and the world. The report is recommended (optional) reading:

Tentative Course Schedule

Date Topic Assigned Reading(s) Coq Exercises
Program-Proof Co-development
Lecture 1:
Mon 8/21
Introduction to Formal Methods and Secure Software Development
Lecture Slides
Coq Transcript
Assignment 1 due 8/30
(Coq Basics)
Lecture 2:
Wed 8/23
Higher-order Types
Coq Transcript
Software Foundations: "Basics" chapter, up to and including the first two exercises (nandb, andb3).
Lecture 3:
Mon 8/28
Machine-verified Proofs
Coq Transcript
F. Williams. Investigating SANS/CWE Top 25 Programming Errors. Tech. Rep. ICTN 6870, E. Carolina University, 2009.
Lecture 4:
Wed 8/30
Inductive Propositions
Coq Transcript
J. Walden, J. Stuckman, and R. Scandariato. Predicting Vulnerable Components: Software Metrics vs Text Mining. In Proc. 25th Int. Sym. Software Reliability Engineering (ISSRE), pp. 23–33, November 2014.
No Class:
Mon 9/4
No Class: Labor Day Assignment 2 due 9/13
Lecture 5:
Wed 9/6
Proof Tactics
Coq Transcript
P. Hudak. Conception, Evolution, and Application of Functional Programming Languages. ACM Computing Surveys, 21(3):359–411, May 1989.
  • Required sections: Abstract, Introduction, 2.1, 2.3, and 2.4
Lecture 6:
Mon 9/11
Logical Operators
Coq Transcript
Veridrone Source Repository on GitHub
G. Malecha, D. Ricketts, M.M. Alvarez, and S. Lerner. Towards Foundational Verification of Cyber-physical Systems, Invited Paper. In Proc. Science of Security for Cyber-Physical Systems Workshop (SoSCyPS), April 2016.
Lecture 7:
Wed 9/13
Contradiction, Constructivistic Logic, Non-termination
Coq Transcript
no assigned reading
Lecture 8:
Mon 9/18
Inversion, Verifying Total Correctness
FCF Source Repository on GitHub
Coq Transcript
A. Petcher and G. Morrisett. The Foundational Cryptography Framework. In Proc. Int. Conf. Principles of Security and Trust (POST), 2015. Assignment 3 due 9/20
Software Security Fundamentals
Lecture 9:
Wed 9/20
The Science of Security
Lecture Slides
J. Bau and J. C. Mitchell. Security Modeling and Analysis. IEEE Security & Privacy 9(3):18–25, 2011.
Lecture 10:
Mon 9/25
Formally Verified Compilation
Lecture Slides
X. Leroy. Formal Verification of a Realistic Compiler. Communications of the ACM, 52(7):107–115, 2009. Assignment 4 due 9/27
Lecture 11:
Wed 9/27
Machine Code Validation
Lecture 12:
Mon 10/2
Software Model-checking
Lecture Slides
M. Müller-Olm, D. Schmidt, and B. Steffen. Model-Checking: A Tutorial Introduction. In Proc. 6th Int. Sym. Static Analysis (SAS), pp. 330–354, September 1999.
  • Required sections: 1–4.2 and 5
  • I will not quiz you on the parts about fixed point theory in Section 3.3, but do read the paragraph on Computational Tree Logic.
Assignment 5 due 10/4
(Tactics & Logic)
Code-reuse Attacks and Defenses
Lecture 13:
Wed 10/4
Return-oriented Programming
Lecture Slides
E.J. Schwartz, T. Avgerinos, and D. Brumley. Q: Exploit Hardening Made Easy. In Proc. 20th USENIX Security Symposium, 2011.
Lecture 14:
Mon 10/9
Artificial Diversity
Lecture Slides
H. Shacham, M. Page, B. Pfaff, E.-J. Goh, N. Modadugu, and D. Boneh. On the Effectiveness of Address-Space Randomization. In Proc. ACM Conf. Computer and Communications Security (CCS), pp. 298–307, 2004. Assignment 6 due 10/23
(Inductive Propositions)
Lecture 15:
Wed 10/11
Control-flow Integrity
Lecture Slides
M. Abadi, M. Budiu, Ú. Erlingsson, and J. Ligatti. Control-Flow Integrity: Principles, Implementations, and Applications. In Proc. ACM Conf. Computer and Communications Security, pp. 340–353, 2005.
Lecture 16:
Mon 10/16
Binary Stirring
Lecture Slides
R. Wartell, V. Mohan, K.W. Hamlen, and Z. Lin. Binary Stirring: Self-randomizing Instruction Addresses of Legacy x86 Binary Code. In Proc. ACM Conf. Computer and Communications Security (CCS), 2012.
Lecture 17:
Wed 10/18
Binary Software Security Retrofitting
Lecture Slides
R. Wartell, V. Mohan, K.W. Hamlen, and Z. Lin. Securing Untrusted Code via Compiler-Agnostic Binary Rewriting. In Proc. 28th Annual Computer Security Applications Conf. (ACSAC), pp. 299–308, December 2012.
In-lined Reference Monitors
Lecture 18:
Mon 10/23
Theory of IRMs
Lecture Slides
F.B. Schneider. Enforceable Security Policies. ACM Transactions on Information and System Security, 3(1):30–50, 2000.
Lecture 19:
Wed 10/25
Aspect-Oriented Programming and IRMs
Lecture Slides
M. Jones and K.W. Hamlen. Disambiguating Aspect-oriented Security Policies. In Proc. 9th Int. Conf. Aspect-Oriented Software Development (AOSD), pp. 193–204, March 2010. Project
Lecture 20:
Mon 10/30
Project Group Meetings G. Morrisett, G. Tan, J. Tassarotti, J.-B. Tristan, and E. Gan. RockSalt: Better, Faster, Stronger SFI for the x86. In Proc. 33rd ACM SIGPLAN Conf. Programming Languages Design and Implementation (PLDI), pp. 395–404, June 2012.
Lecture 21:
Wed 11/1
TBA no assigned reading
Lecture 22:
Mon 11/6
Model-checking IRMs
Lecture Slides
K.W. Hamlen, M.M. Jones, and M. Sridhar. Aspect-oriented Runtime Monitor Certification. In Proc. 18th Int. Conf. Tools and Algorithms for the Construction and Analysis of Systems (TACAS), pp. 126–140, March–April 2012.
Lecture 23:
Wed 11/8
Language-based Web Scripting Security
Lecture Slides
P.H. Phung, M. Monshizadeh, M. Sridhar, K.W. Hamlen, and V.N. Venkatakrishnan. Between Worlds: Securing Mixed JavaScript/ActionScript Multi-party Web Content. IEEE Transactions on Dependable and Secure Computing (TDSC), 12(4):443–457, July–August 2015.
Information Flow Controls
Lecture 24:
Mon 11/13
Intro to Information Flow
Lecture Slides
A. Sabelfeld and A.C. Myers. Language-Based Information-Flow Security. IEEE J. Selected Areas in Communications, 21(1):5–19, 2003.
Lecture 25:
Wed 11/15
Type-based Information Flow Controls
Lecture Slides
A.C. Myers. JFlow: Practical Mostly-Static Information Flow Control. In Proc. 26th ACM Sym. Principles of Programming Languages (POPL), pp. 228–241, 1999.
No Class:
Mon 11/20
No Class: Fall Break
No Class:
Wed 11/22
No Class: Fall Break
Software Cyber Deception
Lecture 26:
Mon 11/27
Honeypots and Honey-patching F. Araujo, K.W. Hamlen, S. Biedermann, and S. Katzenbeisser. From Patches to Honey-Patches: Lightweight Attacker Misdirection, Deception, and Disinformation. In Proc. ACM Computer and Communications Security (CCS), 2014.
Lecture 27:
Wed 11/29
Dynamic Secret Redaction F. Araujo and K.W. Hamlen. Compiler-instrumented, Dynamic Secret-Redaction of Legacy Processes for Attacker Deception. In Proc. USENIX Security Symposium, 2015.
Lecture 28
Mon 12/4
Object Flow Integrity W. Wang, X. Xu, and K.W. Hamlen. Object Flow Integrity. In Proc. ACM Computer and Communications Security (CCS), 2017.
Wed 12/6
Project Presentations
Projects Due
Mon 12/18 11:59pm
Submit projects via eLearning
Project Specifications
Project Deliverables:
  • Written report (LaTeX-generated PDF preferred, but any digital format acceptable): project motivation & problem background (cite any related works), summary of accomplishments, high-level methodology/approach, team coordination (who did what), suggestions for future work
  • Implementation (sources and optionally pre-compiled binaries)
  • README: Step-by-step instructions for launching and testing project (including any external tools or modules).
  • Presentation slides