jump to

UTD Software Languages Security Lab

Kevin Hamlen


Current externally funded research projects:

  • ENCORE: ENhanced program protection through COmpiler-REwriter cooperation (NSF, 2015–2018): Software can be secured by its producers (e.g., by equipping compilers with advanced code analyses so that they generate more secure programs from source code), or by its consumers (e.g., by retrofitting the compiled binary code with extra security enhancements). Unfortunately, both approaches suffer a trade-off: producers enforce policies that generalize to all consumers, not those specific to individual consumers; whereas consumers lack source code, and therefore cannot enforce policies that require that information. This project invents a middle-ground technology whereby producers can release software amenable to late-stage retrofitting by consumers. This will facilitate a new paradigm of more secure, intended-to-be-customized software.
  • Automated, Binary Evidence-based Attribution of Software Attacks (AFOSR, 2014–2018): A comprehensive defense against cyber-threats cannot be limited to merely detecting and weathering unrelenting streams of cyber-attacks. Staunching the flood of attacks requires defenders to identify who is responsible for the attacks, so that more aggressive action can be taken against persistent threats. Unfortunately, attacks are increasingly anonymous; attackers enjoy myriad strategies for concealing network routes. To better attribute anonymized attacks, our research is advancing the state-of-the-art in binary reverse-engineering, honey-potting, and secure data mining to uncover attacker-identifying clues in seas of binary-level attack data.

    In April 2014 our work made international news when we applied our research to create an antidote for the famous Heartbleed bug. See UTD's press release for more information.

  • Binary Retrofitting of Untrusted Software for Security (ONR, 2013–2019): Security-sensitive organizations, such as military agencies, face a difficult challenge when it comes to selecting good software. The most up-to-date, feature-filled, and well-tested software tends to be commercial products whose developers prioritize sales over security. Our research is developing algorithms that can automatically retrofit commercial, binary software with augmented security dictated by product consumers. This offers security-conscious consumers the best of both worlds—they get the rich feature sets that come with mass-produced software, plus the iron-clad, organization-specific security required for mission-critical operations.

Completed projects:

  • Language-based Security for Polymorphic Malware Protection (NSF CAREER, 2011–2017): This project developed hybrid static-dynamic code analyses for malware detection and defense. It recasts traditionally static programming language-based security approaches, such as strong type-checking, model-checking, and in-lined reference monitoring, as hybrid static-dynamic algorithms that detect and prevent malicious program behaviors as the untrusted code executes. SIGNAL magazine, FEDcyber.com, and ACM Technews ran a news story about the project, and UTD publicized it in a press release.
  • Securing Web Advertisements (NSF Trustworthy Computing, 2011–2016): Web ads introduce unique challenges for end-to-end software security. This project developed tools and algorithms for malicious ad detection and trust negotiation at the level of ad developers, ad distributors, and ad recipients. The sidebar of this press release summarizes the project.
  • Reactively Adaptive Malware (AFOSR Active Defense, 2010–2014): Traditional polymorphic malware undergoes undirected (random) mutation as it propagates so that no two instances look exactly alike. This makes the malware harder to detect. This project examined more powerful directed mutation strategies that allow next-generation malware to reactively learn and adapt to deployed malware defenses. Anticipating this next generation of malware is critical for keeping pace with the cyber-security arms race. UTD devoted a press release to the project in May 2010. Subsequently, our work has been reported on by hundreds or thousands of news outlets. (See the "In the News" sidebar to the right for a few.) See also this UTD press release.
  • Certified, Automated In-lined Reference Monitors (AFOSR Young Investigator, 2008–2011): In-lined Reference Monitors (IRMs) automatically modify untrusted code to make it provably safe, rather than merely examining it to try to decide its safety purely statically. The approach is more powerful than purely static analysis, and more flexible than traditional OS- or VM-level execution monitoring. This project developed IRM and machine-verification systems for ActionScript (Flash) binaries, Java binaries, and x86 native code binaries.
  • Secure, Peer-to-peer Data Management (NSF EAGER, 2009–2011): Cloud computing is an increasingly essential paradigm for supporting management of large databases and distributed computations. This project develops fully decentralized data integrity, confidentiality, and privacy enforcement algorithms for cloud computing based on structured peer-to-peer networking protocols and economic theories of utility optimization and risk.

TV News Coverage of my Research

CBS 11 News (April 14, 2014)

CW33 Nightcap News (April 15, 2014)

Full Story at CW33 NightcapTV


The following is a list of research papers and theses that I've authored, co-authored, or supervised. Each is provided in PDF form.

Conference & Workshop Publications

All of the following conference and workshop publications are peer-reviewed. Acceptance rates are provided whenever they are known (except for invited papers that underwent a separate peer review process that does not pertain to the conference's general acceptance rate).

Journal Articles

The following are peer-reviewed journal papers I've authored or co-authored.