Protecting program execution is a critical issue in today's computing systems. Even with all the efforts and dramatic advances in computer system protection, system attacks continue to evolve to explore existing and new vulnerabilities in computer systems. In

this thesis, we propose several ecient and practical methods to validate program control flow integrity as extra layers of system protection. In general, the approach to control flow integrity checking is to detect anomalies of program behavior given that control flow

information is closely coupled with program execution correctness and can be consider as the DNA of a program. It is hard, if not impossible, for two programs to expose identical control flows during program execution. Therefore, control flow integrity checking can effectively prevent malicious code implants from executing.

This thesis proposes three new protection schemes based on control flow integrity checking, each with its own assumptions of hardware and/or application scenarios. The first study, IBMON (Indirect Branch MONitor), utilizes existing hardware features to

eciently observe unusual control flow transfers and check them for any abnormality. Prototype systems for proof of concept have been successfully implemented on three different system platforms to demonstrate its efficacy. By using the hardware features,

IBMON can effectively protect a system from malicious control

ow modification trans-parently to the target applications. We have successfully built prototype systems on real machines using several processors. Although the prototype system exhibits the best

performance among other control flow validation mechanisms, it still incurs moderate performance overhead. We further propose IBF-Cache, an enhanced IBMON system with special hardware support, to minimize the performance overhead associated with IBMON. Although it requires an extension of existing processors, the cost is negligible and the run-time of IBMON is reduced to virtually zero.

Control flow validation is also an effective approach to detect malicious program because control flow transfers in program are unique. There are limitations of detection method for malicious software in traditional anti-virus program. Anti-virus software

cannot consistently detects the new generation of malicious program such as polymorphic program. The thesis also explores the extension of the control flow validation mechanism

to augment the ability of anti-virus software for detecting polymorphic malware. The RCFI (Recent Control Flow Inspection) system is proposed to validate recent control flow transfers in run-time with enhanced hardware features. The RCFI system can effectively detect polymorphic malware that uses various obfuscation tools to evade the detection mechanism of static binary scanning.