A vulnerability in the stack protection feature in LLVM’s Arm backend becomes ineffective when the stack protector slot is re-allocated. This was notified as a vulnerability note in the Software Engineering Institute of the CERT Coordination Center.
The stack protection feature is optionally used to protect against buffer overflows in the LLVM Arm backend. A cookie value is added between the local variables and the stack frame return address to make this feature work. After storing this value in memory, the compiler checks the cookie with the LocalStackSlotAllocation function. The function checks if the value has been changed or overwritten. It is terminated if the address value is found to be changed.
If a new value is allocated later on, the stack protection becomes ineffective as the new stack protector slot appears only after the local variables which it is supposed to protect. It is also possible that the value gets overwritten by the stack cookie pointer. This happens when the stack protection feature is rendered ineffective.
When the stack protection feature becomes ineffective, the function becomes vulnerable to stack-based buffer overflow. This can cause the return address to be changed or the cookie to be overwritten itself, thus causing an unintended value to be passed through the check.
This year saw many cases of buffer overflow vulnerabilities. In the June release of VLC 3.0.7, many security issues were resolved. One of the high security issues resolved was about the stack buffer overflow in the RIST Module of VLC 4.0.