Security and root
The Linux kernel contains about 10M lines of code that runs as root. The X server contains 16M lines of code of which a lot runs as root. If youre looking for security holes, where do you have better odds? There is no technical reason requiring the X server to run as root. In the name of cross platform compatibility, the current X Server runs as root in order to program the video hardware from user space. Linux has a solution for this. You put the privileged code into a device driver and run the user space application without privilege. The privileged device driver for an average video card runs about 100KB. That is a whole lot less code to audit than 16M lines.