While setting up a virtual image on my NAS today I wanted to remove all the X stuff from Ubuntu. Here's how:
sudo apt-get purge -y libx11.* libqt.* libgtk.* sudo apt-get autoremove -y