I would highly suggest to the author (or anyone else trying this) to install the Hacker's Keyboard [1]. It gives you a five row keyboard that makes living in a terminal so much easier.
I've been running Debian and U untu chroots on all of my Android devices going back to the G1. It actually works very well. One key for me was making sure to bind mount the sdcard inside of the chroot environment so I have access to all of my local files. As far as X goes, I find the vnc solution works okay provided you aren't trying to watch videos but for programs like Writer and Geany, it's fine. The X server you can download from the market needs a lot of work to be useful but hopefully that'll happen.