SSH - relatively secure and very useful.
If you have your SSH port open, filtering by IP/Subnet might not be a bad idea. Certainly keep up with the latest patches or versions for your OS as there are vunerabilities found in SSH implementations from time to time.
Windows users can also tunnel Remote Desktop or VNC sessions over SSH for secure remote access...get a desktop from your home PC from a secure, encrypted channel from work for example. You can also copy files without having to set up an FTP server.
»www.portknocking.org/ is an interesting idea that SSH users who run no other services might want to look into.
Use UltraVNC. It has an encrypted plugin built into the GUI that someone wrote that goes up to 128-bit encryption (of STREAMING data). It's very nice... and port 22 is one of the few 'off' ports corporations will allow to pass through the firewall for say, home access.
Ever met someone from Microsoft Q/A? ...EXACTLY!
btw, i think there are vulnerabilititties with the file xfer function, but just turn it off when you're done using it on the host machine=o.