That is, is there anything that stops them 'stealing' and forging my (or your) address?

Not really. As Tony says, there's a certain amount of double-checking that you can do to ensure that the envelope and headers match, and that the mail came from a server responsible for the relevant domain. This last check is not recommended, however.

If you want to make sure that an email actually came from the person it claims to, then you'll need to rely on digital signatures. PGP (or GPG) will do this for people. There's nothing in the SMTP protocol that will authenticate servers against each other.

You could (fairly easily) modify TMDA (for example) to require that emails be signed by the sender. It would be a PITA for people who didn't understand this stuff.