Do you have more details on this?
Try this link: http://www.arrl.org/digital-modes It introduces RTTY, PSK-31 and others. RTTY is the old protocol and it just transmits to any radio listening in. PSK-31 is more like a computer modem in that a radio receiving the signal can check for transmission errors and ask for a re-transmit of portions of messages that do not come over cleanly.
As per the FCC, if you are in the US, you can make up your own protocol so long as the specs are "published" somewhere. So there are a number of other systems out there.