Explaining Usenet

Once in a while, people will ask questions in a Usenet forum that have nothing to do with the actual forum topic. One of our colleagues asked about the nature of Usenet itself in our comp.databases.pick newsgroup. I’ve posted a lot of info like this in CDP in the past but it seems to get lost in the ether, so now I’m posting it here.

The original inquiry:
> I’m the first to admit I have no real understanding of the
> topology of Usenet. I use Google groups these days, and
> I don’t even understand where all these messages are stored.
> But, I have to ask… How do you change the name of the topic
> or subject when you respond?

Let’s start at the beginning. E-mail is based on specs that are over 20 years old. RFC 821 is the Request For Comments for SMTP, the Simple Mail Transport Protocol. Similarly, Usenet is based on a spec almost as old, RFC 977 for NNTP, the Network News Transfer Protocol. Consider the difference is that E-mail is more of a one-to-one protocol where Usenet is more of a many-to-many protocol. Unfortunately over the years, e-mail has turned into a many-to-many protocol through common use. In my opinion it’s been seriously stressed and abused – our problems with spam and viruses are a direct result of the popular use of e-mail as a broadcast medium. I might write about this in another blog entry.

So Usenet was the original peer to peer network if you compare it to today’s file swapping networks. I post a note to my server and it broadcasts my note to a bunch of others, who in turn broadcast it to others. Google happens to have servers that get the "feed". This is different from centralized forums or web servers where a document is located in one place, or email where a single document makes its way from one server to another with no trail left behind (we hope – look at headers in your emails, it’s scary how many Received lines there are, which indicate the number of servers your email has passed through). Sometimes we will see a reply to a comment where we didn’t see the original post – that’s because the original note for some reason didn’t get proliferated all the way through servers that the final consumer/reader needs. (And that’s what causes outcries like this.)

Companies like Google (and DejaNews prior to them) get a feed and wrap the text in HTML for their audience of consumers. Google adds value in many ways to get you to use their interface, which they brand name as Google Groups, but they are neither Usenet nor CDP, just one of thousands of server hosts that get a Usenet feed. Other hosts also provide services for their consumers, like storing lots of history, or eliminating spam and viruses from postings – some do better than others, some are free some for free. Google tends to get a feed for just about every Usenet forum available, but some NNTP servers only host some groups, and many of the free ones are read-only.

Note also that not all NNTP servers are public. Many companies have a system running an NNTP server for internal use or as their public interface for their client base. It’s just like a HTTP web server, FTP server, or an email server, it just runs on a different socket port. Rather than a hierarchy like comp.databases.pick, they might have something like nebula.supportedproducts.designbais, nebula.supportedproducts.mvdotnet. Some companies prefer NNTP servers to web forums because so many people like to get their news the way they want.

To add a bit of confusion, Google does host their own Groups which are not released out to "real" Usenet servers, but they look exactly like any other Usenet forums that Google makes available. Think of it like your local news station who gets news from the AP wire and other syndicators, but they also have their own local news team to make their program unique enough to get you to stay with them rather than some other station that does nothing but broadcast reports that someone else produces.

I use a client program that was intended for Usenet (Agent). There are many like it. The Usenet protocol, like email, allows us to change subjects. The web interface presented by Google Groups does not. They present forum postings to you in a completely customized web application. The fact that the source of their data is Usenet is purely coincidental, so we can’t expect them to completely expose all Usenet features to all users. I would guess that changing the subject line would be an enhancement suggestion for their perpetual Beta interface. (At one point I wrote a Usenet client in Pick BASIC – it did support changing subjects. Ha!)

You may have another option though. As a real implementation of convergence of technologies, the way cell phones have cameras and web browsers, Google allows us to receive and send Group postings via email, and that’s the way a lot of people get CDP, so they think it’s just a big email list. I post to my Usenet server, Google gets the feed, they broadcast as email, someone responds to Google, they in turn broadcast the feed back so that I can see the response in my Usenet reader. Well, if you get your feed from Google via e-mail you should be able to change the subject there.

I’ll also add that some standard e-mail programs have been enhanced to process NNTP through the same interface as SMTP/POP3. One prime example is Microsoft Outlook Express (not Microsoft Outlook from the Office package!) All this does is further blur the line between e-mail and news, because it allows people to subscribe directly to a Usenet server and get Usenet postings delivered to their mailbox just like personal mail. Through here too, you can change the subject of a forum posting as easily as you’d modify the subject of an e-mail item. I highly discourage anyone from using Outlook Express – it’s a prime target for viruses and it’s usually very susceptible to accepting them as well.

Other pages about Usenet:

• Dictionary.com
• WikiPedia
• Lots of Usenet FAQs at faqs.org
• Harley Hahn’s Introduction to Usenet (explains way it’s comp dot databases doc pick …)
• Some discussions in CDP: Jan 2006 , Jan 2006 , Oct 2005 … and there have been many others.

I hope this was helpful. Please feel free to ask questions and I’ll try to answer, though my usual theme here is to give you enough info for you to go figure it out on your own – or you can send me a purchase order if you’d like consultation.