intertwingly

Ah, you people with your fancy-pants programs that are smart enough not to rebuild everything all the time: every comment, every TrackBack ping, rebuilds my RSS file, since it's just another index template as far as Movable Type is concerned.

Sam, I knew you'd come over to the light side eventually. Actually, you don't even need to do a HEAD--the server supports If-None-Match. If you do a GET with "If-None-Match: <etag goes here>" then Joel's server only returns the page if it's changed. We really need to work on getting aggregators to implement this.

Phil, obviously this isn't an issue for Joel, since he's writing the tool that builds the RSS. ;-) But if it's a problem for you, you should bug the MT guys.

Hey, waitaminute, do comments rebuild your index RSS? If not, your slash:comments is lying to me (once I build an aggregator that uses it, that is). If so, and if we say it's okay to do a HEAD/conditional GET every five minutes, then I can affect your bandwidth bill just by commenting. Cool!

Right! Amphetadesk is doing the right thing, getting the head and noticing nothing changed and going away.

NetNewsWire is not. It always refetches the whole thing.

Radio is not. It refetches the whole thing.

The aggregator that claims to be "Mozilla/3.0+(compatible)" -- somebody please tell me what that is -- is doing the wrong thing.

Phil, on intertwingly.net, I have complete control over when my index.rss is rebuilt and where/if it is redirected to.

Furthermore, what you think you see as index.rss2 is a figment of our collective imaginations... no such file exists on my server.

Finally, polling every 5 minutes would probably be considered rude. Fortunately, there are ways of dealing with such rudeness.

I might just be making this up, but I'm sure I saw somewhere that some webservers don't implement HEAD correctly. That might have been in, like, 1995 or something, mind.

Hey, how about response code 410 for the suggestion I saw someplace(?) to indicate that a feed was gone?

Is it a coincidence that HTTP can do this? Nope. 8-)

Ack! Don't deny me my intertwingly!

I was doing my usual conflating several different things: what appeals to me about using ttl is that you can say it's okay to check more often than once an hour, when you think it will be productive to do so. A hot comment thread, getting a new comment every couple of minutes, might want to set a really small ttl until it slows down. But if everybody has enough sense to do If-Modified-Since, even without ttl we could still maybe ease off a bit on the once an hour commandment: I'd rather return four 304s an hour than one 200, and I'd far rather return four 304s on index.rss than have someone force a refresh on index.html because they can't believe I still haven't updated.

Joel, "Mozilla/3.0 (compatible)" is Feedreader. I was using it until today, but coincidentally had decided to dump it since it's too unstable. Finding it doesn't support 304s gave me one reason more.

eh; why are the HEAD requests any use?

GET requests with Etags and (simpler) If-Modified-Since headers should do the same but always keep you down to one request.


- ask

For what it's worth I posted a feature request in NetNewsWire's bug database since there didn't seem to be a feature request for this behavoir yet.

Ask and ye shall receive.

Brent added support for Etags and If-Modified-Since headers to the latest NetNewsWire beta. It's very cool. He added it after hints and pressure from among others Joel, Phil, Sam and Mark; I'll refrain from pointing out that I suggested it to him several months ago, although my justification was to make it faster for people on slow connections. :-) Charles Miller wrote "HTTP Conditional Get for RSS Hackers". In his section about dealing with it on dynamically generated pages he... [more]

Trackback from Ask Bjørn Hansen

Brent added support for Etags and If-Modified-Since headers to the latest NetNewsWire beta. It's very cool. He added it after hints and pressure from among others Joel, Phil, Sam and Mark; I'll refrain from pointing out that I suggested it to him several months ago, although my justification was to make it faster for people on slow connections. :-) I noticed in my logs that a new version of Radio Userland is doing the right thing too. Charles Miller wrote... [more]

Trackback from Ask Bjørn Hansen

Much of the Atom discussion of late has related to bandwidth concerns. The End of RSS (me) Knitwork (me) NNTP for Atom (me) Let's do some analysis. There are two major issues w/ the way the current blogosphere publishes and consumes RSS feeds. HTTP...

Phil Ringnaldaweighs in on Joel's RSS problem. It is worth noting that there is an existing solution that significantly reduces bandwidth without affecting latency or content. It is called HTTP HEAD requests. Amphetadesk already supports it. Check...