It’s just data

HTTP Best Practices

Rob Howard: We're going to try and get back to the point where full RSS feeds can be offered. While there are lot of good RSS clients out there, there are also plenty of bad ones that don't honor all the HTTP codes.

To be fair, most of the HTTP headers involved are optional, and most developers are not accustomed to dealing with the protocol at this level.

Despite being optional (which is fair given the range of applications HTTP supports), many of the HTTP features were explicitly designed to solve the problems posed by applications such as syndication.  After all, If implemented properly, the check for new content is an "infinitesimal" burden.

What may help is guides and compliance test suites.