Anne van Kesteren: The problem here is that OpenID does not use a proper HTML parser (not even one that follows the HTML 4 specification). <head> et cetera are actually required
In order for a Consumer to know the Identity Provider authoritative for an Identifier, the End User must add markup to the HEAD section of the HTML document located at their URL.
FWIW, I have a similar issue with the fact that the specs don’t distinguish between temporary and permanent redirects. From § 3.2.1:
Consumers MUST canonicalize the Identifier URL, following redirects, and note the final URL. The final, canonicalized URL is the End User’s Identifier.
In OpenId 2.0, when making a Yadis file available for discovery, one can always set things up so that an HTTP response is given that include an X-XRDS-Location header. This avoids making relying parties parse HTML.
My id does a temporary redirect (302) to my Yadis file if the accept HTTP header includes application/xrds+xml. Despite this response explicitly being a temporary redirect, many OpenID libraries (such as the JanRain ones) treat my Yadis file as my identity.
Because I had the same problem as Sam, I no longer do conditional redirection, but content negotiation instead, with a .var file at the root of my Web site (Apache), with the following rules (among others):
Sam: the OpenID libraries are functioning according to spec: they are actually required to normalise your identifier by following those redirects. If you want to do yadis/XRDS based discovery, there are two options here:
serve an X-XRDS-Location header at your identity URL that points at the XRDS document.
perform content negotiation as suggested by Alkarex.
James: I said “I have a similar issue with the fact that the specs don’t distinguish between temporary and permanent redirects.” In other words, I take issue with the specs as currently written.