I probably met with 70 to 80 people over the last few days. Some were from large companies/organizations, many more were from small companies or informal organizations. I have a large number of specifics that I will inject into the wiki over the course of the upcoming weekend, but for now, some overall themes.
The final overall theme that seemed consisitent is the importance of security in an API. Given the number of people blogging in conferences and public locations, simple md5 or sha1 hashes of passwords is not sufficient. What's needed is a solution that is not only easy to implement, but also one that is easy to administer.
I probably met with 70 to 80 people over the last few days. Some were from large companies/organizations, many more were from small companies or informal organizations. I have a large number of specifics that I will inject into the wiki over the course of the upcoming weekend, but for now, some overall themes.
First, I won't pretend that this sample size was representative or scientific. But the feedback seems to me to be amazingly consistent across a diverse set of people, and bears further reflection.
The most striking input was about the wiki. It's biggest strength and weakness is that it is inclusive and chaotic. Shortly after the 0.1 snapshot was published, the wiki example evolved in a radically different direction - replacing most elements with attributes. This stayed for a few weeks and then snapped back. This is leading many to the conclusion that the best course of action is to simply sit things out for now. This is OK with me, if it truly represents an "either way is fine with me" vote.
What I want to avoid is people with something worth contributing only doing so after it is too late. The only solution I know of to this is time. Publishing infrequent snapshots, with clear rationale for every change, and encouraging experimentation.
Another clear example of this is the inability of the project to come to agreement over a name. The wiki was amazingly efficient in ferreting out the trademark issues with the name Echo. It may have been similarly efficient in doing so with Atom, but many worry that it may have been a bit too efficient, and perhaps even hasty in this process. There are people who have expressed a willingness to donate professional legal resources towards resolving this issue - but are reluctant to do so if such an effort would be disregarded.
To be clear, my favorite name so far was, and continues to be, 'pie'. But I am clearly in the minority on this, and I fully accept that. Beyond that, any name on the current FinalVote list (with the additions of Echo and Atom, but not Necho) are acceptable to me - subject only to a legal clearance on the name. In my opinion, what's best for the project is to let go, and submit this list to people who are willing to dedicate the resources to research this properly - and in so doing, be willing to accept the outcome of the result.
The final overall theme that seemed consistent is the importance of security in an API. Unsurprisingly there is widespread agreement that sending passwords in the clear is not sufficient. What was more surprising is the input that, given the number of people blogging in conferences and public locations, md5 or sha1 hashes of passwords is also not sufficient. These, too, can be sniffed and replayed. What's needed is a solution that is not only easy to implement, but also one that is easy to administer. Not only for large, hosted, blogging providers, but also for small individually hosted weblogs.
Pie gets my vote as well.
HTTP digest auth, whilst using hashes, is safe from replay attacks.
Sam,
I'd like to add my voice to those who feel the wiki is too chaotic to track. According to the formerlyEcho blog there are 700 pages which is way too much to track unless you attempt to do it fulltime. My current stance is that I'll wait till you folks produce specs and if assuming you have a reasonable comment period I'll send in my feedback.
As for security I agree with Simon that something like HTTP Basic and Digest Authentication seems to hit the sweet spot.
Interesting. I guess the security issues could use a little focus in the near future then.
I'm not sure how infrequent snapshots should be - isn't it time for another?
I think what's needed is:
a) A chair who can declare consensus and some way to distinguish it from discussion and proposals on the Wiki. Someone needs to say "we're done here, let's move on" and have it stick. (Presumably this would be Sam, but I don't see him doing it.)
b) A major WikiWeeding.
Mark, has anyone tried asking CornerHost if they would enable the digest auth module? In 1.3 all I had to do was add:
LoadModule digest_auth_module /usr/lib/apache/1.3/mod_auth_digest.so
to httpd.conf.
"has anyone tried asking CornerHost if they would enable the digest auth module?"
That's not really the point. Although CornerHost might, many other hosts would refuse (I've had a lot of hassle with a lot of hosts over other modules), thus making it useless for a lot of people.
Could HTTP Digest auth perhaps be implemented in a CGI script, bypassing the need for the server to support it completely?
My instincts tell me this should be possible, but I've not done CGI for a while now. I spend most of my time in mod_perl land...
Is a specific authentication method something that has to be standardized for Atom?
There are several authentication methods, and the strongest one supported by both client and server should be used. If there are no standard ways to do that, it could be added to the introspection API.
The solution that appeals most to me is to leave the password in-the-clear in the API, and protect the entire transaction with SSL (or SSH tunneling).
Setting up your server with a certificate (even a self-signed one) is a royal pain, however, and not a barrier that most people will even try to hurdle.
"The wiki was amazingly efficient in ferreting out the trademark issues with the name Echo. It may have been similarly efficient in doing so with Atom, but many worry that it may have been a bit too efficient, and perhaps even hasty in this process. There are people who have expressed a willingness to donate professional legal resources towards resolving this issue - but are reluctant to do so if such an effort would be disregarded."
Sam, are you saying that some lawyer is willing to donate time to secure the Atom trademark, but only if the community agrees to accept that name in advance?
Carlos, the community seemed to coalese around Echo until it was rejected for trademark infrigement reasons. I kinda suspect that in this case, the concern was valid. Then the community seemed to coalese around Atom until it was apparently rejected for (again) trademark infringement reasons. This time I wasn't so sure that the concern was as valid. Now the community is in the process of picking another name, and quite frankly I'm not sure that when this is done I will be able to confidently say that there is (or is not) a trademark issue with that name.
What I would prefer to see done at this point is to hand somebody a short list of names, and ask them to return back with a single selection that has been cleared by their legal staff.
Yes, I have received offers to do that. What I don't have is offers to go speculatively research names that may later be rejected by the community.
I agree with Ken about not inventing new security mechanism. I also agree with Aaron's points a and b, about someone taking consensus and the wiki weeding. It's getting more than a bit confusing now. Most seems to be about the names. There are so many name pages, not sure which ones to look at anymore.
I can see Sam why you might not want to do this, because this could open the door for some pushback. But I'm also concerned in that this effort is losing momentum because folks are not sure whats going on. The wiki is quite hard to follow.
Tough one. If you stay pure wiki, then people are unhappy about the chaos (including me). But if you start enforcing some order, you're going to get hit with other things. Not an easy situation.
As a start, you could follow through with your last comment -- okay, this is what you think should be done with names? Then pursue this.
Personally, I'd go with Pie -- something, anything at this point.
The weblogging consortium I talked about long ago doesn't seem like such a bad idea at this time -- a smaller group of empowered people, stake holders primarily, but with open avenues of communication with the community.
Shelley,
The weblogging consortium I talked about long ago doesn't seem like such a bad idea at this time -- a smaller group of empowered people, stake holders primarily, but with open avenues of communication with the community.
It's interesting to see how your perspective has changed. I've always been against the design-by-anarchy approach of the wiki because I'd seen how difficult coming up with a technical specification can be, even with a handful of parties with different goals (from internal work to W3C committees) and imagining that you could scale the process to "anyone with a web browser who is interested in syndication" just didn't make sense to me.
I agree the Wiki has been a good place to get peoples ideas and opinions to the fore but if progress is going to be made I personally believe a traditional standards process makes more sense. A core of people who debate and edit the spec in an open (e.g. WWW-TAG) or closed (most other standards groups) manner. Stakeholders can provide comments on the evolving spec at regular milestones and join in the discussions when they have something to contribute.
Even if Sam decides to keep the wiki, I'd appreciate regular milestones where we have a wide window of time to respond to the spec. I'm sure this will be appreciated by folks like Ted Leung who mentioned that it looks like it will be September before he'll be caught up with Atom.
Shelley and Ken,
There was definitely a concern with having a secure authentication mechanism that is easier to implement. Now Basic authentication isn't secure, and Digest is, but it is non-trivial to implement. That is, in Apache, digest needs to be turned on for the server via the config files, and many providers don't do that, mine for example. Also, if Digest or Basic are used, you have to rely on Apache doing the authentication for you, unless you are willing and able to run your service as an Apache module, another non-trivial task. Also, many hosting services do not allow you use .htaccess files or they restrict what can be done in them.
These are obstacles to a 'simple' weblog deployment. The requirements I heard expressed were a system that:
1. Is processed via the CGI.
2. Is a challenge response authentication system.
3. Works without running in an apache module, without requiring changing the server config, and without having the ability to use .htaccess files.
Now that may seem a tad restrictive, but that is the condition that a majority of sites live under, and regardless of what you think of their environment, they deserve a decent security mechanism.
Hopefully someone can come up with a solution that meets the above requirements.
Actually I think that given the fact that there will be some REST or SOAP API, we should stay with what the HTTP spec gives us and that's basic auth, digest auth and all this combined with SSL (and maybe SSL based client certificates, too). I don't see any value in defining something else beside this in the AtomAPI - there is a problem, there is a solution, use the solution. The not widespread installation of digest auth isn't an argument: digest auth won't get widespread use if people don't ask for it. And regardless of any API provisions, people will allways be able to shoot their own feet, so just keep it simple and stay with what is there.
The area of security and authentication protocols is one where you can do much more wrong than right, so keep that stuff to those who know what they are doing.
There are existing solutions that meet all of the criteria specified above.
I also want to acknowledge the process concerns that everybody have expressed. To those who suggest that there should be more leadership on my part, if you are predisposed to trust my judgment, then please be willing to trust the way in which I have been shepherding this process.
As I said in the blog post above, the key ingredient is time. When all is said and done, everyone who wishes to participate will agree that they have had ample opportunity to do so.
Tomorrow, it will have been one month since Joe Gregorio first published his API RFC. The latest draft is revision 6. Feel free to comment on the wiki or mailing list, or even on your own weblog - just please leave a pointer in one of the above two places so that everybody else can find it.
re: "The not widespread installation of digest auth isn't an argument: digest auth won't get widespread use if people don't ask for it."
This is a very, very dangerous argument. You are essentially saying that Atom should be used to push a particular technology into the mainstream, and that's wrong. Atom will have a difficult enough time getting traction; it doesn't need any extra obstacles or showstoppers.
Many people including myself believe that (aside from political issues) the #1 technical issue that killed RSS 1.0 was the belief that it should be the shining example that would get everyone to start using RDF. I absolutely do not want to put this project in the position of forcing people to upgrade their existing infrastructure in order to use it. That kind of snotty architecturally-holier-than-thou approach is why people hate engineers.
It's one thing to tell developers (like Dare) that they have to write some code to parse xml:base. There's a problem (relative links), we want a solution, and all possible solutions require writing some code. Dare can do it; every developer in his position can do it. (Some already have.) It's quite another thing to tell end users that they have to upgrade their infrastructure. It's beyond their capability, it's a showstopper, and if your only answer to them is "well, screw you, you should change hosts/be smarter/run your own server/stick with crappy security", that's unacceptable. It is possible to give those users better security, and as Joe said, they deserve it.
It is unfortunate that this will probably involve some wheel re-invention. If anyone has a better idea that doesn't involve giving the big ol' middle finger to CGI-and-no-htaccess users, I'm listening.
This isn't an attack, but Sam, your answer did raise another thought:
I know you want the wiki because you want open participation, contribution, and to forestall future pushback, but have you considered that the wiki is actually preventing people from giving feedback and participating? What happens for those people who just can't make their way through it, or keep up with the changes? There is a survival of the wiki-fittest thing going on here a bit.
Dare, actually this isn't a change of mind for me -- I wanted an open standards process, where everyone had input, and decisions were made in public. But long ago I also recommended in my weblog that there be a core of stake holders to pull things together when needed.
Just thoughts, Sam. NOT criticism -- just another viewpoint.
Shelley, that was most definately not an attack. It is, however, a criticism. A valid one. One that I promise to address.
One thing I would like to foster in this process is an environment where people are allowed - and perhaps even encouraged - to change their minds.
Not particularly related... just an interesting bit of trivia. My first stadandards related activity was WebDAV.
Well, as long as you don't cross me out, Sam ;-)
WebDAV, eh? Cool. My first standards effort was for Boeing, in conjunction with IBM, for PDES/STEP. Even presented a paper on beta-testing IBMs information repository as part of this effort for the old IBM user group, which I can't remember the name. SHARE, I think? I remember that IBM would have an open bar every night for the attendees.
Chicago eventually banned the conference because of the rowdiness of the happy conference attenders.
Not me, of course.
Naturally.
Sam, I posted my thoughts on "process issues" a while ago here: http://www.dynamicobjects.com/d2r/archives/002148.html
As far as I can see, the situation is that whoever can spend more time in the wiki can take over a discussion and effectively steer it any way they want. This, rather than being open is the law-of-the-loudest (not to mention that at times decisions apparently "just happen"--like the name change to atom even though the process to choose name itself wasn't yet agreed on). And yet I think it's clear that there are a number of stakeholders on this process, including people from MT, blogger, livejournal, etc., and others, like you, Mark, etc. Others, like Joe Gregorio, have become stakeholders by contributing a lot of work to it.
And why is it important what those stakeholders think?
For example, you mentioned the problem of security on the API. Ok. There's another issue that should precede that: ie., coming to the conclusion on whether a single transport method for the API will be used or not. This goes to the heart of interoperability and to whether small developers will end up worse off with the AtomAPI or not. And yet the Wiki still has discussions for SOAP, XML-RPC, and REST.
My point is: if Blogger and MT and Livejournal agree that they'd like to implement, say, a SOAP interface, and nothing else, then adding another interface to the discussion only creates confusion and the potential for lack of interoperability, which everyone (I hope) would like to avoid.
As I mentioned in my posting, I really think that techniques like those used by Apache are necessary, and other ideas from open source like the Benevolent Dictator concept. Not only to add direction to the discussion, not only to prevent the process from varying wildly (the attributes change you mentioned is a good example), but also to make it more transparent and, indeed, open.
Diego - I remember that post, and had stashed away that URL for a rainy day. It isn't that day yet.
Let's look at your concrete example: hypothetically let's suppose that we were to discover that there was a preexisting standard for doing authentication with SOAP that was both easy to implement and easy to administer in the target environments that we are exploring. Let me be clear so that there is no confusion: there is such a standard, but the easy to implement and administer have yet to be proven, so this truly meant only to be a plausible and yet hypothetical postulate at this point.
With that disclaimer understood: how do you think the community would have reacted had I prematurely declared that SOAP was the answer? In fact, lets take a look at the larger issue: does it make sense to determine the transfer protocol at all before we understand the requirements? Will asynch posting be a significant use case?
Now let me do something that I rarely do: self-promotion. I have been involved with a number of "real" standards efforts: WebDAV, ECMAScript (a.k.a. JavaScript), CLI (a.k.a. .Net's runtime), and C#. Some of these have been quite large and quite contensious. I also have been highly involved with a number of high profile open source projects, most specifically Apache (where I am on the board of directors) and PHP (where I was elected a member of the group). Finally, I have been employed by a large software vendor (IBM) for 22 years and have been directly involved with the development of commercial shipping products.
So, when the time comes, rest assured that I will be quite willing to step in and make an executive decision. The time is coming close for that on the naming issue. Had I initially stated the name was "pie" - end of discussion, there would have been quite a bit of resentment. Now a number of people who originally hated the name and would have strongly pushed back are more open to the idea. Amazing how that works.
However, what I am more likely to do in this particular case, and in general, is to give those who actually implement the thing (which in this case translates to actually doing the appropriate legal clearance) a range of options and community input and the freedom to make a choice within that constrained range. This does have the affect of giving the person who actually implements the decision a bigger vote than others. And, quite frankly, I don't have a problem with that.
Diego, please include UserLand in your lists. Lots of users of Radio and Manila, and lots of feeds. I'm sure MT and Blogger would like to see you factor out UserLand, but if you want consensus, you have to be inclusive.
BTW, I still believe the best bet in APIs is the MetaWeblog API, and we should press Blogger on what they need to support it, and what the delay is about. I've asked directly, but they don't seem to want to explain. Maybe it would help if more people asked.
Sam, what can I say except: excellent! :-)
In particular re: API as an example, the Wiki is definitely good to flesh out ideas, I agree. But afterwards a choice has to be made--so that issues specific to the implementation can be fleshed out. Otherwise a lot of energy is spent defining behavior for something that potentially won't be used--and replicating that effort across different parallel tracks.
And: if this is the way it works in general, it's not very clear I think. It is quite possible that it's just me (i.e., not seeing the obvious!) but in any case, adding something along the lines of your comment to the wiki as a statement of how the process works would make sure that
a) those that are contributing are aware that this can (and most probably will) happen in some cases
b) those that are not contributing and maybe are not fully involved can see that there is a process and that therefore a goal will be achieved, and that it will be backed up by those in a position to implement it.
Ok. I'll shut up now. :)
Dave,
I'm not sure why you're bringing up pressuring Blogger to support the MetaWeblog API in this thread since this rather offtopic to the current discussion at hand. As for what's wrong with the MetaWeblog API, you know what its problems are because I have pointed trhem out to you before at
http://www.kuro5hin.org/story/2003/6/29/22931/0682
in a post which you have carefully avoided comment on.
If you insist on pushing your MetaWeblog API, the least you could do is take a look at the current draft of the ATOM API at http://bitworking.org/rfc/draft-gregorio-06.html and say why that is inferior to the MetaWeblog API.
PS: Sam, I apologize for engaging in this offtopic discussion on your blog.
Dare and Dave, neither comment is off topic.
The topic is inclusiveness. Thing to add to the discussion: Warnock's Dilemma, the Rules of Open-Source Programming and in particular, Sunir's corollaries.
I have been coaching Joe and others on the ways to build a community. Being brilliant and correct doesn't do it. Finding ways to get others to directly participate (not merely vicariously, but to directly provide input) is the key to building a successful community.
Joe is getting lots of feedback. And both of you are welcome to participate.
Dare, someone must like the MWA, because it is the one of the most widely deployed API in bloggerdom. Maybe if you looked closer you'd find reasons to implement it. I think the security issue is orthogonal and as you can see in this thread there is no obvious answer that works for everyone.
Diego, thanks -- I think some of the people here are interested in routing around and exclusion, I'm glad to get that issue out in the open and glad you're not one of them.
Joe, I have been following the iterations of your API. I think it's fun to start with a clean slate, but I'm not sure if it's a productive use of all our time to start from scratch.
I've said my piece on REST many times before, that the juice in XML-RPC is that it does the serialization and deserialization once so you don't have to keep rewriting the code. It was the same basic feedback I had about the Technorati API. My observation is that people who use XML-RPC understand what I'm saying and people who don't, don't.
In the end I would love to hear what you think are the advantages of your approach to the MWA, in practical terms. I don't really care about the aesthetic issues of REST advocates.
Sam,
Joe is getting lots of feedback. And both of you are welcome to participate.
My feedback: http://www.kuro5hin.org/story/2003/8/3/133458/0512
Dave,
I've said my piece on REST many times before, that the juice in XML-RPC is that it does the serialization and deserialization once so you don't have to keep rewriting the code.
Much like SOAP except that with SOAP you don't have to run away from the fact that you are primarily exchanging XML documents. Thus you can have your cake and eat it too.
If anything, I'd like to know what the plans are for a SOAP-based version of the ATOM API because this lets people who just want to do object<->XML mapping do their thing while folks who want to access XML infosets directly can do their thing.
Dare, A month ago Blogger proposed a SOAP based version of the ATOM API, and put an example on the wiki.
What would be better, IMHO, if is somebody were to propose that the custom header be replaced with a standard header that is designed to do digest authentication of passwords. Is that something that you could help with?
For those that aren't familiar with the history to this point, please see section 4 "Proposed Consensus", of this page http://www.intertwingly.net/wiki/pie/XmlRpcDiscussion to see where the REST API came from. There was a raging debate over REST vs XML-RPC vs SOAP. A consensus was reached and the votes cast are still on that page for all to see. The consensus is:
"Let's first create an API in the REST Architectural Style. The SOAP and XML-RPC APIs can then easily be derived from that."
Sam,
Thanks for the Warnocki's Dilemma link; it was enjoyable. It helped me to take into account the comments about no specs when Joe is up to Version Six of the Editing API.
Those calling for "alpha maleness" seem to overlook that it is your wiki and you could take it and go home anytime, rather than let the play continue.
On the other hand, there could be a problem with too many words (rather than too many pages). To paraphrase another linked quotation, which was a paraphrase on the original quotation by someone quite fond of his own words, "Give it refactoring or give it death."
And, while paraphrasing let me add that "Yes, wikis suck, just remember that all the other forms of online collaboration suck more." Be mindful that
there is a difference between saying "chaos is bad" and "chaos is uncomfortable."
There may have been a mistake made in the "Note to First Timers" on the Front Page of the project wiki. It emphasized RefactorOk and DeleteOk annotations. In other wikis the operating principle is your writing could be refactored or deleted.