This week Google released Protocol Buffers to much acclaim, receiving coverage on slashdot, osnews and other places. I must admit that I haven't looked too closely at Protocol Buffers implementation. This is because I read as far into the documentation to discover that there's a set number of primitive types that can be encoded onto the wire. This basically equates to yet another adaption of the Sun XDR data encoding method used in the original RPC.

I must admit that I did do a little astroturfing on the Protocol Buffers google group which did actually result in a few hits to the einet web site. It didn't equate to any downloads or any actual interest in Argot which is quite disappointing. Obviously astroturfing is not the right way to advertise a new technology.

There's also been a few responses in the blogosphere regarding this release from Ted Neward, Steve Vinoski and Stefan Tilkov. The latter two have simply deferred to Ted's entry which comprises a good analysis of Protocol Buffers. As I read this analysis, I started to think how Argot solved many of the issues pointed out in Protocol Buffers. I was going to respond to Ted's blog, but its probably easier to do this here on my own Blog.

Before getting into responding to Ted's analysis, I should say what Argot is all about. At its most basic Argot is about defining meta data for describing binary data formats and a library for encoding and decoding that data. What makes Argot interesting is that the meta data itself is encoded in binary. This creates the ground work for dynamic data agreement and various other things. There's a full description of Argot here, so I won't bother with the technical implementation details right now.

The first issue Ted found with Protocol Buffers is their claim of language and/or platform-neutrality. I must admit I do claim the same with Argot. I also still regard this as a reasonable claim, and believe Protocol Buffers claim is also reasonable. In my mind, this is not a claim that the solution has complete coverage of all languages. It's simply a claim that no special language tricks or platform specifics have leaked into the data format. Compare this against RMI or other language specific solution. To implement RMI on another language would be difficult as its core data encoding requires Java specific information. I do understand Ted's issue with this and agree that XML's coverage is huge. Currently I have implementations of Argot in Java, C and C#, although I'm currently only working on updating and improving Java. So yes Argot is designed to be language and platform neutral but is currently best supported by Java.

Next Ted reminisces to when binary formats were big. Actually, I don't remember them being big at all. I remember CORBA being the big thing. Other than ASN.1 there has been very few attempts to build structured binary data formats that are independent of RPC or other system. There's some big problems with binary formats being inflexible, tightly-coupled, etc because there has been no real interest in actually solving these problems at a binary level. I will come back to many of these issues soon, as these are the exact problems I built Argot to solve.

Ted moves on to talk about all the advantages of XML. I agree with most of his comments on XML. For what XML was designed to be, it has done very well. And it has stayed true to its initial principles. This is great, but XML is not solving many of the things it was not designed to solve. Yes this makes little sense intentionally. There are many situations where binary data formats are required for speed, size constraints, etc. XML is not the data presentation layer silver bullet. The problem Argot, Protocol Buffers and others are trying to solve is how to move binary data around in a fast, efficient way. Ideally many of the advantages and lessons learned with XML and XML Schema can be applied to binary data formats.

As a side note, one area that I've just started working in the last few months is investigating the issues of interoperability for home area network (HAN) devices in the Energy industry. There's some really interesting work being done to build interoperable systems. They are defining the data formats to ensure all the smart appliances will work together in the home. These are really hard problems which require binary data format agreements so even the smallest of devices can work together. This is not an area where XML will be accepted. Its a market where manufacturing items are costed in cents and the CPU processing power required for XML is not acceptable. However, these industries need to face all the problems that binary solutions provide without the tools and solutions that XML provides. Surely there's a better way than just accepting the old issues in binary data solutions.

To bring it back to Ted's analysis of Protocol Buffers. I'm sure he understands that there's a need for binary data. I think he takes issue that Protocol Buffers is claiming to provide similar benefits of XML in a binary format. I agree completely that Protocol Buffers does not provide most of the benefits of XML. The problem is that in this increasingly connected world where convergence will mean that every appliance in the house might soon be conversing, we really do need a solution. We actually need more people working on this and not just saying how wonderful XML is. I accept that XML is great in all that it achieves, but I don't accept that we can't have most of the benefits of XML in a binary format for where it is required.

Ted also refers to the XML binary InfoSet specification in his analysis. I think I need to be clear here that when I talk about requiring many of the benefits of XML in a binary format, I don't believe that XML binary Infoset or similar is the answer. These solutions still require complex decoding mechanism which once again do not work in many situations. We need the benefits of XML at a really basic binary level.

I'll now move onto the guts of Ted's analysis of Protocol Buffers. I'll now examine some of the biggest short comings of Protocol Buffers identified by Ted. I'll discuss: if Argot solves it, if its required by binary communications and if it can be done effectively.

• Binary is hard to work with. Agreed. However, as already discussed its a requirement in many situations. Lets make it as easy as possible to work with. Argot solves the problem by creating and incredibly flexible meta data structure that can be extended. In fact I'm willing to claim that the Argot meta data format (which has similarities to S-Expressions) is more flexible than XML Schema and anything else out there. I agree that binary data will never be as easy as text for humans to read, however, in the situations where it is needed it is not humans reading it. In these situations we should build tools to make it easier for humans to read. The way to achieve this is through good meta data.
• Binary is tightly-coupled. This is probably the biggest issue with binary. In fact for most situations binary solutions actually need to be tightly-coupled for each individual message. This is where Argot shines the brightest. It is the only solution I know that can allow two peers to dynamically discover each element's meta data. What this means is that a server and client can support multiple versions of the same data structure and pick the format that both will use. I believe this is a very important capability which is needed in binary solutions. It gives clients and servers the ability to change over time and still support sets of data formats. For more details read about the TypeMap in the Argot documentation. I call this allowing tight-binding and loose-coupling. An important note is that I have proved this with a server implementation of around 7kb. It does not take a huge amount of data as the meta data format is itself binary.
• XPath. This is something I believe can be achieved using Argot, however, it is not something I've worked on. At the moment, I don't believe its an area that developers need solutions for when binary is required. Its a nice to have that could be added to Argot tools.
• XSLT. I agree this is a great example of what can be built over the top of XML. I have a long term plan to build something like this, however, it requires a language around Argot which is not yet developed. Once again, this is not a core requirement for working with binary solutions that is required today.
• Structureless parsing. This is an area that I've just started working on. I believe that with enough meta data you can parse anything. In the next release of Argot I've built a parser that is able to read binary data into a form of DOM tree without knowing the meaning of the data. This is an important step to building XPath, XSLT and generic binary data editor tools. This also makes it possible to connect to a server and dynamically discover its interfaces and message formats and then dynamically build requests and read responses. Solving this ability at a binary level should lead to some interesting results when all appliances in the home can publish their message formats and capabilities.

There's probably a lot more that I could add here about Argot. However, I'm trying not to make this a sales pitch. The point I'm trying to make is that XML is a great tool for many situations. However, there are plenty of situations where binary data is required. We need more people understanding what is needed to build binary systems that support many of the features already supported by XML. We also need to recognise that the situations binary data is used is different from XML. The set of tools and use cases are different. I know that it is possible to build tightly-bound and loosely-coupled servers that communicate in binary as I've done it with Argot. Maybe someone can prove me wrong with Argot; I'd enjoy the discussion.