The End-to-End Argument Principle

Deciding where to place functions in a distributed system is a very difficult proposition. One principle for guiding the placement of functions among the modules of a distributed computer system is known as the "end-to-end argument", as put forth by David P. Reed, et al in a classic paper entitled "End-to-end Arguments in System Design" and published in 1984 in the ACM Transactions in Computer Systems.

The principle suggests that functions placed at low levels of a system may be redundant or of little value when compared with the cost of providing them at that low level. Examples include bit error recovery, encryption, duplicate message suppression, recovery from system crashes, and delivery acknowledgement. Low level mechanisms to support these functions are justified only as performance enhancements.

I don't want to oversimplify the principle, but basically when you have a chain of modules or layers of software, possibly spread over multiple computer systems in a network, it can be counterproductive (and possibly cause a significant lose of productivity) for each module or layer to replicate a complete set of error detection, redundancy, and other housekeeping functions, when the optimal solution would be to optimize all those functions over the entire length of the chain (hence, "end-to-end"). In some cases, it may be easier and more efficient to compensate for errors and data losses and other anomolies at the ends of the processing chain.

I do think the end-to-end argument is very worth discussing for the design of any distributed system, but I'm not completely convinced that this argument is always valid. That said, in practice the principle is more likely to flush out "bad ideas" than to scuttle the occasional great idea. My model is what I would call the "Usually End-to-End Argument".