Bits of Wisdom: HOPL III's History of Erlang

Lots of interesting bits in Joe Armstrong's HOPL III paper on the History of Erlang. The paper is available as a PDF for ACM members or others wanting to purchase just that paper. Not available outside the walled garden of the ACM, that I know of.

Anyway on to just some of the bits I appreciated.

In 1985, when I joined the Lab, SPOTS had finished and DOTS was starting. I asked my boss Bjarne D¨acker what I should do. He just said “solve Ericsson’s software problem.” This seemed to me at the time a quite reasonable request...

We were... lucky in being the first group of people in the company to get our hands on a UNIX operating system, which we ran on the VAX. What we were supposed to do was “to find better ways of programming telephony” (a laudable aim for the members of the computer science lab of a telecommunications company). This we interpreted rather liberally as “program basic telephony in every language that will run on our Unix system and compare the results.” This gave us ample opportunities to a) learn new programming languages, b) play with Unix and c) make the phones ring...

“small languages” were thought desirable:

“Large languages present many problems (in implementation, training etc) and if a small language can describe the application succinctly it will be preferable.”...

My own contribution to LOTS was to program POTS. This I did first in Smalltalk and then in Prolog. This was fairly sensible at the time, since I liberally interpreted Bjarne’s directive to “solve all of Ericsson’s software problems” as “program POTS in Smalltalk.”

(Aside: POTS is Plain Old Telephony System. LOTS is a system supporting Lots of pOTS. On to more bits...)

Erlang began to change rapidly. We now had two people working on the implementation (Robert and myself) and a large user community (three people). We would add features to the language and then try them out on our users. If the users or implementors liked the changes, they stayed in. If the users disliked the changes or if the implementation was ugly, the changes were removed. Amazingly, the fact that the language was changing under their feet almost every day didn’t particularly bother our users. We met our Bollmora users once or twice a week for about six months. We taught them programming, they taught us telephony and both sides learned a lot...

I always considered the morning coffee break to be the key forum where the brilliant ideas you had on the way to work were trashed and where all the real work was done. It was in these daily brainstormings that many a good idea was created. It’s also why nobody can quite remember who thought of what, since everybody involved in the discussions seems to remember that it was they who had the key idea...

In designing Erlang, we wanted to abstract all hardware as reactive objects. Objects should have “process semantics;” in other words, as far as the software was concerned, the only way to interact with hardware was through message passing. When you send a message to a process, there should be no way of knowing if the process was really some hardware device or just another software process. The reason for this was that in order to simplify our programming model, we wanted to model everything as processes and we wanted to communicate with all processes in a uniform manner. From this point of view we wanted software errors to be handled in exactly the same manner as hardware errors. So, for example, if a process died because of a divide by zero it would propagate an {’EXIT’,Pid,divideByZero} signal to all the processes in its link set. If it died because of a hardware error it might propagate an {’EXIT’,Pid,machineFailure} signal to its neighbors. From a programmer’s point of view, there would no difference in how these signals were handled.

The average increase in productivity was a factor of 8. This factor and the conclusion of the report were highly controversial and many theories were advanced to explain away the results. It seemed at the time that people disliked the idea that the effect could be due to having a better programming language, preferring to believe that it was due to some “smart programmer effect.” Eventually we downgraded the factor to a mere 3 because is sounded more credible than 8. The factor 3 was totally arbitrary, chosen to be sufficiently high to be impressive and sufficiently low to be believable. In any case, it was significantly greater than one, no matter how you measured and no matter how you explained the facts away...

1989 also provided us with one of our first opportunities to present Erlang to the world outside Ericsson. This was when we presented a paper at the SETSS conference in Bournemouth. This conference was interesting not so much for the paper but for the discussions we had in the meetings and for the contacts we made with people from Bellcore. It was during this conference that we realised that the work we were doing on Erlang was very different from a lot of mainstream work in telecommunications programming. Our major concern at the time was with detecting and recovering from errors. I remember Mike, Robert and I having great fun asking the same question over and over again: “what happens if it fails?”— the answer we got was almost always a variant on “our model assumes no failures.”We seemed to be the only people in the world designing a system that could recover from software failures...

I started writing the emulator myself in C but soon Mike interfered and started making rude comments about my code. I hadn’t written much C before and my idea of writing C was to close my eyes and pretend it was FORTRAN. Mike soon took over the emulator, threw away all my code and started again. Now the Erlang implementor group had expanded to three, Mike, Robert and myself. Mike wrote the inner loop of the emulator very carefully, since he cared about the efficiency of the critical opcodes used for concurrent operations. He would compile the emulator, then stare at the generated assembler code, then change the code compile again, and stare at the code until he was happy. I remember him working for several days to get message sending just right. When the generated code got down to six instructions he gave up...

The AXD301 was a spectacular success. As of 2001, it had 1.13 million lines of Erlang code contained in 2248 modules. If we conservatively estimate that one line of Erlang would correspond to say five lines of C, this corresponds to a C system with over six million lines of code. As regards reliability, the AXD301 has an observed nine-nines reliability —and a four-fold increase in productivity was observed for the development process...

Just when we thought everything was going well, in 1998, Erlang was banned within Ericsson Radio AB (ERA) for new product development. This ban was the second most significant event in the history of Erlang: It led indirectly to Open Source Erlang and was the main reason why Erlang started spreading outside Ericsson...

...projects that were already using Erlang were allowed to continue but had to make a plan as to how dependence upon Erlang could be eliminated. Although the ban was only within ERA, the damage was done. The ban was supported by the Ericsson technical directorate and flying the Erlang flag was thereafter not favored by middle management...

Since we had spent the last ten years designing and building fault-tolerant telecoms devices, we turned our attention to Internet devices, and our first product was a fault-tolerant e-mail server called the mail robustifier. Architecturally this device has all the characteristics of a switching system: large numbers of connections, fault-tolerant service, ability to remove and add nodes with no loss of service. Given that the Bluetail system was programmed by most of the people who had designed and implemented the Erlang and OTP systems, the project was rapidly completed and had sold its first system within six months of the formation of the company...

The plans within Ericsson to wean existing projects off Erlang did not materialise and Erlang is slowly winning ground due to a form of software Darwinism. Erlang projects are being delivered on time and within budget, and the managers of the Erlang projects are reluctant to make any changes to functioning and tested software.

The usual survival strategy within Ericsson during this time period was to call Erlang something else. Erlang had been banned but OTP hadn’t. So for a while no new projects using Erlang were started, but it was OK to use OTP. Then questions about OTP were asked: “Isn’t OTP just a load of Erlang libraries?”—and so it became “Engine,” and so on.

After 2002 some of the surviving Bluetail members who moved to Nortel left and started a number of 2nd-generation companies, including Tail-F, Kreditor and Synapse. All are based in the Stockholm region and are thriving.

Outside Sweden the spread of Erlang has been equally exciting. In the UK, an ex-student of mine started Erlang Consulting, which hires out Erlang consultants to industry. In France, Processone makes web stress-testing equipment and instant-messaging solutions. In South Africa, Erlang Financial Systems makes banking software. All these external developments were spontaneous. Interested users had discovered Erlang, installed the open-source release and started programming. Most of this community is held together by the Erlang mailing list, which has thousands of members and is very active. There is a yearly conference in Stockholm that is always well attended.

Recently, Erlang servers have begun to find their way into highvolume Internet applications. Jabber.org has adopted the ejabberd instant messaging server, which is written in Erlang and supported by Process-one.

The ban within Ericsson ERA was apparently a power struggle, with the side in favor of the ban promoting popular languages like C++ and Java, and "off the shelf" components. The results are speaking for themselves now, with Joe Armstrong back at Ericsson, and Erlang thriving in more of their products.

Luke Gorrie wrote of this a couple months ago in LtU...

I'm not an Ericsson insider but I can tell you that this is old history from when C++/Java/UML were hyped towards executives. Today Erlang is bigger than ever within Ericsson and they're shipping major new products on it. They even managed to hire Joe Armstrong back.

Joe Armstrong wrote of this about a year ago on the erlang list...

In 2004 I rejoined Ericsson, after 6 years, working in start-ups and research.

Had things changed? - Yes

Had the ban (which caused us to leave) worked? - No.

"Was Erlang still banned?" - I asked, "Don't ask, just use it", they said...

Ericsson has no corporate policy, regarding Erlang.

Corporate policy is way more abstract than talking about individual technologies - nobody in above middle management knows how a phone works...

We (Ericsson) have a number of products written in Erlang - these earn stuff called MONEY...

We (and this includes me) are developing "secret-project-number-1" and "secret-project-number-2" etc. these we hope will one day earn MONEY