Erlang began to change rapidly. We now had two people working
on the implementation (Robert and myself) and a large user community
(three people). We would add features to the language and
then try them out on our users. If the users or implementors liked
the changes, they stayed in. If the users disliked the changes or if the
implementation was ugly, the changes were removed. Amazingly,
the fact that the language was changing under their feet almost every
day didn’t particularly bother our users. We met our Bollmora
users once or twice a week for about six months. We taught them
programming, they taught us telephony and both sides learned a lot...
I always considered
the morning coffee break to be the key forum where the brilliant
ideas you had on the way to work were trashed and where all the
real work was done. It was in these daily brainstormings that many
a good idea was created. It’s also why nobody can quite remember
who thought of what, since everybody involved in the discussions
seems to remember that it was they who had the key idea...
In designing Erlang, we wanted to abstract all hardware as reactive
objects. Objects should have “process semantics;” in other
words, as far as the software was concerned, the only way to interact
with hardware was through message passing. When you send
a message to a process, there should be no way of knowing if the
process was really some hardware device or just another software
process. The reason for this was that in order to simplify our programming
model, we wanted to model everything as processes and
we wanted to communicate with all processes in a uniform manner.
From this point of view we wanted software errors to be handled
in exactly the same manner as hardware errors. So, for example, if
a process died because of a divide by zero it would propagate an
{’EXIT’,Pid,divideByZero} signal to all the processes in its
link set. If it died because of a hardware error it might propagate an
{’EXIT’,Pid,machineFailure} signal to its neighbors. From a
programmer’s point of view, there would no difference in how these
signals were handled.
The average increase in
productivity was a factor of 8.
This factor and the conclusion of the report were highly controversial
and many theories were advanced to explain away the
results. It seemed at the time that people disliked the idea that the
effect could be due to having a better programming language, preferring
to believe that it was due to some “smart programmer effect.”
Eventually we downgraded the factor to a mere 3 because is
sounded more credible than 8. The factor 3 was totally arbitrary,
chosen to be sufficiently high to be impressive and sufficiently low
to be believable. In any case, it was significantly greater than one,
no matter how you measured and no matter how you explained the
facts away...
1989 also provided us with one of our first opportunities to present
Erlang to the world outside Ericsson. This was when we presented
a paper at the SETSS conference in Bournemouth. This conference
was interesting not so much for the paper but for the discussions
we had in the meetings and for the contacts we made with people
from Bellcore. It was during this conference that we realised that
the work we were doing on Erlang was very different from a
lot of mainstream work in telecommunications programming. Our
major concern at the time was with detecting and recovering from
errors. I remember Mike, Robert and I having great fun asking the
same question over and over again: “what happens if it fails?”—
the answer we got was almost always a variant on “our model
assumes no failures.”We seemed to be the only people in the world
designing a system that could recover from software failures...
I started writing the emulator myself in C but soon Mike
interfered and started making rude comments about my code. I
hadn’t written much C before and my idea of writing C was to close
my eyes and pretend it was FORTRAN. Mike soon took over the
emulator, threw away all my code and started again. Now the Erlang
implementor group had expanded to three, Mike, Robert and
myself. Mike wrote the inner loop of the emulator very carefully,
since he cared about the efficiency of the critical opcodes used for
concurrent operations. He would compile the emulator, then stare at
the generated assembler code, then change the code compile again,
and stare at the code until he was happy. I remember him working
for several days to get message sending just right. When the
generated code got down to six instructions he gave up...
The AXD301 was a spectacular success. As of 2001, it
had 1.13 million lines of Erlang code contained in 2248 modules. If we conservatively estimate that one line of Erlang would
correspond to say five lines of C, this corresponds to a C system
with over six million lines of code.
As regards reliability, the AXD301 has an observed nine-nines
reliability —and a four-fold increase in productivity was observed
for the development process...
Just when we thought everything was going well, in 1998, Erlang
was banned within Ericsson Radio AB (ERA) for new product
development. This ban was the second most significant event in
the history of Erlang: It led indirectly to Open Source Erlang and
was the main reason why Erlang started spreading outside Ericsson...
...projects that were already using Erlang were allowed
to continue but had to make a plan as to how dependence
upon Erlang could be eliminated. Although the ban was only within
ERA, the damage was done. The ban was supported by the Ericsson
technical directorate and flying the Erlang flag was thereafter
not favored by middle management...
Since we had spent the last ten years designing
and building fault-tolerant telecoms devices, we turned our attention
to Internet devices, and our first product was a fault-tolerant
e-mail server called the mail robustifier.
Architecturally this device has all the characteristics of a switching
system: large numbers of connections, fault-tolerant service,
ability to remove and add nodes with no loss of service. Given that
the Bluetail system was programmed by most of the people who
had designed and implemented the Erlang and OTP systems, the
project was rapidly completed and had sold its first system within
six months of the formation of the company...
The plans within Ericsson to wean existing projects off Erlang
did not materialise and Erlang is slowly winning ground due to a
form of software Darwinism. Erlang projects are being delivered on
time and within budget, and the managers of the Erlang projects are
reluctant to make any changes to functioning and tested software.
The usual survival strategy within Ericsson during this time
period was to call Erlang something else. Erlang had been banned
but OTP hadn’t. So for a while no new projects using Erlang were
started, but it was OK to use OTP. Then questions about OTP
were asked: “Isn’t OTP just a load of Erlang libraries?”—and so
it became “Engine,” and so on.
After 2002 some of the surviving Bluetail members who moved
to Nortel left and started a number of 2nd-generation companies,
including Tail-F, Kreditor and Synapse. All are based in the Stockholm
region and are thriving.
Outside Sweden the spread of Erlang has been equally exciting.
In the UK, an ex-student of mine started Erlang Consulting,
which hires out Erlang consultants to industry. In France, Processone
makes web stress-testing equipment and instant-messaging solutions.
In South Africa, Erlang Financial Systems makes banking
software. All these external developments were spontaneous. Interested
users had discovered Erlang, installed the open-source release
and started programming. Most of this community is held together
by the Erlang mailing list, which has thousands of members and
is very active. There is a yearly conference in Stockholm that is
always well attended.
Recently, Erlang servers have begun to find their way into highvolume
Internet applications. Jabber.org has adopted the ejabberd
instant messaging server, which is written in Erlang and supported
by Process-one.
