Memoirs: Mango
October 14, 2020After Dolphin, my next gig was at Mangosoft. My wife still jokes that I moved up the evolutionary scale when I went from Clam to Dolphin, then back down when I went to Mango. It was the second most unfortunate name for a company that I worked for, and if you pronounce it as man-go-soft it's even worse.
Mango's thing was a distributed filesystem for Windows. By "distributed" I mean truly flat, symmetrical distribution, no servers, just borrowing and aggregating space from a bunch of clients. It ran on both Windows NT and Windows 95. Filesystem development on NT was not fully supported by Microsoft at the time, though OSR had done an excellent job reverse-engineering it and teaching classes on what they had learned. Similar development on Windows 95 was even trickier. A book came out later, but at that time a lot of the necessary knowledge had been learned the hard way within Mango.
The product at Mango was called Medley. Architecturally, it consisted of two layers: a software DSM (Distributed Shared Memory) abstraction based on work done by John Carter at Utah, and then a filesystem layer built on top of that. Another interesting aspect of the system is that a lot of the code was actually written in a local higher-level language allowing programmers to write asynchronous network-based code in a synchronous style - much like people are (finally!) doing now with coroutines, but a slightly different mechanism. A function in this HLL was converted into an object, local variables became class members, and each continuous piece of logic within the asynchronous whole became a method. These objects were then thrown into a "flow scheduler" to handle all of the network back-and-forth, calling these methods as appropriate when messages were received. It was a pretty elegant approach IMO.
Long after my time there, all of these innovations became the subject of lawsuits against both Skype and Oracle, and were eventually sold to Google. The Oracle lawsuit was particularly interesting, because as far as I can tell the infringing product from Oracle was something they'd already ripped off from Dolphin. We (Dolphin) had discussed with them the idea of using our hardware-assisted shared memory to create a shared cache in an Oracle cluster. They claimed there wouldn't be enough benefit for the project to be worth it and declined to partner with us ... then implemented the very same idea (in even less efficient form sans hardware assist) themselves in the next version of Oracle, claiming credit for it and calling it Cache Fusion. I think this lesser-known bit of history explains some of Oracle's odd interpretations of common terms in the Mango suit, and frankly I also think Dolphin had a stronger claim against Oracle than Mango ever did.
Going back to the "present" in 1997, there are two other things I remember about Mango. One was the amazing debugging team. Yes, there was a separate team that specialized in interpreting NT/95 crash dumps, because that was just such an arcane skill that even developers focused on higher-level code had trouble with it. The amount of information they could squeeze out of a crash dump, despite Microsoft's horrible tooling, was incredible. The other thing that was memorable at Mango was the test system. Given a specification for versions of code and test suites to run, it handled all of the details to allocate and provision an entire cluster, run tests, collect results, and email you the results. We're all familiar with Continuous Integration systems today, but even today it's hard to find good examples for distributed systems. It's even harder to find any that can handle crash tests and collect information about unexpected crashes instead of just hanging, but Mango had this in 1997. Very impressive.
Unfortunately, there were also some less-ideal things about working at Mango. One is that they used ClearCase for version control, which was pretty cool in terms of functionality but hard to keep running. We had one incident where it blew up so badly that hardly anyone was able to get work done for a whole month. Secondly, despite the "cool factor" and some good initial press, the actual market interest in Medley was rather low. This was one of several projects underscoring another important lesson from my startup years.
When everything becomes about that One Big Contract that will save the project/company, you're probably doomed.
I don't even remember who that one big customer was supposed to be - Japanese, I'm pretty sure, maybe Nissan - but the whole thing seemed too reminiscent of the Fort Huachuca affair at Encore. The last thing that really soured my experience at Mango was the way that it was a super cliquish place. All high-level designs for the next-gen product were very closely controlled by a few insiders (mostly people who had worked together since DEC), excluding even people who had highly relevant experience or perspective such as I felt I had from Clam and Dolphin. There were some real problems with what they were designing, but instead of trying to learn from others' hard-won experience the insiders kept going with their own favorite but non-workable solutions. No wonder the project failed and Mango turned into a patent troll.
As you can probably tell, before long I was feeling pretty dissatisfied with how things were going. My tenure there wasn't as short as my tenure at Sequoia had been (seven months vs. two) but I ended up going "back to the future" with some of my other Clam friends elsewhere.