After Gluster
August 29, 2018Life's full of odd little twists and turns sometimes. A while back, I wrote Beyond Gluster about the design of a system I'd like to work on once I'm done with Gluster. Unknown to me at the time, there were some shifts occurring that will ultimately lead to me to wind down my involvement with Gluster. The article and the change in my status are not entirely unrelated, in the sense that they both have to do with evolving technology and user needs, but they really are separate threads. As (weak) evidence, I offer the fact that the system I'll be working on is not new and doesn't match the description in Beyond Gluster to any significant degree beyond being a high-scale storage system.
So what will I be working on? It's called Warm Storage. There have been some presentations about it, so I'm not revealing the existence of anything that wasn't public already. It's a building block for multiple higher-level storage systems, including two of the biggest at Facebook (and therefore in the world). I'm particularly going to be working on the "storage node" piece. This is where fun things like erasure coding and repair happen, roughly correspondent to the RADOS layer in Ceph except that I'll be dealing with nice convenient fixed-size blocks. Aggregation of blocks into files, of files into directories, and so on is in other layers. There are a lot of interesting problems that still need to be solved, or need to be solved better, and due to the scale of the system this will allow me to have much greater "impact" than would ever be possible working on Gluster.
I'm not entirely sure when or how I'll stop being involved in Gluster. I'm not going to disappear into an immediate black hole like so many ex-Gluster folks have done. I certainly won't copy ideas from Gluster into directly competing proprietary systems, as has happened a couple of times. I value my Gluster experience and my Gluster friends, so I will continue to support the project and the community in lesser ways for a while yet. As the storage word continues to evolve, I'm sure I'll run into folks at FAST or HotStorage or wherever, and might even work with some again.