Don't Dump On Your Users
May 29, 2018Over the years, I've worked on many systems that rely on either verbose logging or all-inclusive "state dumps" as primary debugging tools. Gluster is such a system. If you look on the mailing list, "please send logs" is practically always the first response to bug reports, and "please perform this incantation and send us the magical statedump file from its secret location" is often the second. I'm sure users really appreciate that. After all, who wants to learn how to debug a problem themselves when they can just ping a developer nine time zones away who might not respond for days because their employer doesn't care about supporting the open-source version of the project? How about almost everyone who's seriously trying to support their own users?
Basically, people who rely on logs and statedumps never grew out of debugging via printf. It might even work for them, because they have inside knowledge and have their own habits/tools that let them find a particular piece of information quickly. Problem is, it works only when they're the ones doing it. They're hoarding that ability, not sharing it with others. For ops, it would be far more useful for each piece of information in a statedump to be separately available in a more machine-friendly way, such as an API or at least a structured (e.g. JSON) file. For other devs, gdb macros or pretty printers would be the way to go. Even kernel developers, who are often forced to debug via printf, provide /proc and /sys and a bunch of more specialized tools for extracting specific information of interest.Not only do such methods make the information itself more readily available, but they also make it much easier to document what each piece really means. That way both operators and other developers will be better able to diagnose and fix problems themselves. I suppose that's bad for people who rely on inscrutability as a form of job security, but it's good for everyone else - and even for those same developers if they want to move on to more interesting things instead of being pinned down debugging the same problems over and over.