If someone buys a hammer and hits themselves on the head with it repeatedly, should there be some padding on the hammer?
This sounds ridiculous, but I asked it very seriously in a recent meeting.
The issue at hand was that the user of a client library had ramped up some new traffic pattern which went above the system's configured limits, and the library logged an error on every request. The logging was so voluminous that it caused instability in the application. And although the new traffic was failing 100% of its requests, the change was ramped to a large number of application instances before the problem was noticed.
In the post-mortem, it was suggested that the client library should reduce its logging volume, as it caused instability. But if the application's stability (and the health of the new traffic pattern being ramped) mattered at all, then why was it not verified that a single instance of the application behaved correctly before increasing the blast radius of the change to a large number of instances?
Another, less open-ended, version of the opening question is:
If someone buys a hammer and hits themselves on the head with it once, should they stop hitting themselves on the head, or should there be some padding on the hammer?