There are legitimate reasons to log requests. But far too often, developers that are used to 'printf debugging' will abuse the system. In many cases, what they were actually looking for was either some form of APM, distributed tracing, or much more commonly, would be better done by collecting metrics - which tend to be much more lightweight.
At work I've been fighting a battle to turn multiple dozens of terabytes of logs a day into something manageable, for years now. Sometimes I wonder if I could use the logging system itself in lieu of the application, as practically all data is dumped there! It's difficult battle to convince multiple groups to just use a counter instead of spitting out the same line of text every time the same kind of event happens.
Anyway, back to the point: no matter how busy the system is, you can surely engineer a mechanism to collect error logs, even if you have to sample them. But while things are running smoothly, track metrics. And, if you have a reason to suspect things are wrong, selectively turn on some logging.
I wish more people were aware of the costs of logging. Even more so nowadays, with the microservices craze.
At my previous company, one of the devs added some debug output, but they added it to a hot loop and forgot to if it out in production.
As a result, all of our frontend servers slowed to a crawl because the logging calls were thrashing our disk I/O - and keep in mind, this is an application which never hits the disk and lives entirely in memory, mostly just assembling data structures from other APIs and combining them into a response for the client. To see a server like that brought to its knees by disk I/O from three lines of debug logging was... impressive, to say the least.
There are some tools out there that run on-box to transform log events into metrics. Not the ideal, but scales much better than dealing with a firehose of useless logs in a central log system.
Good.
There are legitimate reasons to log requests. But far too often, developers that are used to 'printf debugging' will abuse the system. In many cases, what they were actually looking for was either some form of APM, distributed tracing, or much more commonly, would be better done by collecting metrics - which tend to be much more lightweight.
At work I've been fighting a battle to turn multiple dozens of terabytes of logs a day into something manageable, for years now. Sometimes I wonder if I could use the logging system itself in lieu of the application, as practically all data is dumped there! It's difficult battle to convince multiple groups to just use a counter instead of spitting out the same line of text every time the same kind of event happens.
Anyway, back to the point: no matter how busy the system is, you can surely engineer a mechanism to collect error logs, even if you have to sample them. But while things are running smoothly, track metrics. And, if you have a reason to suspect things are wrong, selectively turn on some logging.
I wish more people were aware of the costs of logging. Even more so nowadays, with the microservices craze.