325. What Data Should We Log/Trace And What Should We Ignore

0:00Logging lets us know where the problems are happening. It also lets us see how our app is being used. But if you aren't careful, logging can actually make it harder to understand where the problems are occurring and how your application is being used. So let's talk about how to properly implement logging in a way that will actually benefit our application in today's episode of DevQuestions. Software development is more than just writing code. So let's talk about the rest of it. Specifically, let's talk about how to properly implement logging. And let's start with an illustration. Back when I was a very new IT director at a small college, I was working in my network administrator here to implement server logging. Up until that point, we didn't have any central logging. Errors were discovered by the users or we stumbled across them or maybe if we looked through

1:05the event viewer. And we had a full network with Active Directory and Exchange Server and SQL Server and File servers and all the rest. So we found this software that would essentially manage all these logs. It would track the different things going on in our server as it would make sure that it looked out for things like, hey, you got low disk space on this server or hey, this server is hitting a very high CPU all day long. Or we've got a bottleneck in memory here or whatever the case may be with our exceptions or whatever. So this software was great and it could do so much stuff. We went to training, we figured out how to make the whole thing work correctly, how to get the most of the software and then we implemented it. And I think of like 20 servers. And so we implemented all the different servers, we set it up and configured it so we make sure we looked at all the different errors, all different issues and made sure that we were tracking all the things we needed to in order to have a really good bird's eye view

2:08of what's going on with the entire network. And we barely ever used it. And here's why because the first day, my network administrator was getting the emails for all of the errors, all of the big problems he'd get the emails for and his inbox was flooded to the point that he created a rule that's, hey, if it's coming from the logging server, put in different folder because I can't even see my normal emails. Then what do you think happened? At first we started going through those emails, but then some of them were like, hey, that server is getting low on this space, but we knew that and that's not some sort of address. Or, hey, that server is kind of going at 80% plus of CPU usage, but that's a good thing. We want it to be there. And so we started seeing all these issues where for some servers, that would have been an issue, but for others, it's not. And so we started ignoring issues. But then we had a hard time finding the issues that were really issues.

3:12So what we found out is just because an issue seems critical in a vacuum doesn't mean it actually is. And if you're not going to do something about it, that's not actually an issue. So either critical alerts do be dealt with or they need to be downgraded. Otherwise you drown out the real critical alerts. And that's really hard to do because when you're looking at things, you're saying that is an issue. But if you're not going to deal with it, is it? I mean, it may be something that you want to address. It may be on your, I would love to take care of that list. But unless it's an issue you're going to say, yes, I'm going to address that now, it's not really something you should have in your critical alert list. And so what happened was his inbox was constantly full of alerts. And we just never seem to do a good job of figuring out which ones are critical, which ones aren't. So yes, we found a problem when I use a report something, we go to the log store and

4:12go, oh, yep, that issue was there. We got the alert for 18 times, but it got lost in the middle of the thousands of other alerts that were going off. And so just because those things seemed like they should be logged, doesn't mean they should have been. We should have been logging just what we're going to fix. So let's talk about how to set up logging that works. Now we're talking more about logging inside of your application. So it's not exactly about server logs, but the things still apply. So let's talk about how to set up the logging that actually works for you. Number one, only mark issues as critical if you absolutely need to do something about them. Too many people say, well, that's a big deal. Is it though? Again, if it's a big deal, but you don't do anything about it, it's not a big deal. That's just information at that point.

5:14If you're not going to act on it, there's no reason to make it a critical issue, make an informational issue. And yes, I know sometimes you may go, but that's important. It's only important if you do something about it. That's really what it comes down to. It doesn't matter what you say, it's what you do. This is a fact for all of life. It doesn't matter what you say, it matters what you do. So if you say, this is critical, but then you do nothing, it's not critical. So be very careful. The only bubble up things that are critical, things that you're actually going to do something about. Number two, only log errors as errors. If they're unexpected or need to be addressed, such as, hey, this service is crashed. That needs to be addressed. That's an error that needs to be addressed. If there's an exception that happens, but it's not something you're going to address,

6:18then don't log it as an error. Even though the error logs are the ones that take the exceptions, and you may go, well, I have to put them there. No, you don't. So you don't need to log. The network connection went down for the user. You don't need to log that unless you need to know, oh, the network is going down in this area. For example, if you have microservices, and two different microservices, maybe one's an API, I was talking to the API, maybe that network connection is all internal. And if that goes down, yeah, you should know about that. And that would be something critical. But if the user is on their laptop and they go through a tunnel and they lose the connection to the cell tower and it comes back up afterwards, that's not an issue that you need to be logging. So even though it might throw an exception on the user's machine, even though it might be triggered as an error, you need to go, that's not really an error. And so that's where being granular about catching these exceptions and going, this one's

7:23important. This one's not, is important. If you're doing a try catch, it has catching EX, the raw exception EX, where you're not specifying which exceptions and you're just saying, hey, everyone is an error. That's not very efficient as to unclog up your errors in your logs. So maybe at first you would do that, but then figure out, oh, working these network drop offs and this network drop offs aren't really a big deal. Well then in your catch, have another catch before your generic exception EX, that's like network exception or whatever it is that the actual network error catch that one and maybe let the user know, hey, you don't have a network catch right now. But don't, don't log that. There's no reason to log that if you're not going to do something with it. Number three, only log warnings that are precursors to actual errors or critical issues.

8:23If there's something that is an indicator that something is going to happen in the future and it will be a major issue in the future, that's a great warning. So a simple example might be, hey, I know us we're getting low on disk space, okay? That's something that should be a warning because that's leading up to, hey, we're out of disk space. So yes, that could be a good warning because again, you're going to do something about it. Maybe you won't do something about it right away, but you'll do something about it in a week or within the next few days as you get time because that warning let you know if it became a critical issue. So warnings are good to have, but again, don't log things as warnings that aren't precursors to actual errors or critical issues. If it's just a, hey, the user doesn't like us or, hey, these are put bad data in. These are not warnings. This is just information that you've elevated wrongly to a warning level.

9:28So again, it has to be based upon something that you're going to do something about. If you've noticed a theme so far, this is what logs are for. Logs are not for the pack rats, the digital pack rats who want to store all this information. That's not what logs are for. Logs or are for doing something with. And if you're not doing something with your logs, you're not using them correctly. So number four, understand your applications importance and just what you capture and where. Point me out, me, me, me, me. Well, not all applications are create equal. Some applications are mission critical applications. I worked on a software service for a company that their whole lot of business was a software as a service. If that software as a service went down, then we didn't get paid. All of our customers wouldn't pay us because we're not actually providing them what they need. The whole business would shut down.

10:28That's mission critical. And that application was a log differently. Now, I've used our built other applications where it supported the development staff, where it made deployments easier or it made, you know, clean up easier. It made, you know, simple task documentation easier. These are not important applications. These are support applications that we can get by without or that are nice to have, but no one is coming in the weekend to fix. So you need to adjust your applications logs accordingly. So just because this application as a critical failure doesn't mean it's crazy important to do. Again, you're not coming out of Sunday to fix this application if it's not an important application. So you need to understand your applications importance to the organization and adjust your logging accordingly.

11:30Also where you log because there's going to be the central place that everybody should look at. Again, read your logs. Everyone should look at that tells you what's going on, what's critical, what's an error, what's a warning. Those are things you should be looking at, probably on a daily basis. You should be aware of what's going on and be fixing those things because they're priorities. If you have minor applications, they probably shouldn't go in the same system. Maybe it's be logged to a text file or maybe it's be logged to a secondary server that you check once in a while when you have time. When you're looking to make a change to a secondary system, you also look and see, hey, are there any critical error warnings in there that I should address as well? So where you log is also important because also logging is expensive and you don't need to be logging lots of messages from non-critical systems into your main logging system that's

12:32going to cost more money. And again, it's going to clog up that main system with a whole bunch of critical error and warnings that aren't actually critical errors or warnings that need to be dealt with right away. So make sure not to clog up your systems with too many false positives. I very hard to have zero false positives. Number six, do not capture information. You're not going to act on every day until unless you're investigating an issue. So when you're capturing information, don't gather information. Not going to act on. So if you're saying, well, I need to know what the user does when they're using the application. Do you? Or is that more like a testing thing? Or is that more like a sometimes thing? And that's where you have things like information logs where you can turn those on just for investigating issue or just for trying to figure out how users are using your application,

13:35not all the time. So don't log information. Don't capture things that you aren't going to act on. That also means what types of data you capture. So this might be, you know what? We need to know that we're a disk space, but we don't need to know what you were doing when you found out you're a disk space. We don't need to know that information. We just need to know that the system went, hey, I try to save a file and we're a disk space. That's all you need to know. You don't need to know who was trying to do something. That's irrelevant information. Capture the information you're actually going to act on. And again, you can use those lower log levels for when you're investigating issues or investigating how the system is used, et cetera. You can turn those on, maybe log them somewhere else or maybe just make sure they're filtered out of the main system and only used for those investigations. Number seven, do not save data beyond when you will reasonably use it.

14:40I have found that in most cases, three days is about all you need. Because if you get beyond that, you're not actually using the data. First of all, if you have a critical issue that stays in the critical issues list for over three days, you're not using the critical data system correctly. You need to have addressed it before now. That needs to turn into a ticket. Then you need to start working on right away. So if you have data for more than 30 days, why? Why is it there? Why is it not converted into an issue? The most part, delete it after 30 days. That's all you need at most. So you can have a lock system that turns over quite frequently. That even handles things like, hey, we had Christmas break and we're off for a week. No problem because you have three days. You still have 21 days of usable data. So there's still a lot of data there for you. Even if you have breaks, even if you have people going to vacation, even if you have both combined at the same time, like, 30 days is a lot of data if you were acting on it.

15:47The times when I've seen people say I need more than 30 days is when they are not acting on it and they are only looking at it every once in a while. They say, why do you know historically how things are working? You should have been fixing these problems so historically shouldn't matter because you should have fixed these problems by now. So therefore if you have a critical error or a critical issue that's happened over 30 days long, then you're doing something wrong. Or it's not taking you trying to fix it, but at that point it's already on tickets. So why do you need it in a logging system? So trying to save data beyond when it's reasonably useful. Otherwise what happens is you can make a digital pack wrap and you got so much information there that you just never use it, but it keeps getting more and more expensive to store. And it can actually cause some other problems where you are storing things beyond when you should. So don't save data beyond when it's reasonably useful.

16:47Number eight, do not log sensitive data unless it's absolutely necessary. This is one that often trips people up and this is where a lot of security breaches can come in because you're logging the sensitive data and taking it out of a secure system and putting it into a less secure system. So first of all, you're logging system should be pretty secure. And then you should also try not to include anything that's of person-identifiable information or other sensitive data in your logs. Yes, there's exceptions. Yes, there are times that you might have to. But in that case, you need to make sure that your logging system is locked down secure and it's also in compliance with all the regulations around forgetting the user if they ask you to. That means you need to go through your log system and clean out those logs referring to that user or change that data. So being compliance is harder if you're logging sensitive information.

17:48So maybe try not to. And again, these regulations are there for a reason because people are putting sensitive data in systems that aren't as secure. They think, well, it's just a logging system. Or it's not as protected because everyone has to have access to it, right? And then next thing you know, you're leaking sensitive data and inundation your users because you were sloppy. Don't do that. So number nine, number nine is don't log to the browser's console in production. This is one where you know, if you're using something like a reactor, angular review or even like blaze web assembly, where you are, are front and fully. Try really hard not to put console logs to your. Use an actual logging service. Now sometimes that logging service is not available where maybe you have a network issue or another thing where you need to log or tell a user in some way, try and tell the user directly.

18:50Don't try very hard not to put it into the console log. We really don't want to have to ask users to open up the developer console. Look at the actual console message and fear the message says that's not good. So if we're not asking them to do that, then why are you putting those messages in there? Because the other thing is those messages can leak information about what's going on. Now everything you do in the browser is going to be unsecured or at least unsecured as for that person. So that person logged in and they have their credentials. Well, sure, they have access to all their things. But that also means that any code that runs in the front end is going to be completely exposed. The user, they can read all the code, they can read all the data. So it's not quite the same as logging like a C-sharp applications information and displaying that. If you have a Blazer web app where you're showing the server side logs, you have to more information leakage than a console log would be.

19:52But still, don't put it in the console logs. Not if you can help it. There are exceptions. There are exceptions, no pun to it. But there are exceptions to that rule. But for the most part, try not to put errors in your console. Try and expose it to the user or send it to a logging system if you're going to act on. If your logs have too much information, they become useless. If your application treats too many things as important, nothing is important. If you aren't addressing the actual issues as they come up, you're going to hide the real issues. You need to make sure you're only looking at logs you'll actually act upon. That's how you protect your applications. It's how you improve your user experience and it's how you ensure you'll actually get value out of your logs. Thanks for listening. As always, I am Tim Corey.