Python Logging Maxbytes

Python:

Python is an interactive and more accessible language than any other programming language. The python programming language uses a variety of libraries to perform the operations in a faster way. The python language can also be used in web development; Django and Flask are the frameworks used to create web applications using Python. In Python, indentation is the main concept; if we do not follow proper indentation, then the program will not run properly, and we will get an error in the output.

Python programming language contains methods or functions to reduce the size of the code, and the python programming language provides built-in functions and user-defined functions. We can import the functions in the python programming language through the libraries, which can be downloaded using the python package manager ( pip ). While working on the project and we want to develop the project using the python programming language. Python programming language is an object-oriented and high-level language it is easier to learn when compared to other programming languages.

The python programming language contains mainly six built-in datatypes; these six data types help solve the problem efficiently and faster. The python programming language consists of a built-in function and provides libraries and modules that can be imported to solve the problem more efficiently. Generally, there are many versions of python interpreters available. Still, from them, we need to download the version of Python more significantly than or equal to 3.4 so that the code runs faster and we can observe the output in the console.

Logging in an Application

Any application should have logging since it gives programmers and those who support the application crucial insight into how their systems are operating. Without adequate logging, we are unable to diagnose the root causes of our apps' failures and take effective corrective action.

Imagine you were working on a vital application that was necessary for your business to make money. Imagine for a moment that your application has malfunctioned at, say, 3 in the morning on a Saturday night, causing the business to stop making money. Your best ally in this situation for figuring out what went wrong is logging. Checking the error logs for your program is the first thing you'll do after logging in.to identify precisely what and where the application failed. In this specific case, it is obvious right away that there is a memory problem and that you need to increase the RAM on the computer hosting your system. Your business is back online as soon as you launch a new AWS EC2 instance with a little bit more RAM and deploy your app.

Overall, because of the logging, this process only took a little under 30 minutes and cost your business very little money. Without proper recordkeeping, you might have found yourself puzzling for hours into the wee hours of the morning and become increasingly stressed while your business lost money.

This illustration demonstrates why we should pay attention to how your systems record events, and in this lesson, we'll go over several best practices you may use to construct your own logging systems.

Module for Logging

HereL Official Docs is where you can find the logging module's official documentation.

One of the best tools you can use to include a logging system into your Python applications is the Logging module. When it comes to logging, it is the go-to option for enterprise programmers because of its well developed features.

Creating Formatted Log Statements

You have the authority to specify the precise format of your logging messages using the Logging module. This is effective because it enables you to including process/thread names in every log statement without explicitly stating them in the message you want to log, for example.

We can specify the format within the logging module by doing something like this:

#Import the required libraries
importlogging
importlogging.handlersashandlers
importtime
log=logging.getLogger('app')
log.setLevel(logging.INFO)
formatter=logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logHandler=handlers.TimedRotatingFileHandler('timed_app.log',when='M',interval=1,backupCount=2)
logHandler.setLevel(logging.INFO)
logHandler.setFormatter(formatter)
logger.addHandler(logHandler)
# Creating the  function
defmain():
whileTrue:
time.sleep(1)
logger.info(" Log Statement")
main()

Output:

2022-06-1121:45:49,106-app-INFO-LogStatement
2022-06-1121:45:50,108-app-INFO-LogStatement

Making an Easy File Logger:

In this example, we'll make a very basic log file called my app.log that will store all INFO level log messages that are sent to it in the same directory as our application.

Code:

importlogging
log=logging.getLogger('app')
log.setLevel(logging.INFO)
uh=logging.FileHandler('app.log')
uh.setLevel(logging.INFO)
logger.addHandler(uh)
defmain():
logger.info( “First Log Statement")
main( )

Rolling Log Files:

It could be a good idea to divide the log files produced by long-running services into appropriate time-based files if you are creating services that are up constantly.

For example, you could create a fresh log file for each day of the week. In the event that something goes wrong, you won't have to load and analyse a huge log file thanks to this. Additionally, because your logs are stored in numerous files rather than just one, it reduces the risk that you will lose them all.

Both the RotatingFileHandler and the TimedRotatingFileHandler classes are available in the logging module.

RotatingFileHandler:

When the current log file reaches a specific size, we can rotate our log statements into a new file using a RotatingFileHandler. In this example, we'll configure it to rotate into a new file if it reaches 500 bytes, up to a maximum of two backups.

Example:

importlogging
importlogging.handlersashandlers
importtime
log=logging.getLogger('app')
log.setLevel(log.INFO)
logHandler=handlers.RotatingFileHandler('app.log',maxBytes=500,backupCount=2)
logHandler.setLevel(log.INFO)
log.addHandler(logHandler)
defmain():
whileTrue:
time.sleep(1)
log.info(" Sample Log Statement")
main()

When this is run, you should observe that if app.log grows by more than 500 bytes, it is closed and renamed app.log.x, where x increases until it hits the value that we have set backupCount to.

TimedRotatingFileHandler

We can take a time slice to capture log files using the TimedRotatingFileHandler.

importlogging
importlogging.handlersashandlers
importtime


log=logging.getLogger('my_app')
log.setLevel(logging.INFO)


logHandler=handlers.TimedRotatingFileHandler('app.log',when='M',interval=1)
logHandler.setLevel(log.INFO)
log.addHandler(logHandler)
defmain():
whileTrue:
time.sleep(1)
log.info(" Sample Log Statement")
main()

External Services:

Implementing your own logging systems might not be adequate for some more important enterprise-level projects, and you might need to communicate with outside logging services like LogStash. These services take care of gathering, analysing, and converting your application's recorded output for you.

You might be wondering what the point is, but as we go toward more micro-service-based solutions, it is just impractical to store log files everywhere and look through them when anything goes wrong. Consider deploying 10 separate servers with 10 instances of a service. The only effective remedy is to implement the Instances of your service output your logs to a system like Logstash so you may browse an aggregated view of your logs much more quickly.

Logstash and Python:

If you choose to follow this course of action, I advise you to look at projects like vklochan/python-logstash, which makes it very simple to connect Logstash with your Python applications.

Utilization of Proper Log Levels:

Not every event that is logged in your application will be of equal value when it comes to weight. For example, you could want to record fatal errors so that they can be retrieved by a different log handler that stores these logs in a different file. This lessens the requirement to remove all the fat from a typical log assertions and, in the unlikely event that something crucial has gone wrong, get right to the point where your application failed.

The RotatingFileHandler and the TimedRotatingFileHandler are two different categories of file handlers that were previously covered in this article. We can log all error messages to a rotating file using these two different sorts of handlers, but we can also log all other log files to a TimedRotatingFileHandler because we anticipate seeing a lot more of those than error messages.

The next example will show you how to divide up two different log levels—INFO and ERROR—to two separate locations. CRITICAL, ERROR, WARNING, INFO, DEBUG, and NOTSET are the levels by default.

importlogging
importlogging.handlersashandlers
importtime
log=logging.getLogger('app')
log.setLevel(logging.INFO)
formatter=logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
logHandler=handlers.TimedRotatingFileHandler('normal.log',when='M',interval=1,backupCount=0)
logHandler.setLevel(logging.INFO)
logHandler.setFormatter(formatter)


errorLogHandler=handlers.RotatingFileHandler('error.log',maxBytes=5000,backupCount=0)
errorLogHandler.setLevel(log.ERROR)
errorLogHandler.setFormatter(formatter)


log.addHandler(logHandler)
log.addHandler(errorLogHandler)


defmain():
whileTrue:
time.sleep(1)
logger.info(" Sample Log Statement")
logger.error("error log statement")


main()