Python Tutorial

Introduction Python Features Python Applications Python System requirements Python Installation Python Examples Python Basics Python Indentation Python Variables Python Data Types Python IDE Python Keywords Python Operators Python Comments Python Pass Statement

Python Conditional Statements

Python if Statement Python elif Statement Python If-else statement Python Switch Case

Python Loops

Python for loop Python while loop Python Break Statement Python Continue Statement Python Goto Statement

Python Arrays

Python Array Python Matrix

Python Strings

Python Strings Python Regex

Python Built-in Data Structure

Python Lists Python Tuples Python Lists vs Tuples Python Dictionary Python Sets

Python Functions

Python Function Python min() function Python max() function Python User-define Functions Python Built-in Functions Python Recursion Anonymous/Lambda Function in Python apply() function in python Python lambda() Function

Python File Handling

Python File Handling Python Read CSV Python Write CSV Python Read Excel Python Write Excel Python Read Text File Python Write Text File Read JSON File in Python

Python Exception Handling

Python Exception Handling Python Errors and exceptions Python Assert

Python OOPs Concept

OOPs Concepts in Python Classes & Objects in Python Inheritance in Python Polymorphism in Python Python Encapsulation Python Constructor Python Super function Python Static Method Static Variables in Python Abstraction in Python

Python Iterators

Iterators in Python Yield Statement In Python Python Yield vs Return

Python Generators

Python Generator

Python Decorators

Python Decorator

Python Functions and Methods

Python Built-in Functions Python String Methods Python List Methods Python Dictionary Methods Python Tuple Methods Python Set Methods

Python Modules

Python Modules Python Datetime Module Python Math Module Python Import Module Python Time ModulePython Random Module Python Calendar Module CSV Module in Python Python Subprocess Module

Python MySQL

Python MySQL Python MySQL Client Update Operation Delete Operation Database Connection Creating new Database using Python MySQL Creating Tables Performing Transactions

Python MongoDB

Python MongoDB

Python SQLite

Python SQLite

Python Data Structure Implementation

Python Stack Python Queue Python Linked List Python Hash Table Python Graph

Python Advance Topics

Speech Recognition in Python Face Recognition in Python Python Linear regression Python Rest API Python Command Line Arguments Python JSON Python Subprocess Python Virtual Environment Type Casting in Python Python Collections Python Attributes Python Commands Python Data Visualization Python Debugger Python DefaultDict Python Enumerate

Python 2

What is Python 2

Python 3

Anaconda in Python 3 Anaconda python 3 installation for windows 10 List Comprehension in Python3

How to

How to Parse JSON in Python How to Pass a list as an Argument in Python How to Install Numpy in PyCharm How to set up a proxy using selenium in python How to create a login page in python How to make API calls in Python How to run Python code from the command prompt How to read data from com port in python How to Read html page in python How to Substring a String in Python How to Iterate through a Dictionary in Python How to convert integer to float in Python How to reverse a string in Python How to take input in Python How to install Python in Windows How to install Python in Ubuntu How to install PIP in Python How to call a function in Python How to download Python How to comment multiple lines in Python How to create a file in Python How to create a list in Python How to declare array in Python How to clear screen in Python How to convert string to list in Python How to take multiple inputs in Python How to write a program in Python How to compare two strings in Python How to create a dictionary in Python How to create an array in Python How to update Python How to compare two lists in Python How to concatenate two strings in Python How to print pattern in Python How to check data type in python How to slice a list in python How to implement classifiers in Python How To Print Colored Text in Python How to open a file in python How to Open a file in python with Path How to run a Python file in CMD How to change the names of Columns in Python How to Concat two Dataframes in Python How to Iterate a List in Python How to learn python Online How to Make an App with Python How to develop a game in python How to print in same line in python How to create a class in python How to find square root in python How to import numy in python How to import pandas in python How to uninstall python How to upgrade PIP in python How to append a string in python How to comment out a block of code in Python How to change a value of a tuple in Python How to append an Array in Python How to Configure Python Interpreter in Eclipse How to plot a Histogram in Python How to Import Files in Python How to Download all Modules in Python How to get Time in seconds in Python How to Practice Python Programming How to plot multiple linear regression in Python How to set font for Text in Python How to Convert WhatsApp Chat Data into a Word Cloud using Python How to Install Tweepy in Python How to Write a Configuration file in Python How to Install Scikit-Learn How to add 2 lists in Python How to assign values to variables in Python and other languages How to build an Auto Clicker using Python How to check if the dictionary is empty in Python How to check the version of the Python Interpreter How to convert Float to Int in Python How to Convert Int to String in Python How to Define a Function in Python How to Install Pandas in Python How to Plot Graphs Using Python How to Program in Python on Raspberry pi How to Reverse a number in Python How to Sort a String in Python How to build a Virtual Assistant Using Python How to Fix an EOF Error in Python How to make a firewall in Python How to clear screen in Python How to Create User Defined Exceptions in Python How to determine if a binary tree is height-balanced How to Import Kaggle Datasets Directly into Google Colab How to Install Python in Kali Linux

Sorting

Python Sort List Sort Dictionary in Python Python sort() function Python Bubble Sort

Programs

Factorial Program in Python Prime Number Program in Python Fibonacci Series Program in Python Leap Year Program in Python Palindrome Program in Python Check Palindrome In Python Calculator Program in Python Armstrong Number Program in Python Python Program to add two numbers Anagram Program in Python Number Pattern Programs in Python Even Odd Program in Python GCD Program in Python Python Exit Program Python Program to check Leap Year Operator Overloading in Python Pointers in Python Python Not Equal Operator Raise Exception in Python Salary of Python Developers in India

Questions

What is a Script in Python What is the re.sub() function in Python After Python What Should I Learn What Does the Percent Sign (%) Mean in Python What is online python free IDE What is Python online compiler What are the Purposes of Python What is Python compiler GDB What is Ipython shell What does base case mean in recursion What does the if __name__ == "__main__" do in Python What is Sleeping Time in Python What is Collaborative Filtering in ML, Python What is the Python Global Interpreter Lock What is None Literal in Python What is the Output of the bool in Python Is Python Case-sensitive when Dealing with Identifiers Is Python Case Sensitive Is Python Object Oriented Programming language

Differences

Difference between Perl and Python Difference between python list and tuple Difference between Input() and raw_input() functions in Python Difference between Python 2 and Python 3 Difference Between Yield And Return In Python Important Difference between Python 2.x and Python 3.x with Example Difference between Package and Module in Python Difference between Expression and Statement in Python Difference between For Loop and While Loop in Python Difference between Module and Package in Python Difference between Sort and Sorted in Python Difference between Overloading and Overriding in Python Python vs HTML Python vs R Python vs Java Python vs PHP While vs For Loop in Python Python Iterator vs Iterable Set Vs List Python

Python Kivy

Kivy Architecture Kivy Buttons Kivy Layouts Kivy Widgets What is Kivy in Python

Python Tkinter

Application to get live USD/INR rate Using Tkinter in Python Application to Search Installed Application using Tkinter in Python Compound Interest GUI Calculator using Tkinter in Python Create a Table Using Tkinter in Python Create First GUI Application using Tkinter in Python File Explorer using Tkinter in Python GUI Calendar using Tkinter in Python GUI to extract lyrics from a song Using Tkinter in Python GUI to Shut down, Restart and Logout from the PC using Tkinter in Python Loan calculator using Tkinter in Python Make Notepad using Tkinter in Python Rank Based Percentile GUI Calculator using Tkinter in Python Screen Rotation app Using Tkinter in Python Spell Corrector GUI using Tkinter in Python Standard GUI Unit Converter using Tkinter in Python Text detection using Tkinter in Python To Do GUI Application using Tkinter in Python Weight Conversion GUI using Tkinter in Python Age calculator using Tkinter Create a Digital Clock Using Tkinter Create a GUI Marksheet using Tkinter Simple GUI Calculator Using Tkinter Simple Registration form using Tkinter Weight Conversion GUI Using Tkinter Color Game Using Tkinter in Python File Explorer in Python using Tkinter Making of Notepad Using Tkinter Python Simple FLAMES Game Using Tkinter Python ToDo GUI Application Using Tkinter Python

Python PyQt5

Compound Interest GUI Calculator using PyQt5 in Python Create Table Using PyQt5 in Python Create the First GUI Application using PyQt5 in Python GUI Calendar using PyQt5 in Python Loan Calculator using PyQt5 in Python Rank Based Percentile GUI Calculator using PyQt5 in Python Simple GUI calculator using PyQt5 in Python Standard GUI Unit Converter using PyQt5 in Python

Misc

Introduction to Scratch programming SKLearn Clustering SKLearn Linear Module Standard Scaler in SKLearn Python Time Library SKLearn Model Selection Standard Scaler in SKLearn Accuracy_score Function in Sklearn Append key Value to Dictionary in Python Cross Entropy in Python Cursor in Python Data Class in Python Imread Python Parameter Passing in Python Program of Cumulative Sum in Python Python Program for Linear Search Python Program to Generate a Random String Read numpy array in Python Scrimba python Sklearn linear Model in Python Scraping data in python Accessing Key-value in Dictionary in Python Find Median of List in Python Linear Regression using Sklearn with Example Problem-solving with algorithm and data structures using Python Python 2.7 data structures Python Variable Scope with Local & Non-local Examples Arguments and parameters in Python Assertion error in python Programs for Printing Pyramid Patterns in Python _name_ in Python Amazon rekognition using python Anaconda python 3.7 download for windows 10 64-bit Android apps for coding in python Augmented reality in python Best app for python Not supported between instances of str and int in python Python comment symbol Python Complex Class Python IDE names Selection Sort Using Python Hypothesis Testing in Python Idle python download for Windows Insertion Sort using Python Merge Sort using Python Python - Binomial Distribution Python Logistic Regression with Sklearn & Scikit Python Random shuffle() method Python variance() function Removing the First Character from the String in Python Adding item to a python dictionary Best books for NLP with Python Best Database for Python Count Number of Keys in Dictionary Python Cross Validation in Sklearn Drop() Function in Python EDA in Python Excel Automation with Python Python Program to Find the gcd of Two Numbers Python Web Development projects Adding a key-value pair to dictionary in Python Python Euclidean Distance Python Filter List Python Fit Transform Python e-book free download Python email utils Python range() Function Python random.seed() function Python PPTX Python Pickle Python Seaborn Python Coroutine Python EOL Python Infinity Python math.cos and math.acos function Python Project Ideas Based On Django Reverse a String in Python Reverse a Number in Python Python Word Tokenizer Python Trigonometric Functions Python try catch exception GUI Calculator in Python Implementing geometric shapes into the game in python Installing Packages in Python Python Try Except Python Sending Email Socket Programming in Python Python CGI Programming Python Data Structures Python abstract class Python Compiler Python K-Means Clustering NSE Tools In Python Operator Module In Python Palindrome In Python Permutations in Python Pillow Python introduction and setup Python Functionalities of Pillow Module Python Argmin Python whois Python JSON Schema Python lock Return Statement In Python Reverse a sentence In Python tell() function in Python Why learn Python? Write Dictionary to CSV in Python Write a String in Python Binary Search Visualization using Pygame in Python Latest Project Ideas using Python 2022 Closest Pair of Points in Python ComboBox in Python Best resources to learn Numpy and Pandas in python Check Letter in a String Python Python Console Python Control Statements Convert Float to Int in Python using Pandas Importing Numpy in Pycharm Python Key Error Python NewLine Python tokens and character set Python Strong Number any() Keyword in python Best Database in Python Check whether dir is empty or not in python Comments in the Python Programming Language Convert int to Float in Python using Pandas Decision Tree Classification in Python End Parameter in python __GETITEM__ and __SETITEM__ in Python Python Namespace Python GUI Programming List Assignment Index out of Range in Python List Iteration in Python List Index out of Range Python for Loop List Subtract in Python Python Empty Tuple Python Escape Characters Sentence to python vector Slicing of a String in Python Executing Shell Commands in Python Genetic Algorithm in python Get index of element in array in python Looping through Data Frame in Python Syntax of Map function in Python Python AIOHTTP Alexa Python Artificial intelligence mini projects ideas in python Artificial intelligence mini projects with source code in Python Find whether the given stringnumber is palindrome or not First Unique Character in a String Python Python Network Programming Python Interface Python Multithreading Python Interpreter Data Distribution in python Flutter with tensor flow in python Front end in python Iterate a Dictionary in Python Iterate a Dictionary in Python – Part 2 Allocate a minimum number of pages in python Assertion Errors and Attribute Errors in Python Checking whether a String Contains a Set of Characters in python Python Control Flow Statements *Args and **Kwargs in Python Bar Plot in Python Conditional Expressions in Python Function annotations() in Python Image to Text in python import() Function in Python Import py file in Python Multiple Linear Regression using Python Nested Tuple in Python Python String Negative Indexing Reading a File Line by Line in Python Python Comment Block Base Case in Recursive function python ER diagram of the Bank Management System in python Image to NumPy Arrays in Python NOT IN operator in Python One Liner If-Else Statements in Python Sklearn in Python Cube Root in Python Python Variables, Constants and Literals Creating Web Application in python Notepad++ For Python PyPi TensorFlow Python | Read csv using pandas.read_csv() Run exec python from PHP Python coding platform Python Classification Python | a += b is not always a = a + b PyDev with Python IDE Character Set in Python Best Python AI Projects _dict_ in Python Python Ternary Operators Self in Python Python Modulo Python Packages Python Syntax Python Uses Python Bitwise Operators Python Identifiers Python Matrix Multiplication Python AND Operator Python Logical Operators Python Multiprocessing Python Unit Testing __init__ in Python Advantages of Python Python Boolean Python Call Function Python History Python Image Processing Python main() function Python Permutations and Combinations Conditional Statements in python Confusion Matrix Visualization Python Nested List in Python Python Algorithms Python Modules List Method Overloading in Python Python Arithmetic Operators Assignment Operators in Python Python Division Python exit commands Continue And Pass Statements In Python Colors In Python Convert String Into Int In Python Convert String To Binary In Python Convert Uppercase To Lowercase In Python Convert XML To JSON In Python Converting Set To List In Python Covariance In Python CSV Module In Python Decision Tree In Python Dynamic Typing In Python BOTTLE Python Web Framework Introducing modern python computing in simple packages Reason for Python So Popular Returning Multiple Values in Python Spotify API in Python Spyder (32-bit) - Free download Time. Sleep() in Python Traverse Dictionary in Python YOLO Python Nested for Loop in Python Data Structures and Algorithms Using Python | Part 1 Data Structures and Algorithms using Python | Part 2 ModuleNotFoundError No module named 'mysql' in Python N2 in Python XGBoost for Regression in Python Explain sklearn clustering in Python Data Drop in Python Falcon Python Flutter Python Google Python Class Excel to CSV in Python Google Chrome API in Python Gaussian elimination in python Matrix List Comprehension in Python Python List Size Python data science course StandardScaler in Sklearn Python Redis Example Python Program for Tower of Hanoi Python Printf Style Formating Python Percentage Sign Python Parse Text File Python Parallel Processing Python Online Compiler Python maketrans() function Python Loop through a Dictionary Python for Data Analysis Python for Loop Increment Python Kwargs Example Python Line Break Kite Python Length of Tuple in Python Python String Lowercase Python Struct Python Support Python String Variable Python System Command Python TCP Server Python Unit Test Cheat String Python Validator Unicode to String in Python An Introduction to Mocking in Python An Introduction to Subprocess in Python with Examples Anytree Python API Requests using Python App Config Python Check if the directory exists in Python Managing Multiple Python Versions With pyenv os.rename() method in Python os.stat() method in Python Python Ways to find nth occurrence of substring in a string Python Breakpoint Find Last Occurrence of Substring using Python Python Operators Python Selectors Python Slice from Last Occurrence of K Sentiment Analysis using NLTK String indices must be integers in Python Tensorflow Angular in Python AES CTR Python Crash Course on Python by Google Curdir Python Exrex Python FOO in Python Get Bounding Box Co-ordinates Python Hog Descriptor Opencv Python Io stringio Python iobase Python IPython Display Iterate through the list in Python Joint Plot in Python JWT Decode Python List Comprehension in Python List in Python Map Syntax in Python Python Marshmallow PyShark in Python Python Banner Python Logging Maxbytes Python Multiprocessing Processor Python Skyline Python Subprocess Call Example Python Sys Stdout Python Win32 Process Python's Qstandarditemmodel Struct Module in Python Sys Module in Python Tuple in Python Uint8 Python XXhash Python Examples XXhash Python Handling missing keys in Python dictionaries Python Num2words Python Os sep OSError in Python Periodogram in Python Pltpcolor in Python Poolmanager in Python Python pycountry Python pynmea2 Add a key-value pair to dictionary in Python Add Dictionary to Dictionary in Python Add Element to Tuple in Python Add in Dictionary Python Arithmetic Expressions in Python Array to String in Python AX Contour in Python Best Way to Learn Python for Free Captcha Code in Python with Example CatPlot in Python Change Data Type in Python Check if a String is Empty in Python Algorithm for Factorial of a number in Python chr() and ord() Functions in Python Class and Static Methods in Python Convert List to Array in Python Copying a file from one folder to Another with Python Cx_Oracle Python with Example Enumerate() Function in Python Event Key in Python Exclusive OR in Python Exponentiation in Python Expressions in Python Filter List in Python Find key from value in dictionary python Find Words in String Python First unique character in a string Python Fsolve in Python GET and POST requests using Python Gethostbyname() function in Python Comment starts with the symbol in Python Isodate Python Isreal() Python Mrcnn Python OS Module in Python Paramiko Python Example Python BytesIO Python Deep Copy and Shallow Copy Python Glob Python Memory Management Python Operator Precedence Python Parser Python Project Ideas Python sklearn train_test_split Python SymPy Python Syntax Error Invalid Syntax Python Tricks: The Book Sort a dataframe based on a column in Python Spark and Python for big data with pyspark github Standard Scalar in Python STL in Python Sublime Python Sum of Prime Numbers in Python XML parsing in Python The ZEN of Python THONNY IDE Tic Tac Toe Python Tweepy Python Types of Functions in Python Virtualenv Python3 Python Visual Studio Z Pattern in Python Aggregation data across one or more columns Boolean Literals in Python Find() Function in Python Heap Sort in Python .iloc function in Python Integers and floating-point numbers in Python Label and Integer-based Slicing Technique in Python ML Cancer Cell Classification Using Scikit-learn Mutable and Immutable Data Types in Python PEMDAS in Python Precedence Order In Python Product of Two Numbers in Python Python Built-in Exceptions Python Decorator Wraps Python dedent Python deep copy object Python deep learning library Python def Python duplicate list Python Hashlib Python Introduction Notes Python is Easy to Learn or Not Python list files in Directory Python Long Int Python Loop Questions Python Program to find the Largest Number in a List Python Programming Practice Questions Python Raise an Exception Reduce Function in Python Reverse a Tuple in Python Reverse of Array in Python Sieve of Eratosthenes in Python Special Literals in Python String to Int in Python 3 Temperature Conversion Program in Python Two Conditions in If Statement Python Types of Knapsack Problems in Python XOR Function in Python Best First Search Program in Python Binary to Decimal Conversion in Python Environment Variables in Python Escape Sequence Characters in Python Export WhatsApp Chat History to Excel using Python Fizzbuzz Program in Python Float in Python fromtimestamp() function of Datetime.date class in Python Get Random Dog Images in Python Handling EOFError Exception in Python Python Delete Folder and Contents Python Expanduser Python gdb Breakpoint Remove special characters from string in Python DRF Serializer in Python Logistic Regression in Python Python del multiple variables Python Delete a Directory with Files Python Delete Files from Directory Python Delete Key Types of Functions in Python Understanding Variable Length Arguments in Python XOR in Python Python Competitive Programming Questions Top 10 Python Built-In Decorators That Optimize Python Code Significantly Python Degree Symbol Agglomerative Clustering with and without Structure in Scikit-Learn Sum of two Columns in OpenPyxl and Result in a new Column How to Find Median in Python? Complex Python Programs How to Print Space in Python Python Program to Find Compound Interest Python program to find uncommon words from two Strings Python Program to Swap Two Elements in a List Cache Memory in Computer

Python Memory Management

In order to manage memory, Python uses a private heap that contains all of its data structures and objects. The Python memory manager is responsible for the internal management of the private heap. Different parts of the Python memory manager deal with different aspects of dynamic storage management, such as sharing, segmentation, preallocation, or caching.

Python Memory Management

We will gain a better understanding of some of Python's behaviors if we are familiar with its internal workings. I hope we will also develop a new interest in Python. There is a lot of logic running in the background to make sure our program performs as we anticipate.

 Considering Memory as an Empty Book:

Start by visualizing a computer's memory as a blank book meant for short stories. The pages are blank at this time. Different writers will eventually emerge. Each author wants a little room to write their story. They have to be careful which pages they have to write on because they can't write on top of one another. They speak with the book's manager before they start writing. The manager then decides where they are permitted to write in the book.

This book has been around for a very long time, so many of the stories in it are outdated. The stories are taken down to make room for new ones when no one reads them or makes any references to them.

Computer memory is analogous to that blank book, in essence. This analogy works pretty well because fixed-length contiguous blocks of memory are frequently referred to as pages.

The authors can be compared various programs or processes that require memory to store data. The manager serves as a kind of memory manager by determining where the authors can write in the book. The garbage collector is the one who takes out the old stories to make room for the new ones.

Memory Management: Hardware and Software

Applications read and write data using memory management. Where to store the data for an application is decided by a memory manager. The manager must locate some free space and provide it to the application because memory is limited, similar to the book's pages. Memory allocation is the general term used to refer to this memory provisioning process.

On the other hand, data can be deleted or released when it is no longer required. 

When we run our Python programs, a physical device inside of our computer stores data. Before the objects actually reach the hardware, the Python code must pass through several levels of abstraction.

The operating system is one of the primary layers just above hardware (like a Hard drive or RAM). Requests to read and write the memory are carried out or get rejected.

Applications sit on top of the OS, with the default Python implementation being one of them (included in our OS or downloaded from python.org.) The Python application manages the memory used by our Python code. This article focuses on the memory management algorithms and structures used by the Python application.

Python's default implementation:

In reality, CPython, the default Python implementation, is created in the C programming language. A language with written characters from another language is really awesome. An English-language reference manual contains a definition of the Python language. That manual alone, though, isn't all that helpful. We still require a means of decoding written code based on the manual's rules.

To actually run interpreted code on a computer, we also need something. Both of these requirements are satisfied by the standard Python implementation. Our Python code is transformed into instructions, which are then executed on a virtual machine.

Python is known as an interpreted programming language. Our Python code is actually reduced to bytecode, which is a more readable form of instruction for computers. When we run our code, a virtual machine decodes these instructions.

Have we ever come across an a.pyc file or a folder called "pycache". That is the bytecode that the virtual machine interprets.

It's significant to remember that CPython is not the only implementation available. For use with Microsoft's Common Language Runtime, IronPython is compiled down. In order to run just on Java Virtual Machine, Jython is translated into Java bytecode. Then there is PyPy, but we won't go into detail about that because it deserves its own entire article. We will concentrate on how Python's default implementation, CPython, handles memory management for the purposes of this article.

Okay, so CPython interprets Python bytecode and is written in C. The memory management structures and algorithms are present in the C code for CPython. We must gain a fundamental understanding of CPython in order to comprehend how Python manages its memory.

C, the language used to create CPython, does not come with native support for object-oriented programming. The CPython code contains a lot of intriguing designs as a result.

Everything within Python is an object, including types like int and str, as we may have heard. In CPython, it's accurate in terms of implementation. Every other object in CPython utilizes a struct called a PyObject.

The PyObject, the ancestor of all Python objects, only has two components:

  • ob_refcnt: reference count
  • ob_type: pointer to another type

For garbage collection, references are counted. The real object type is then indicated by a pointer. That object type simply describes a Python object using another struct (such as a dict or int).

Each object has a unique memory allocator that is only capable of allocating memory for that particular object. Additionally, each object has a memory deallocator that "frees" the memory when it is no longer required.

All this discussion of allocating and freeing memory, though, is missing one crucial element. On a computer, memory is a shared resource, and if two processes attempt to write to the same location simultaneously, bad things may result.

The Global Interpreter Lock (GIL):

The GIL offers a solution to the frequent issue of managing shared resources, such as computer memory. Two threads may step on each other's toes if they attempt to modify so same resource at the same time. Both threads may end up with the opposite of what they intended in the final product, which can be a jumbled mess.

Think about the book analogy once more. Let's say that two writers decide obstinately that it is their turn to write. Additionally, they must both write simultaneously on the same page of the book.

They both start writing on the page while ignoring each other's attempts to tell a story. Two stories are ultimately stacked on top of one another, rendering the entire page completely illegible.

A single, global lock on the interpreter whenever a thread interacts with the shared resource is one approach to solving this issue (the page in the book). To put it another way, just one author can be writing at once.

By getting locked the entire interpreter, Python's GIL achieves this, making it impossible for a different thread to trample over the active one. CPython makes use of the GIL to ensure the safe handling of memory. This strategy has advantages and disadvantages, and the GIL is hotly contested within the Python community. 

Garbage Collection:

Let's go back to the book analogy and say that some of the stories are getting very old in the book. No longer are those stories read or used as references. You could get rid of something to make space for fresh writing if no one is reading it or using it in their own works. That old, unsourced writing is comparable to a Python object with a reference count of 0. Keep in mind that each Python object has a reference count and a pointer to its type.

There are several reasons why the reference count rises. If we assign the reference count to another variable, for instance, it will rise:

num = [1, 2, 3]
# Reference count is 1
more_numbers = num
# Reference count is 2

Additionally, it will rise if we use the thing as an argument:

total = sum(num)

The reference count will rise if we include the object in a list, as in the following final example:

matrix = [num, num, num]

The sys module in Python enables us to view an object's current reference count. We can use sys.getrefcount(numbers), but be aware that doing so adds 1 to the reference count because the object is added when using getrefcount().

In any case, the object's reference count is higher than 0 if it is still needed to remain in our code. The object does have a specific deallocation function that is called once it drops to 0, "freeing" the memory for use by other objects.

What other objects use the memory that has been "freed," and what does that mean exactly? Let's get started with CPython's memory management right away to learn more about it.

Memory Management in CPython:

Let's focus on  CPython's memory architecture and algorithms. CPython's memory management uses Data structures and algorithms to manage our data which is very important.

As previously mentioned, there are several abstraction layers between the physical hardware and CPython. Applications (which include Python) can access the virtual memory layer that is created by the operating system (OS), which abstracts the physical memory. A portion of memory is set aside by an OS-specific virtual memory manager for the Python process. The Python process now owns the darker grey boxes in the image below.

Python uses non-object memory and a chunk of the memory for internal use. The remaining space is used for object storage (dict, your int, and the like). Please be aware that this was a bit simplified. Evaluate the CPython source code, where all of this memory management takes place, for a complete picture.

The task of allocating memory in the object memory space is carried out by an object allocator in CPython. The majority of the magic takes place in this object allocator. It is called whenever a new object needs to be deleted or allocated space.

For Python objects like int and list, adding and removing data typically doesn't require a lot of data at once. As a result, the allocator's design has been adjusted so that it can handle small amounts of data at once. Additionally, it tries to wait until memory allocation is absolutely necessary.

The allocator is described as a quick, special-purpose memory allocator for smaller chunks to be used on top of a general-purpose "malloc" in the source code comments. The memory allocation library function for C in this instance is called malloc.

We'll now examine CPython's memory management approach. We'll first discuss the three main components and how they correspond to one another. Arenas are the biggest memory blocks and are positioned on a page boundary. An OS-used fixed-length contiguous chunk of memory's edge is referred to as a page boundary. Python makes the 256-kilobyte page size assumption for the system.

Python Memory Management

Pools, which are of one virtual memory page (four kilobytes), are found within the arenas. These resemble the pages in our analogous book. These pools are divided up into smaller memory blocks.

Each block in a specific pool belongs to the same "size class". Given a certain amount of requested data, a size class defines a particular block size. The graph below was directly extracted from the comments in the source code:

Request in bytesBlock size allocatedSize class IDX
1-880
9-16161
17-24242
25-32323
33-40404
...so on......
497-50450462
505-51251263

For instance, if 34 bytes are needed, the data will be stored in a block of size 40 bytes.

Pools:

Pools are made up of blocks that belong to the same size class. Every pool keeps a double-linked list of all the other pools in its size class. Thus, even across various pools, the algorithm can quickly locate space that is available for a given block size.

A usedpools list keeps track of all the pools for each size class that still have some data space available. The algorithm checks this usedpools list from the list of pools for a given block size when that block size is requested.

 Pools themselves have to be in one of the three states: used, full, or empty. Blocks that are available in a used pool can be used to store data. Blocks in a full pool are allotted and have data in them. When required, any block size class can be assigned to an empty pool, which has no data stored.

All the pools that are currently empty are tracked by a freepools list. 

Assume that an 8-byte memory block is required by our code. A new, empty pool is created and initialized to store 8-byte blocks if there are no pools in the used pools of the 8-byte size class. The usedpools list is then updated to include this new pool, making it accessible for incoming requests.

A full pool, for example, might release some of its blocks because the memory is no longer required. The usedpools list for that size class would be updated to include that pool once more. We can now see how this algorithm allows pools to freely have a transition between these states (or even memory size classes).

Blocks:

Python Memory Management

Pools constitute a pointer to their "free" memory blocks, as shown in the diagram above. There is a slight variation in how this operates. The comments in the source code state that this allocator "strives at all stages (pool, arena, and block) never  make contact a piece of memory until it's actually needed."

Thus, a pool may contain blocks in any one of three states. These states fit the following definition:

  • Untouched: memory that has not been allocated
  • free: memory that was made available but later declared "free" by CPython because it no longer contains important information
  • allocated: a space in memory that is actually used to store important information.

A singly linked list of free memory blocks is referenced by the free block pointer. In other words, a list of places where data can be stored. The allocator will receive some untouched blocks from the pool if more is required than the free blocks that are currently available.

Blocks that are now "free" are added to the forefront of the free block list as the memory manager makes them so. The actual list might not be made up of consecutive memory blocks, as shown in the first delightful diagram. It might resemble the illustration below:

Python Memory Management

Arenas:

Pools are found in arenas. Those pools could be filled, empty, or in use. However, compared to pools, arenas don't have as many explicit states.

Instead, a doubly linked list termed usable arenas is used to organize arenas. According to how many free pools are available, the list is arranged. The arena is located nearer the top of the list, the fewer free pools there are.

Python Memory Management

This implies that new data will be added to the arena that already contains the most data. Why not, though, do the opposite? Why not store data in the location with the most space?

This introduces the notion of memory that is actually free. We will notice that I frequently use the word "free" in quotation marks. The cause is that, even though a block is marked as "free," the operating system never actually receives that memory back. The Python process reserves it and will use it in the future for fresh data. When memory is actually freed, the operating system can once again use it.

The only things that can surely be freed are arenas. Therefore, it makes sense to allow those arenas that really are close to being empty to do so. In this manner, the memory usage of our Python program as a whole can be decreased, and that portion of memory can actually be freed.

The ability to manage memory is essential for using computers. For better or worse, Python manages nearly all of it in the background. Many of the tedious details of working with computers are abstracted away by Python. This gives us the freedom to develop our code at a higher level without having to stress about where and how all those bytes are being stored.