Python Parallel Processing
By performing more jobs concurrently, your software may complete more tasks in a shorter amount of time. These aid in solving major issues. The following subjects will be covered in this section:
Basics of Parallel Processing
- Python parallel processing library for multiple processing
- The problem must be divided into components that are independent of one another to be parallelized (or less dependent).
- The term embarrassingly parallel refers to a situation where the sub-units are completely independent of one another.
- As an illustration, consider an array's element-wise operation. In this situation, the operation must be aware of the specific aspect it is managing right now.
- In a different instance, an issue that has been broken down into smaller pieces needs to share certain data to operate.
- Due to the communication expense, this causes a performance issue.
- To manage concurrent programmes, there are two main approaches:
Shared Memory
- The sub-units willcommunicate with one another using the same memory area in shared memory.
- The benefit is that you can write and read from the memory shared using this method without explicitly handling communication.
- However, a difficulty occurs when numerous processes attempt to access and modify the same memory address simultaneously.
- Through the use of synchronization mechanisms, this conflict can be avoided.
Distributed memory
- Each process has its memory area and is completely independent of distributed memory.
- One method for achieving parallelism on memory which is shared is to use threads. These are the autonomous sub-tasks that share memory and are descended from a process.
- The Global Interpreter Lock (GIL) prevents threads from being used in Python to boost performance.
- GIL is a device that restricts the number of Python instructions that can run concurrently in a Python interpreter. Using processes rather than threads will get around GIL limitations.
- Compared to shared memory, using processes has fewer drawbacks, like less efficient inter-process communication, but it is more flexible and explicit.
Multi Processing in Parallel
- We may effectively parallelize straightforward operations using the built-in multiprocessing module by spawning child processes.
- This module has a user-friendly interface and several tools for managing task submission and synchronisation.
Import multiprocessing
Import time
Class demo(multiprocessing.process):
def __init__(self, if):
super(process, self).__init__()
self.if = id
def run(self)
time.sleep(1)
print(“the process id is “.format(self.id))
if __name__ == “__main__”:
p = process(0)
p.start()
p.join()
p = process(1)
p.start()
p.join()
Output:
The process id is :0
The process id is :1
- We must initialise our Process object and call the Process.start() method in order to spawn the process.
- Here, the method Process.start() will start a new process and call the method Process.run().
- The code that follows p.start() will be run just before process p completes its work. You can use the Process.join command to wait for task to finish ()