Python Parallel Processing

By performing more jobs concurrently, your software may complete more tasks in a shorter amount of time. These aid in solving major issues. The following subjects will be covered in this section:

Basics of Parallel Processing

• Python parallel processing library for multiple processing
• The problem must be divided into components that are independent of one another to be parallelized (or less dependent).
•  The term embarrassingly parallel refers to a situation where the sub-units are completely independent of one another.
• As an illustration, consider an array's element-wise operation. In this situation, the operation must be aware of the specific aspect it is managing right now.
• In a different instance, an issue that has been broken down into smaller pieces needs to share certain data to operate.
• Due to the communication expense, this causes a performance issue.
• To manage concurrent programmes, there are two main approaches:

Shared Memory

• The sub-units willcommunicate with one another using the same memory area in shared memory.
• The benefit is that you can write and read from the memory shared using this method without explicitly handling communication.
• However, a difficulty occurs when numerous processes attempt to access and modify the same memory address simultaneously.
•  Through the use of synchronization mechanisms, this conflict can be avoided.

Distributed memory

• Each process has its memory area and is completely independent of distributed memory.
• One method for achieving parallelism on memory which is shared is to use threads. These are the autonomous sub-tasks that share memory and are descended from a process.
• The Global Interpreter Lock (GIL) prevents threads from being used in Python to boost performance.
• GIL is a device that restricts the number of Python instructions that can run concurrently in a Python interpreter. Using processes rather than threads will get around GIL limitations.
• Compared to shared memory, using processes has fewer drawbacks, like less efficient inter-process communication, but it is more flexible and explicit.

Multi Processing in Parallel

• We may effectively parallelize straightforward operations using the built-in multiprocessing module by spawning child processes.
• This module has a user-friendly interface and several tools for managing task submission and synchronisation.

Import multiprocessing
Import time
Class demo(multiprocessing.process):
	def __init__(self, if):
		super(process, self).__init__()
		self.if = id
	def run(self)
		time.sleep(1)
		print(“the process id is “.format(self.id))
if __name__ == “__main__”:
	p = process(0)
p.start()
p.join()
p = process(1)
p.start()
p.join()

Output:

The process id is :0
The process id is :1

• We must initialise our Process object and call the Process.start() method in order to spawn the process.
• Here, the method Process.start() will start a new process and call the method Process.run().
• The code that follows p.start() will be run just before process p completes its work. You can use the Process.join command to wait for task to finish ()