Python Parallel Processing
By performing more jobs concurrently, your software may complete more tasks in a shorter amount of time. These aid in solving major issues. The following subjects will be covered in this section:
Basics of Parallel Processing
- Python parallel processing library for multiple processing
- The problem must be divided into components that are independent of one another to be parallelized (or less dependent).
- The term embarrassingly parallel refers to a situation where the sub-units are completely independent of one another.
- As an illustration, consider an array's element-wise operation. In this situation, the operation must be aware of the specific aspect it is managing right now.
- In a different instance, an issue that has been broken down into smaller pieces needs to share certain data to operate.
- Due to the communication expense, this causes a performance issue.
- To manage concurrent programmes, there are two main approaches:
- The sub-units willcommunicate with one another using the same memory area in shared memory.
- The benefit is that you can write and read from the memory shared using this method without explicitly handling communication.
- However, a difficulty occurs when numerous processes attempt to access and modify the same memory address simultaneously.
- Through the use of synchronization mechanisms, this conflict can be avoided.
- Each process has its memory area and is completely independent of distributed memory.
- One method for achieving parallelism on memory which is shared is to use threads. These are the autonomous sub-tasks that share memory and are descended from a process.
- The Global Interpreter Lock (GIL) prevents threads from being used in Python to boost performance.
- GIL is a device that restricts the number of Python instructions that can run concurrently in a Python interpreter. Using processes rather than threads will get around GIL limitations.
- Compared to shared memory, using processes has fewer drawbacks, like less efficient inter-process communication, but it is more flexible and explicit.
Multi Processing in Parallel
- We may effectively parallelize straightforward operations using the built-in multiprocessing module by spawning child processes.
- This module has a user-friendly interface and several tools for managing task submission and synchronisation.
Import multiprocessing Import time Class demo(multiprocessing.process): def __init__(self, if): super(process, self).__init__() self.if = id def run(self) time.sleep(1) print(“the process id is “.format(self.id)) if __name__ == “__main__”: p = process(0) p.start() p.join() p = process(1) p.start() p.join()
The process id is :0 The process id is :1
- We must initialise our Process object and call the Process.start() method in order to spawn the process.
- Here, the method Process.start() will start a new process and call the method Process.run().
- The code that follows p.start() will be run just before process p completes its work. You can use the Process.join command to wait for task to finish ()