How to run parallel tasks in Python

concurrent.futures module provides a high-level interface for launching parallel tasks in Python. Parallel tasks can be launched with threads or with processes. For threads concurrent.futures has class ThreadPoolExecutor and for processes it has class ProcessPoolExecutor. In this article we will see how to implement ThreadPoolExecutor.

Parallel execution using threads in Python

Import modules


import concurrent.futures
from timeit import default_timer as timer
import time

Calculate Time required for running the tasks without implementing ThreadPoolExecutor


import concurrent.futures
from timeit import default_timer as timer
import time

def test_func(value):
    msg = f"Getting  {value } from list_of_values"
    time.sleep(1)
    print(msg)

list_of_values = ["first", "second", "third", "fourth", "fifith", "sixth"]

start = timer()
for i in list_of_values:
    test_func(i)
end = timer()
print("Time taken without parallel execution is ",end - start)

Output


  Getting  first from list_of_values
  Getting  second from list_of_values
  Getting  third from list_of_values
  Getting  fourth from list_of_values
  Getting  fifith from list_of_values
  Getting  sixth from list_of_values
  Time taken without parallel execution is  6.023686041785554

So it took around 6 seconds to complete the task with parallel execution, lets implement parallel execution and calculate the execution time.

Execute the below code and check the execution time


import concurrent.futures
from timeit import default_timer as timer
import time

def test_func(value):
    msg = f"Getting  {value } from list_of_values"
    time.sleep(1)
    print(msg)


list_of_values = ["first", "second", "third", "fourth", "fifith", "sixth"]

start = timer()
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
    executor.map(test_func,list_of_values)
end = timer()
print("Time taken with parallel execution is ",end-start)

Output


Getting  third from list_of_values
Getting  fourth from list_of_values
Getting  first from list_of_values
Getting  second from list_of_values
Getting  fifith from list_of_values
Getting  sixth from list_of_values
Time taken with parallel execution is  2.01107863221897

We can see parallel execution reduced time significantly as it took only 2 seconds this time in comparison to 6 Seconds previously without implementing ThreadPoolExecutor.

Categories: Python