Last Updated: 2024-11-21

How would you parallelize a slow operation in a bash loop?

I started off with the following:

while read -r key; do
  aws s3api put-object-acl --bucket mybucket --key "$key" --grant-full-control 'emailaddress="someone@example.com"'
done < s3-keys.txt

This was too slow so I wanted a way to parallelize. I came up with the combination of backgrounding the slow command with ampersand, limiting the total number of processes, and waiting for all subprocesses to finish with the wait command once a certain number were running.

N=50 # set number of processes
while read -r key; do
  echo "Processing: $key"
  # The double parentheses construct permits arithmetic evaluation, and we do
  # some modulo math to determine whether to call `wait`
  ((i=i%N)); ((i++==0)) && wait
  aws s3api put-object-acl --bucket mybucket --key "$key" --grant-full-control 'emailaddress="someone@example.com"' &
  echo "Done\n"
done < s3-keys.txt

Resources

• https://unix.stackexchange.com/questions/103920/parallelize-a-bash-for-loop
• https://tldp.org/LDP/abs/html/dblparens.html