GNU Parallel

Run commands or scripts in the shell in parallel. You can even use different PCs.

This command can send data and scripts to the other PC, collect the results and clean up.

Tutorials

Very good video tutorial : https://www.youtube.com/watch?v=OpaiGYxkSuQ&list=PL284C9FF2488BC6D1&index=1

Examples

For example, create a compressed file for each subfolder of PARENT-PATH: 

 parallel -eta -j 4 --noswap tar cJf /PARENT-PATH/{= s:/$:: =}.tar.xz {} ::: */
  • -eta : tries to guess how long it will need for the job
  • -j 4 : run with four tasks
  • –noswap : don't swap
Log In