How many concurrent threads? (GNU Parallel)

Questions

Is it about threads, processors, cores, sockets or what?
Do I need to optimize myself for the number of threads? Or just leave this up to GNU Parallels?
What are the effects for sub-optimized cases?
Answers

What cylinders are in a car, are processors or processing units in a computer. See Processors, cores & threads on this computer (Bash) for details.
GNU Parallel clearly knows what the optimal number of threads is. See below in the testcode for the case with sem -j +0: Here the number of threads is the same as the number of processors, and the statistics confirm this
When optimizing manually, rather choose a bit too high a number of threads, than too low. However, this very much depends on the use case. E.g.: If CPU power is the bottleneck or I/O - I'm quite sure that for me, it's usually CPU-power, though.
Test scripts

################################################################################
# Thread optimalisation
################################################################################
#
# My laptop can do 8 threads. Let's see what happens to performance when I
# force more or less threads:
#
#
# parallel_test_function()
########################################
#
function parallel_test_function()
{
	printf "PTF - Start... "
	i=0
	for ((i; i<=100000; i++))
	do
		i=$i+1
		i=$i-1
	done
	printf "Done. "
}
export -f parallel_test_function


# # Test - 8 threads
# ########################################
# #
# # * Execution time (s): 5, 5, 5, 5 ⇒ 5s
# #
# start=`date +%s`
# #
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# sem -j 8 parallel_test_function
# #
# sem --wait
# end=`date +%s`
# echo ""; echo Execution time was `expr $end - $start` seconds.


# Test - 16 threads
########################################
#
# * Execution time (s): 5, 5, 5, 5, 5 ⇒ 5s
#
# start=`date +%s`
# #
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# sem -j 16 parallel_test_function
# #
# sem --wait
# end=`date +%s`
# echo ""; echo Execution time was `expr $end - $start` seconds.


# # Test - 32 threads
# ########################################
# #
# # * Execution time (s): 6, 6, 5, 5, 6 ⇒ 5.6s
# #
# start=`date +%s`
# #
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# sem -j 32 parallel_test_function
# #
# sem --wait
# end=`date +%s`
# echo ""; echo Execution time was `expr $end - $start` seconds.


# # Test - 4 threads
# ########################################
# #
# # * Execution time (s): 5, 6, 5, 6, 6, 5 ⇒ 5.5s
# #
# start=`date +%s`
# #
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# sem -j 4 parallel_test_function
# #
# sem --wait
# end=`date +%s`
# echo ""; echo Execution time was `expr $end - $start` seconds.


# # Test - 2 threads
# ########################################
# #
# # * Execution time (s): 7, 6, 6, 7, 7, 7 ⇒ 6.7s
# #
# start=`date +%s`
# #
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# sem -j 2 parallel_test_function
# #
# sem --wait
# end=`date +%s`
# echo ""; echo Execution time was `expr $end - $start` seconds.


# # Test - 1 thread
# ########################################
# #
# # * Execution time (s): 12, 12, 12, 12 ⇒ 12s
# #
# start=`date +%s`
# #
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# sem -j 1 parallel_test_function
# #
# sem --wait
# end=`date +%s`
# echo ""; echo Execution time was `expr $end - $start` seconds.


# Test - Auto-optimized
########################################
#
# * Execution time (s): 5, 5, 5, 5, 5 ⇒ 5s
#
start=`date +%s`
#
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function
sem -j +0 parallel_test_function

#
sem --wait
end=`date +%s`
echo ""; echo Execution time was `expr $end - $start` seconds.
How many concurrent threads? (GNU Parallel)

Inhoud

Questions

Answers

Test scripts

Sources

Navigatiemenu

Zoeken
