Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Function I found useful - blockwise indexed task submission #35

Open
UH-Comet-Group opened this issue Oct 11, 2018 · 1 comment
Open

Comments

@UH-Comet-Group
Copy link

UH-Comet-Group commented Oct 11, 2018

I found this function useful - sort of a blockwise pdotimes. One could do the same with map, but
this turned out to be simpler to use for me. If I'm not missing something, it might be a useful standard functionality (after changing the awful name).

I think it's not quite the same as pdotimes, because pdotimes calls its tasks by singles,
so that any overheads are repeated. Below, the task given by func can have expensive overheads.

;; utility function that takes a task indexed i1..i2, and
;; calls (func j1 j2) in parallel to span ranges of i1,i2

(defun chop-up-indexed-tasks-into-threads (i1 i2 func &key (nthreads nil))
  (loop with ncpu = (or nthreads (lparallel:kernel-worker-count))
	with ntot = (- i2 i1)
	with nblock = (ceiling ntot ncpu) ;; number in a block
	with ithread = 0
	with channel = (lparallel:make-channel)
	for j1 = i1 then (+ 1 j1 nblock)
	for j2 = (min (+ j1 nblock) i2)
	do (lparallel:submit-task channel func j1 j2)
	   (incf ithread)
	until (= j2 i2)
	finally
	   (loop for k below ithread
		 do (lparallel:receive-result channel))))
@defunkydrummer
Copy link

Upvoting, this would be very useful to have as standard.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants