multithreading - Python pool.map for list of tuples -
i've got following issue. i'm trying refactor code in order process api calls using multithreading. core data simple list of tuples in following format:
lst = [('/users/sth/photo1.jpg', '/users/sth/photo2'), ('/users/sth/photo1.jpg', '/users/sth/photo3'), (...)]
function use takes lst list , process through api requires pair of photos. after single number returned each pair. far, i'm using loop put tuple function , produce mentioned number. paralellize whole computation in way 1 process takes part of list , calls function tuples inside batch. trying use pool function multiprocessing module:
from multiprocessing.dummy import pool threadpool pool = threadpool(2) results = pool.map(score_function, lst)
however, following error occurs:
ioerror: [errno 2] no such file or directory: 'u'
something strange happening here. tries treat single character tuple argument. ideas how properly?
thank you
@edit
the lack of score_function definition bad. let me update question:
def score_function(pairs): score_list = list() pair in pairs: score = findelement(target = pair[0], source = pair[1]) score_list.append([pair[0], pair[1], score]) return score_list
where findelement defined as:
def findelement(target, source): open(source, 'rb') source_: source_bytes = source_.read() open(target, 'rb') target_: target_bytes = target_.read() score = api_request(target_bytes = target_bytes, source_bytes = source_bytes) return score
your problem loop. breaks tuple individual strings. , should work:
def score_function(pairs): score_list = list() score = findelement(target = pairs[0], source = pairs[1]) score_list.append([pairs[0], pairs[1], score]) return score_list
you assumed score_function
receive lst
variable parameter. not happen. lst
needs list in case, , pool.map automatically splits individual elements , feeds 1 element score_function
, keeps doing until whole list has been processed workers. each call worker receives parameter 1 element supposed work on. individual elements tuples (path1, path2), , when call for
on tuple, receive 1 single path (string) in loop , pair[1] second character of string.
hope helps.
Comments
Post a Comment