multithreading - Python pool.map for list of tuples -


i've got following issue. i'm trying refactor code in order process api calls using multithreading. core data simple list of tuples in following format:

lst = [('/users/sth/photo1.jpg',       '/users/sth/photo2'),       ('/users/sth/photo1.jpg',       '/users/sth/photo3'), (...)] 

function use takes lst list , process through api requires pair of photos. after single number returned each pair. far, i'm using loop put tuple function , produce mentioned number. paralellize whole computation in way 1 process takes part of list , calls function tuples inside batch. trying use pool function multiprocessing module:

from multiprocessing.dummy import pool threadpool  pool = threadpool(2) results = pool.map(score_function, lst) 

however, following error occurs:

ioerror: [errno 2] no such file or directory: 'u' 

something strange happening here. tries treat single character tuple argument. ideas how properly?

thank you

@edit

the lack of score_function definition bad. let me update question:

def score_function(pairs):     score_list = list()      pair in pairs:        score = findelement(target = pair[0], source = pair[1])        score_list.append([pair[0], pair[1], score])      return score_list 

where findelement defined as:

def findelement(target, source):      open(source, 'rb') source_:         source_bytes = source_.read()      open(target, 'rb') target_:         target_bytes = target_.read()      score = api_request(target_bytes = target_bytes,                         source_bytes = source_bytes)     return score 

your problem loop. breaks tuple individual strings. , should work:

def score_function(pairs):     score_list = list()      score = findelement(target = pairs[0], source = pairs[1])     score_list.append([pairs[0], pairs[1], score])      return score_list 

you assumed score_function receive lst variable parameter. not happen. lst needs list in case, , pool.map automatically splits individual elements , feeds 1 element score_function , keeps doing until whole list has been processed workers. each call worker receives parameter 1 element supposed work on. individual elements tuples (path1, path2), , when call for on tuple, receive 1 single path (string) in loop , pair[1] second character of string.

hope helps.


Comments

Popular posts from this blog

python Tkinter Capturing keyboard events save as one single string -

android - InAppBilling registering BroadcastReceiver in AndroidManifest -

javascript - Z-index in d3.js -