Xtykutl

I must process around 45 000 items, it takes about 4 hours on a single core.

I want

• a progress bar to check the process is going forward


• to log errors while the process is running to eventually stop it if I observe it isn't going well enough.

The best I came with is displaying a progress bar but printing logs only at the very end of the process. It involves my function return a string (the log line) and print returned line after getting all the results. The real work creates files.

I get that I get all log lines at the very end of the process because I build the list of result (thus ending the process) before iterating over the log lines. But as I'm not used to multi threading, I don't see how to achieve what I'm aiming.

from multiprocessing import Pool

from tqdm import tqdm





def thread_safe_function(arg):

    # do things

    return log_line





if __name__ == '__main__':

    for log in list(tqdm(pool.imap(thread_safe_function, input_list), total=len(pronunciation_models))):

            print log

I guess the right solution would be to make my core function to print the logs but I didn't have any luck trying it. I believe either the threads my program created didn't share stdout with parent process (thus printing logs somewhere I wouldn't be able to read) or I misused Lock from multiprocessing.

def thread_safe_function(lock, arg):

    # do things

    if error:

        lock.acquire()

        print error

        lock.release()