Python Logging Module, logfile issue: PermissionError: [WinError 32] The process cannot access the file because it is being used by another process

Question

I have never used logging before and am new to Python. My mentor mandated that script must contain a logging file. So I tried to do it in my code following his template. The followign below are excerpts in my code where logging is used:

logfilepath = r"C:\Users\sys_nsgprobeingestio\Documents\dozie\odfs\odfshistory\CSV_ODFS_Probe_Quality-%Y-%m-%d.log"

log_file_name = datetime.now().strftime(logfilepath)
print(log_file_name)

logging.basicConfig(
    filename=log_file_name,
    level=logging.DEBUG,
    format='[Probe Data Quality] %(asctime)s - %(name)s %(levelname)-7.7s %(message)s'    #can you explain this Tenzin?
)

def process_dirconfig_file(config_file_from_sysarg):
    logging.info('started processing config file' + (config_file_from_sysarg) )
    dirconfig_file_Pobj = Path(config_file_from_sysarg)
    try:
        if Path.is_file(dirconfig_file_Pobj):
            try:
                if ("db_string" not in parseddict or parseddict["db_string"] == ""):
                    raise Exception(f"Error: Your config file is missing 'db connection string' for db connection. Please edit config file to section[db string] and db string key val pairs of form: db_string = <db string>")
                    #print(f"Error: Your config file is missing 'error directory' for file processing")

            except Exception as e:
                raise Exception(e)  #logging.exception(e) puts it into log file
                logging.exception(e)
            return parseddict
        else:
            raise Exception(f"Error: No directory config file. Please create a config file of directories to be used in processing")
    except Exception as e:
        raise Exception(e)
        logging.exception(e)

if __name__ == '__main__':
    try:
        startTime = datetime.now()
        parse_dict = process_dirconfig_file(dirconfig_file)
        db_instance = dbhandler(parse_dict["db_string"])
        odfs_tabletest_dict = db_instance['odfs_tester_history_files']
        odf_history_from_csv_to_dbtable(db_instance)
        print(datetime.now() - startTime)
    except Exception  as e:
        logging.exception(e)

Why does this structure give me this error? Of course most of my code is edited out of this for brevity. THe code works. The logging is what's breaking it. I'll also note that a bunch of my functions are used in for loops but I wouldn't think this would be an issue. It's not like log files cannot work in for loops. ANd no I haven't done any threading.

Also how can you make it create a new log file for each run. Right now, the logging info gets appeneded to the logfile?

Error:

[Probe Data Quality] 2020-07-02 09:21:04,217 - root INFO    started processing config fileC:\Users\sys_nsgprobeingestio\Documents\dozie\odfs\venv\odfs_tester_history_dirs.ini
[Probe Data Quality] 2020-07-02 09:21:04,373 - root INFO    started processing config fileC:\Users\sys_nsgprobeingestio\Documents\dozie\odfs\venv\odfs_tester_history_dirs.ini
[Probe Data Quality] 2020-07-02 09:21:04,420 - root ERROR   [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\sys_nsgprobeingestio\\Documents\\dozie\\odfs\\odfshistory\\CSV_ODFS_Probe_Quality-2020-07-02-09-21-04.log' -> 'C:\\Users\\sys_nsgprobeingestio\\Documents\\dozie\\odfs\\odfshistory\\error\\CSV_ODFS_Probe_Quality-2020-07-02-09-21-04.log'
Traceback (most recent call last):
  File "C:/Users/sys_nsgprobeingestio/Documents/dozie/odfs/odfshistory3.py", line 277, in <module>
    odf_history_from_csv_to_dbtable(db_instance)
  File "C:/Users/sys_nsgprobeingestio/Documents/dozie/odfs/odfshistory3.py", line 251, in odf_history_from_csv_to_dbtable
    csv.rename(trg_path.joinpath(csv.name))
  File "C:\ProgramData\Anaconda3\lib\pathlib.py", line 1329, in rename
    self._accessor.rename(self, target)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\sys_nsgprobeingestio\\Documents\\dozie\\odfs\\odfshistory\\CSV_ODFS_Probe_Quality-2020-07-02-09-21-04.log' -> 'C:\\Users\\sys_nsgprobeingestio\\Documents\\dozie\\odfs\\odfshistory\\error\\CSV_ODFS_Probe_Quality-2020-07-02-09-21-04.log'

Answer 1

Looks like odf_history_from_csv_to_dbtable is trying to rename the file while the logger is still using it? might need to move that logic to someplace else, or re-think what it's trying to do to happen after your main script runs. self._accessor.rename(self, target) and looks like it's trying to move it to an error directory ?

As far as having a unique name for each run of the script:

from uuid import uuid4

run_id = uuid4()
logfilepath = f"C:\Users\sys_nsgprobeingestio\Documents\dozie\odfs\odfshistory\CSV_ODFS_Probe_Quality-%Y-%m-%d-{run_id}.log"