Training not working. "Skipping training since max_steps has already saved"

Question

I'm trying to train an object detector with TensorFlow. I downloaded the pre-trained model ssd_mobilenet_v1_coco_2018_01_28, created my .pbtxt file, my train and test records.

I'm using the model_main.py which is in TensorFlow/models/research/object-detection

When I try to start training "python model_main.py --logtostderr model_dir=results/ --pipeline_config_path=training/ssd_mobilenet_v1_pets.config --model_dir=ssd_mobilenet_v1_coco_2018_01_28/"

I get this error:

WARNING:tensorflow:Estimator's model_fn (.model_fn at 0x125031b00>) includes params argument, but params are not passed to Estimator. W1025 21:53:42.973139 4590234944 model_fn.py:630] Estimator's model_fn (.model_fn at 0x125031b00>) includes params argument, but params are not passed to Estimator. INFO:tensorflow:Not using Distribute Coordinator. I1025 21:53:42.973598 4590234944 estimator_training.py:186] Not using Distribute Coordinator. INFO:tensorflow:Running training and evaluation locally (non-distributed). I1025 21:53:42.973851 4590234944 training.py:612] Running training and evaluation locally (non-distributed). INFO:tensorflow:Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600. I1025 21:53:42.974123 4590234944 training.py:700] Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps None or save_checkpoints_secs 600. INFO:tensorflow:Skipping training since max_steps has already saved. I1025 21:53:42.978157 4590234944 estimator.py:360] Skipping training since max_steps has already saved.

How can I fix this and start training??

Answer 1

I had the same problem with model_main.py.

Try train.py from TensorFlow/models/research/object-detection/legacy/ instead.

I used it with the following arguments:

python train.py --logtostderr --train_dir=training/ --pipeline_config_path=training/ssd_mobilenet_v1_coco.config

This worked for me.