upload files from kaggle to google drive

Question

I am using kaggle to train a model and once training is done I would like upload the trained model to google drive as I cant figure out a way to download the model locally. I looked into using https://pythonhosted.org/PyDrive/ after doing pip install pydrive I tried authenticating

    import os
    from pydrive.auth import GoogleAuth
    from pydrive.drive import GoogleDrive
    from oauth2client.client import GoogleCredentials

then

gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)

I get this error ApplicationDefaultCredentialsError: The Application Default Credentials are not available. They are available if running in Google Compute Engine. Otherwise, the environment variable GOOGLE_APPLICATION_CREDENTIALS must be defined pointing to a file defining the credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.

Is there another way to do this?

Also I have not commited my files in kaggle as I stop the training manually so the commit would go on forever and i get a more than 6 subdirectories error after I tried commiting and stopping a commit in the output area

Answer 1

I was facing the same problem and was able download files from Kaggle to Colab then move to Google Drive. For example, if the current directory is /kaggle/working and the file to move is processed_file.zip then,

From Kaggle

from IPython.display import FileLink
FileLink(r'processed_file.zip')

This will generate a link,

https://....kaggle.net/...../processed_file.zip

From Colab

!wget "https://....kaggle.net/...../processed_file.zip"

Mount Google Drive

from google.colab import drive
drive.mount('/content/drive')

Copy File to Google Drive

!cp "/content/processed_file.zip" "/content/drive/My Drive/workspace"

Answer 2

I was trying to do the same: to save my kaggle model to google drive.

When you call gauth.Authorize(), the gauth object is looking for a valid gauth.credentials.

Instead of using gauth.credentials = GoogleCredentials.get_application_default(), I'm using:

# Try to load saved client credentials
gauth = GoogleAuth()
credential_file = 'gdrive.json'
gauth.LoadCredentialsFile(credential_file)

The credential file is created at your local computer by:

gauth = GoogleAuth()
credential_file = 'gdrive.json'
gauth.LocalWebserverAuth()
gauth.SaveCredentialsFile(credential_file)

After getting the the credential file, upload it to kaggle then you can use it in your session.

The credential will expire but I tried my kaggle session after 8 hours and it's still working.

Checking the expire time using:

gauth.credentials.token_expiry