Failed to pull image repository does not exist or may require 'docker login':

Question

I am receiving this error when starting a Pod

Failed to pull image "docker-username/docker-private-repository:latest": rpc error: code = Unknown desc = Error response from daemon: pull access denied for docker-username/docker-private-repository, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

My setup is as follow:

Ceated a Secret service using command line

kubectl create secret docker-registry docker-keys --docker-username=docker-username --docker-password=password [email protected] --docker-server=https://index.docker.io/v1

which generates the following data inside secrets

kubectl get secret docker-keys -o json | jq '.data | map_values(@base64d)'
{
  ".dockerconfigjson": "{\"auths\":{\"https://index.docker.io/v1\":{\"username\":\"docker-username\",\"password\":\"password\",\"email\":\"[email protected]\",\"auth\":\"base64encodedtoken\"}}}"
}

Then in deployment I am using docker-keys secrets

apiVersion: apps/v1
kind: Deployment
metadata:
  name: docker-private-repository
  labels:
    app: docker-private-repository
spec:
  replicas: 1
  selector:
    matchLabels:
      app: docker-private-repository
  template:
    metadata:
      labels:
        app: docker-private-repository
    spec:
      imagePullSecrets:
        - name: docker-keys
      containers:
        - name: docker-private-repository
          image: docker-username/docker-private-repository:latest
          imagePullPolicy: Always
          ports:
            - containerPort: 3000

I did tried to search for a solution, but always ended up with the above setup.

Update1: Secrets service and pod are running in the same namespace default. Pulling manually from docker-hub works

docker pull docker-username/docker-private-repository:latest
latest: Pulling from docker-username/docker-private-repository
0ecb575e629c: Already exists 
...
7467d1831b69: Already exists 
Digest: sha256:153643ecb19c2ce54635839ce9393b2e256ce6c34a2fe75b91c7a41525a6a535
Status: Downloaded newer image for docker-username/docker-private-repository:latest
docker.io/docker-username/docker-private-repository:latest

Update2 kubectl describe pod

I have 2 sercrets services, one for dockerhub credentials and another is token-rzlx6 for whatever reason. Thing is when I run describe pod, I don't see the secrets for dockerhub to be mounted as token-rzlx6, could this be the reason? And why it is not mounted?

...
Volumes:
  default-token-rzlx6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-rzlx6
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
...

Answer 1

So the problem was the docker-server value. Based on tutorials I've watched I was using api V1, while my image was pushed/created to dockerhub with V2, may be it's not relevant and V1 was deprecated. When I create a secret service I need to use V2 server https://index.docker.io/v2/, like:

kubectl create secret docker-registry docker-keys \
  --docker-username=yyyyyy \
  --docker-password=xxxxx \
  [email protected] \
  --docker-server=https://index.docker.io/v2/

A simple thing, that took days to discover, as many articles have V1 or it's not showing it at all or using private docker registry. Docs are here. https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

Update:

One more thing why I wasn't paying attention to API version is docker config file, that has V1 as API and from here I took the endpoint to create secret service, while it worked with V2 only.

cat ~/.docker/config.json                                                 
{
    "auths": {
        "https://index.docker.io/v1/": {}
    },
    "credsStore": "osxkeychain"
}%                              

Update 2: However, when I pulled image locally with docker pull command, image was pulled successfully using v1 url. Assumption is, api V1 works within docker, but not in kubernetes.