MySQL operator deployment keeps crashing with exec format error on Kubernetes

I’m trying to set up a MySQL database operator on my Kubernetes cluster running Ubuntu Server 22.04. Everything seems to go fine during the initial setup, but I keep running into problems.

First I deployed the custom resource definitions and the operator itself:

kubectl apply -f https://raw.githubusercontent.com/mysql/mysql-operator/trunk/deploy/deploy-crds.yaml
kubectl apply -f https://raw.githubusercontent.com/mysql/mysql-operator/trunk/deploy/deploy-operator.yaml

These commands worked without any issues. However, when I check the deployment status, the pod is constantly restarting:

kubectl get deployment mysql-operator --namespace mysql-operator

When I look at the logs, I see this error:

kubectl logs mysql-operator-755b988dd9-jn7f6 --namespace=mysql-operator
exec /usr/bin/mysqlsh: exec format error

The pod describe shows it’s in CrashLoopBackOff state with restart count of 13. The container image being used is container-registry.oracle.com/mysql/community-operator:8.0.33-2.0.10 and it’s trying to run mysqlsh with various arguments.

Has anyone encountered this exec format error before? What could be causing this issue and how can I fix it?

Been there. That exec format error screams architecture mismatch - your container image was built for x86 but you’re running on ARM, or the other way around.

I used to waste hours manually debugging these deployment failures. Now I skip the Kubernetes headache and automate the whole database setup instead.

Build a workflow that detects your environment’s architecture, grabs the right MySQL container, and provisions everything automatically. Throw in fallback logic for when the primary image craps out.

The workflow watches deployment health and keeps trying different configs until something sticks. No more CrashLoopBackOff nightmares.

I’ve done this for mixed architecture clusters - the workflow handles all the detection and image selection so I don’t have to.

Check out https://latenode.com for building these deployment workflows.

Yeah, arch issue for sure. Had the same problem with Oracle images on my Pi cluster. Run uname -m on your nodes first, then force pull with --platform=linux/amd64 or whatever arch you need. Oracle’s MySQL operator is way pickier about this than most operators I’ve dealt with.

That exec format error means your container image and Kubernetes nodes don’t match architecturally. I hit the same thing deploying MySQL operator in a mixed environment - some ARM nodes but Oracle’s image only worked on AMD64. Before you swap images, check your node architecture with kubectl describe node and look for the kubernetes.io/arch label. What fixed it for me was adding nodeSelector to the deployment manifest or using RuntimeClass to force scheduling on compatible nodes. You could also switch to a different MySQL operator image that supports your architecture - Percona’s MySQL operator has way better multi-arch support than Oracle’s. If you’re stuck with Oracle’s operator, check their container registry for architecture-specific tags or just run it on x86 nodes if you’ve got them.

This exec format error means your Kubernetes nodes and the container image have different architectures. The MySQL operator image you’re pulling was probably built for a different CPU type than your cluster uses. Run kubectl get nodes -o wide and check the ARCH column first. If you’re on ARM64 (Apple Silicon or ARM servers) but pulling an AMD64 image, that’s your problem right there. I hit the same issue moving from x86 to ARM nodes. You’ve got two options: find a multi-architecture image or pull the right architecture variant. Check if Oracle has ARM64 builds of their operator, or just use the official MySQL Helm charts - they usually handle architecture selection for you. You can confirm the mismatch by running docker manifest inspect container-registry.oracle.com/mysql/community-operator:8.0.33-2.0.10 to see what architecture the image actually is.

Architecture mismatch is your problem. Everyone’s giving manual fixes, but handling MySQL operator deployments by hand sucks.

I wasted tons of time on these container architecture issues and operator failures. Now I just automate the whole MySQL deployment.

Build a workflow that checks your cluster architecture first, then grabs the right MySQL image. When the Oracle operator craps out, it switches to Bitnami or Percona operators - they handle multi-arch way better.

The workflow watches your deployment and handles retries when stuff breaks. No more kubectl debugging or hunting down container tags.

I run this across different clusters and it works. The automation figures out your architecture and sets up MySQL automatically.

Build something like this with Latenode and skip the deployment headaches: https://latenode.com

The Problem:

You’re encountering the "exec format error" when deploying the MySQL operator on your Kubernetes cluster. This indicates an architecture mismatch between your Kubernetes nodes and the MySQL operator container image. The container is attempting to execute mysqlsh, but the executable format is incompatible with the architecture of your cluster nodes. This results in the operator pod constantly restarting in a CrashLoopBackOff state.

:thinking: Understanding the “Why” (The Root Cause):

The exec format error arises because the operating system’s architecture (e.g., ARM64, AMD64) doesn’t match the architecture for which the container image’s binaries were compiled. The container-registry.oracle.com/mysql/community-operator:8.0.33-2.0.10 image might only be built for amd64 (x86-64), while your Kubernetes nodes might be running on a different architecture, such as arm64. When the container tries to launch mysqlsh, the system cannot execute the binary due to this incompatibility.

:gear: Step-by-Step Guide:

  1. Identify the Architecture of Your Kubernetes Nodes: Determine the architecture of your Kubernetes nodes using the following command:

    kubectl get nodes -o wide
    

    Look for the ARCH column in the output. Common architectures are amd64 (x86-64) and arm64.

  2. Check the Container Image Architecture: Verify the architectures supported by the MySQL operator image:

    docker manifest inspect container-registry.oracle.com/mysql/community-operator:8.0.33-2.0.10
    

    This command will show the supported architectures. If amd64 is listed but your nodes are arm64, this confirms the architecture mismatch.

  3. Solutions for the Architecture Mismatch: You have several options:

    • Use a Multi-Architecture Image (Recommended): If possible, switch to a MySQL operator image that explicitly supports both amd64 and arm64. Many alternative operators, such as Bitnami’s or Percona’s, offer better multi-architecture support. Check their respective repositories or Helm charts for suitable images.

    • Deploy on Compatible Nodes: If you have both amd64 and arm64 nodes, modify your deployment’s nodeSelector to explicitly target only amd64 nodes. This ensures that the operator is scheduled on a compatible node:

      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: mysql-operator
      spec:
        # ... other deployment settings ...
        template:
          spec:
            nodeSelector:
              kubernetes.io/arch: amd64
      
    • Pull Architecture-Specific Image (Less Recommended): If using the Oracle image is a strict requirement and you know the architecture of your nodes, specify the platform during the pull:

      # For amd64 nodes:
      kubectl apply -f https://raw.githubusercontent.com/mysql/mysql-operator/trunk/deploy/deploy-crds.yaml
      kubectl apply -f https://raw.githubusercontent.com/mysql/mysql-operator/trunk/deploy/deploy-operator.yaml --platform=linux/amd64
      
      # For arm64 nodes (replace with correct architecture if needed):
      kubectl apply -f https://raw.githubusercontent.com/mysql/mysql-operator/trunk/deploy/deploy-crds.yaml
      kubectl apply -f https://raw.githubusercontent.com/mysql/mysql-operator/trunk/deploy/deploy-operator.yaml --platform=linux/arm64
      
  4. Redeploy the Operator: After selecting and applying the correct solution, redeploy the operator and monitor its status.

:mag: Common Pitfalls & What to Check Next:

  • Incorrect Architecture Identification: Double-check that you have correctly identified the architecture of both your nodes and the supported architectures of the container image. A simple typo can lead to further issues.
  • Node Selector Conflicts: If using node selectors, ensure there are enough resources and nodes available to support the deployment.
  • Network Connectivity: Ensure your Kubernetes nodes have proper network connectivity to the container registry.
  • Image Pull Secrets: If the image is in a private registry, verify that you have the correct image pull secrets configured in your Kubernetes cluster.

:speech_balloon: Still running into issues? Share your (sanitized) config files, the exact command you ran, and any other relevant details. The community is here to help!

This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.