The Problem

I've been managing production services in my home lab, including running manual backups. This weekend, I decided to automate backups for my MongoDB in Kubernetes. This would not only save me time and protect my data but also offer me an opportunity to dive into Kubernetes CronJobs which I had been meaning to experiment with for the longest time.

The Project: k8s-mongodump-s3

I decided to create a script designed to work with both MinIO(a cloud agnostic self hosted s3 compliant storage) and S3. This script runs a daily MongoDB dump, thanks to the utility mongodump, and uploads the result to the configured storage endpoint. The project is cloud-agnostic, working seamlessly with different cloud providers. Here's how it's configured and what it offers.

Configuration

Configurable via environment variables, this CronJob needs specific details like MongoDB connection string, S3 or MinIO endpoint, and storage path. Additional options include prefixing dump files, MongoDB credentials, query filters, and S3 or MinIO access keys.

Building and Running with Docker

The image thegalah/k8s-mongodump-s3:1.0.0 can be built and run with Docker. Full instructions and environment variable details can be found in the project's readme.

Running as a Kubernetes CronJob

Here's an example YAML for the CronJob, including optional credentials:

apiVersion: batch/v1beta1
kind: CronJob
metadata:
    name: mongodb-backup-cronjob
spec:
    schedule: "0 0 * * *" # Runs daily at midnight
    jobTemplate:
        spec:
            template:
                spec:
                    containers:
                        - name: mongodb-backup
                          image: thegalah/k8s-mongodump-s3:1.0.0
                          env:
                              - name: MONGO_CONNECTION_STRING
                                value: your_connection_string
                              - name: S3_OR_MINIO_ENDPOINT
                                value: your_s3_or_minio_endpoint
                              - name: STORAGE_PATH
                                value: your_storage_path
                              - name: DUMP_PREFIX
                                value: your_prefix # Optional
                              - name: MONGO_USERNAME
                                value: your_username # Optional
                              - name: MONGO_PASSWORD
                                valueFrom: # Optional
                                    secretKeyRef:
                                        name: mongodb-password-secret
                                        key: password
                              - name: MONGO_DATABASE
                                value: your_database # Optional
                              - name: MONGO_COLLECTION
                                value: your_collection # Optional
                              - name: MONGO_QUERY
                                value: your_query # Optional
                              - name: ACCESS_KEY
                                value: your_access_key # Optional
                              - name: SECRET_KEY
                                value: your_secret_key # Optional
                    restartPolicy: OnFailure

Features

Daily MongoDB dump: From a given connection string. Customizable: Including a configurable file prefix. Cloud-Agnostic: Uploads to either an S3 or MinIO endpoint, with optional no-sign request.

Open Source & Licensing

The project is licensed under the MIT License and actively welcomes contributions through issues or pull requests. The open-source nature of the project ensures that it's adaptable to various needs and allows the community to benefit from shared solutions.

Conclusion

This weekend project to create a MongoDB Backup Kubernetes CronJob exemplifies how hosting your own production services can drive innovative solutions. From running a personal MinIO cluster to leveraging Kubernetes CronJobs, this project showcases the flexibility of cloud-native technologies. Whether you run your databases in your home lab or in a large-scale production environment, a robust backup strategy is essential, and CronJob provides a simple, cloud-agnostic solution. Find all details, including a full list of configurable environment variables, in the GitHub repository. https://github.com/thegalah/k8s-mongodump-s3