Back Up Your GOG Games Using Kubernetes, iSCSI and SMB
AI Preface: My opinions on the subject. Minor AI assistance was used for Dockerfile and entrypoint. The post and k8s yamls were hand written and repurposed from other projects.
This is an over-engineered solution to a problem that can be done on a single machine. Here I am using at least three to solve my problem, depends how you want to look at it. I would probably say more like four (control-plane k8s node, node running jobs, iSCSI server, SMB server - yes, that's literally the setup).
So Sean, Why?
Great Question! My previous paragraph was a hint. This is how my homelab is configured. In short, I use NFS/SMB shares for larger stores such as DEF CON media, games and music. iSCSI is used for persistent stores used in the k8s cluster in this case. Lastly, k8s is the foundation for my homelab and self-hosting.
The Plan
- Use gog-cli to locally back up GOG games
- Containerize the application to run in k8s
- Create a job for logging in
- Create a job for syncing
- [FUTURE] automate sync with notification
- [FUTURE] enhanced notification for re-login required
The Image
This is the bit of AI, used for the docker image; it's quick and dirty, not very refined.
The image is in short a very simple python based base image with the installed required dependencies, pull gog-cli package, created directories and entry point script.
FROM python:3.12-slim
# Prevent Python from writing .pyc files and buffer stdout/stderr
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
# Consolidate auth config and library cache into a single /data path
HOME=/data \
XDG_CONFIG_HOME=/data/config \
XDG_CACHE_HOME=/data/cache \
XDG_DATA_HOME=/data/share
# Install aria2 (used automatically by gog-cli for faster downloads) and git
RUN apt-get update && apt-get install -y --no-install-recommends \
aria2 \
ca-certificates \
git \
&& rm -rf /var/lib/apt/lists/*
# Install gog-cli from GitHub
RUN pip install --no-cache-dir git+https://github.com/aleksandarristic/gog-cli.git
# Create persistent directories for state (/data) and game backups (/backups)
RUN mkdir -p /data/config /data/cache /data/share /backups
WORKDIR /backups
COPY entrypoint.sh /usr/local/bin/entrypoint.sh
RUN chmod +x /usr/local/bin/entrypoint.sh
VOLUME ["/data", "/backups"]
ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
I ran into an error where if there is no manifest found, it just craps out. So, I created a condition where if a manifest file is found it runs a "sync", but, if one isn't found it will run the original "backup" command.
#!/bin/sh
set -e
# If arguments are passed to the container, run them directly with `gog`
if [ "$#" -gt 0 ]; then
gog "$@"
fi
echo "==> Refreshing GOG library metadata cache..."
gog refresh
if [ ! -f "/backups/metadata/manifest.json" ]; then
echo "==> No manifest found. Running initial 'gog backup' to /backups..."
gog backup --destination /backups --all --platform windows --language en --yes ${EXTRA_FLAGS} || true
else
echo "==> Existing manifest found. Running 'gog sync' to /backups..."
gog sync --destination /backups --all --platform windows --language en --yes ${EXTRA_FLAGS} || true
fi
These are obviously configurable command line executions. Change them to your desire. I also point out some nuances later on.
Gog Login Job
This job is simply used to login to Gog with the gog-cli tool. It will connect to the iSCSI volume used to store the persistent data such as the manifest file created.
apiVersion: v1
kind: Pod
metadata:
name: gog-login
spec:
imagePullSecrets:
- name: registry-secret
restartPolicy: Never
containers:
- name: gog-login
image: git.hosting.ca/docker-gog-cli:latest
args: ["auth", "login"]
stdin: true
tty: true
volumeMounts:
- name: gog-cli-vol
mountPath: /data
volumes:
- name: gog-cli-vol
iscsi:
targetPortal: 192.168.1.105:3260
iqn: iqn.2005-10.org.seanland-nas.ctl:gog-cli-iscsi
lun: 0
fsType: ext4
readOnly: false
chapAuthDiscovery: false
chapAuthSession: false
The job triggers the "auth login", now we simply have to go through the process. See the steps below.
Login to Gog Using the Job
seanland@birdie:~$ kubectl apply -f gogs-login.yml
pod/gog-login created
# Wait for the pod to be running
# Once running run the command:
seanland@birdie:~$ kubectl logs gog-login
Open this URL in your browser and log in:
https://auth.gog.com/auth?client_id=46899977096215655&redirect_uri=https%3A%2F%2Fembed.gog.com%2Fon_login_success%3Forigin%3Dclient&response_type=code&layout=client2
After logging in, paste the full redirect URL (or just the code value):
# Grab the code from the redirect URL and paste it into the pod
# You will get a URL like this: https://embed.gog.com/on_login_success?origin=client&code=Ajksdfa908uij2@#$J@4lK@#KAS*DFAIOSDFAjksdfa908uij2@#$J@4lK@#KAS*DFAIOSDFAjksdfa908uij2@#$J@4lK@#KAS*DFAIOSDFAjksdfa908uij2@#$J@4lK@#KAS*DFAIOSDF
# Take the piece AFTER "code" that will be what you paste. In this fake example you would paste: Ajksdfa908uij2@#$J@4lK@#KAS*DFAIOSDFAjksdfa908uij2@#$J@4lK@#KAS*DFAIOSDFAjksdfa908uij2@#$J@4lK@#KAS*DFAIOSDFAjksdfa908uij2@#$J@4lK@#KAS*DFAIOSDF
seanland@birdie:~$ kubectl attach -it gog-login
If you don't see a command prompt, try pressing enter.
Ajksdfa908uij2@#$J@4lK@#KAS*DFAIOSDFAjksdfa908uij2@#$J@4lK@#KAS*DFAIOSDFAjksdfa908uij2@#$J@4lK@#KAS*DFAIOSDFAjksdfa908uij2@#$J@4lK@#KAS*DFAIOSDF
keyring write failed — using file-only token storage
Logged in as cooluser1234.
# We can now delete off the pod. If we ever need to log in again, just repeat this.
seanland@birdie:~$ kubectl delete -f gogs-login.yml
pod "gog-login" deleted
To summarize, we triggered the "auth login" which provides you with a URL in the logs. You paste it into your browser, receive a code, that you need to paste back into the job to verify the authentication. After that has been complete you should see "Logged in as
Start Backing Up the Games!
This job holds two volumes, one for the metadata that we used earlier and a new one, the game store. Originally, we were using the iSCSI mount solely, now we are mounting the SMB share for the games.
apiVersion: batch/v1
kind: Job
metadata:
name: gog-sync-job
spec:
backoffLimit: 2
template:
spec:
imagePullSecrets:
- name: registry-secret
restartPolicy: Never
containers:
- name: gog-sync
image: git.hosting.ca/docker-gog-cli:latest
env:
# Optional: Add flags like "--win --role installer" if desired
- name: EXTRA_FLAGS
value: ""
volumeMounts:
- name: gog-cli-vol
mountPath: /data
- name: games-vol
mountPath: /backups
volumes:
- name: gog-cli-vol
iscsi:
targetPortal: 192.168.1.105:3260
iqn: iqn.2005-10.org.seanland-nas.ctl:gog-cli-iscsi
lun: 0
fsType: ext4
readOnly: false
chapAuthDiscovery: false
chapAuthSession: false
- name: games-vol
flexVolume:
driver: fstab/cifs
fsType: cifs
options:
mountOptions: dir_mode=0777,file_mode=0777,iocharset=utf8,noperm,nounix,rw,vers=3.0
networkPath: '//192.168.1.100/games/'
secretRef:
name: game-share-secret
Now, you will start the job, it will create or update the manifest file and process all of the games!
Run the Sync
# Apply the second job
seanland@birdie:~$ kubectl apply -f gogs-sync.yml
job.batch/gog-sync-job created
# Find the pod name, then run the below command to watch the games flow...
seanland@birdie:~$ kubectl logs gog-sync-job-xx5dg -f
==> Refreshing GOG library metadata cache...
keyring read failed — using file token storage
Fetching library page 1...
Fetched 73 library entries.
Refreshing download metadata for 73 games...
Download metadata 1/73: cached Aircraft Carrier Survival
Download metadata 10/73: cached Cannon Fodder
Download metadata 20/73: cached EVERSPACE™
Download metadata 30/73: cached Into the Breach
Download metadata 40/73: cached Mutant Football League: Dynasty Edition
Download metadata 50/73: cached Robin Hood: The Legend of Sherwood
Download metadata 60/73: cached Terra Nil
Download metadata 70/73: cached Way of the Hunter
Download metadata 73/73: cached Zeus + Poseidon (Acropolis)
Refreshed 73 games (+0 added, -0 removed, ~0 changed).
==> No manifest found. Running initial 'gog backup' to /backups...
Backup plan — /backups
Policy: platforms=all languages=en roles=all
Scope: 73 owned | 73 selected | 0 complete | 73 need downloads | 58 missing locally
Downloads: 330 file(s) • 254.3 GB estimated
Local state: 0 already present • 0 orphaned
Filtered out: 145 language-not-selected
────────────────────────────────────────────────────────────────────────
aircraft_carrier_survival — Aircraft Carrier Survival
+ en1installer0 installer windows 1.0 MB
+ en1installer1 installer windows 4.0 GB
+ en1installer2 installer windows 1.1 GB
alone_in_the_dark_1 — Alone in the Dark 1
...
Status Legend:
(OK):download completed.
downloaded Zeus + Poseidon (Acropolis) / extra / -
Summary: 319 succeeded, 11 failed.
Interesting Errors
1. Potential Duplicate Hashes Creating Conflict
Download metadata 74/74: cached Zeus + Poseidon (Acropolis)
Refreshed 74 games (+1 added, -0 removed, ~0 changed).
==> Existing manifest found. Running 'gog sync' to /backups...
Multiple source files map to the same backup path /backups/games/kerbal_space_program/installers/kerbal_space_program_enUS_1_12_5_03190_.pkg: 'en2installer0' and 'en2installer1'
Root cause seems to be related to similar hash on Linux and Mac versions. I decided my workaround was to just download the windows versions... and... run them on linux.
Something else I could do, modify the image to pass different parameters, or make the parameters externally modifiable having an image run to back up Linux, Mac and OS in different sub directories.
2. Exit Code Always Failed
f2d7c7|OK | 0B/s|/backups/games/zeus_poseidon/extras/zeus_poseidon_reference_card.zip
Status Legend:
(OK):download completed.
failed Zeus + Poseidon (Acropolis) / extra / - — aria2c_error: aria2c reported success but output file is missing
downloaded Sid Meier's Alpha Centauri™ Planetary Pack / extra / -
downloaded Sid Meier's Alpha Centauri™ Planetary Pack / extra / -
downloaded Sid Meier's Alpha Centauri™ Planetary Pack / extra / -
Summary: 3 succeeded, 82 failed.
It seems like there would always be a failure exit code based on my configuration. So, if I want the job to not spin off new versions, I have to trigger a complete or no retry. I figured just calling it a success was the best workaround.
Future Implementations
The one thing I need to do is cronjob / automate the task. Ideal end goal, have it run on a periodic basis and be provided a summary of what had occurred. I could also update the image to accept parameters for the operating system eliminating the duplicate hash issue; assuming it is caused by what I think it is.
The Blessing and Curse of Open Source
This is a perfect example of why Open Source is great and troublesome at the same time. First, thank you to Aleksandar Ristic (github, site) for making the gog-cli project, a foundational piece of this project. As, this may or may not have been possible with it. Let's look at my points specifically for this project:
- [BLESSING] The project was build to solve a fundamental problem I have.
- [CURSE] This problem may not have been solved exactly how I wanted to do it.
- [BLESSING] Saved a bunch of time building something from scratch.
- [CURSE] Required work to achieve what I wanted.
- [BLESSING] This software cost no money to deploy.
- [CURSE] No reason for the project to be maintained. (SLAs, Financial motivation, etc - Desire/Interest are feelings that can be placed elsewhere.)
- [BLESSING] It's a niche problem to solve.
- [CURSE] It's a niche problem to solve.
- ... and on and on
- oh yeah, AI.
The point I am trying to make by creating this section is to think about how you can contribute. I want some awareness to come to the project, in turn growing its popularity. I am not one to build out the code, or report bugs (I do want to get better at it). I am hoping my user experience and deployment can promote more activity. Maybe it could lead to popular requests being implemented as features; helping contributors stay engaged and interested in maintaining the project.
Anyways, I hope this post helps the users, maintainers and lurkers thinking about backing up their GOG games! Thanks for reading.
