Minecraft Infrastructure
Infrastructure Repostory for hosting a Minecraft Server at hetzner.de, used a combination of Terraform, Ansible and Vagrant.
Motivation
We play Minecraft since many years, on the same world, mostly on the latest Spigot Version. Sometimes with more players, othertimes with not so many. So we need a scalable environment with mininmal cost, at not used Time, lets go in the Cloud! ;).
Features
Install a ready to use Minecraft Server based on nolte/ansible-minecraft
Provide Different Stages
a Reuseable Vagrant Box with build with Packer.
a Local Test Environment, Vagrant based.
a Production Environment, controlled with Terraform and Ansible, hosted at the hetzner.de/cloud
Handle different Type of Backup Solutions
restic, for local or remote Backups to destinations like minio-server, amazon-s3 etc. (Useable for Restore Servers)
archive as tar, and optional downloading or publish to GDrive.
Minimize the Environment, at not used time like the Summer, so minimize the costs to the storage part (~0,90€/Mon).
Combine different Ansible roles for os hardening, like (sshd, fail2ban, logrotate, restic) (nolte/ansible_playbook-baseline-online-server).
Usage
This Repository can be used to Provide your Server on two different Platforms Vagrant and Hetzner Cloud. For more information take al kook to the Documentation nolte.github.io/minecraft-infrastructure.
Local (Vagrant)
For the Local usage you need a runnable Vagrant Installation.
The Local Vagrant part skip the Terraform installation tasks, like backup volumen handling etc. Only the Server Configuration Part provisioning/maintenance/master_playbook-configure-system.yml
will executed, with the inventory provisioning/inventories/test
.
Hetzner Cloud
As Cloud Provide for the Production we use hetzner.de, it exists a nice RestAPI, and a good Terraform Provider for configure the Infrastructure, i love it ;).
Advanced Informations
This Repository is part of the “Host your own Minecraft Server” Project, other parts of the project are nolte/minecraft-gameserver for public WebPresentation and nolte/ansible-minecraft a Ansible Role for install and configure the Minecraft Server.
The reuseable Base is extracted to nolte/ansible_playbook-baseline-online-server/ and nolte/terraform-infrastructure-modules.
Our Production Ansible Inventory is located add a private GitRepository, for protecting Player Informations.
License
Getting Started
Note
export mc_accept_eula=true
For A Better Python Depenendency Mananagement it is recommended to configure a own Virtualenv.
virtualenv -p python3 ~/venvs/ansible-vagrant
source ~/venvs/ansible-vagrant/bin/activate
pip install -r requirements.txt
pre-commit install
Preconditions
For a full configuration you need some “Commons Scripts” from other Repositories, for example the nolte/ansible_playbook-baseline-online-server Project.
The Dependencies will be managed by vendir and can be installed with the overlay
command.
vendir sync
The Configuration file for the Dependencies is the vendir.yml
.
---
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: provisioning/ext_debs
contents:
- path: ansible_playbook-baseline-online-server
git:
url: https://github.com/nolte/ansible_playbook-baseline-online-server.git
ref: v1.0.4
Local Environment
For local Usage you can use the Vagrant
file, located at the Maintenance Directory.
cd local
vagrant up
The Vagrant Box are configured with a minimal set of features, from the Test Inventory.
After successfull starting you can join the server at localhost:25567
Note
The original gameport will mapped in the Vagrant file from 25565
to 25567
.
Usefull Commands
vagrant ssh
(SSH Connect to the Local Vagrant Box)vagrant provision
(Reexecute the Installation Steps)vagrant global-status
vagrant ssh-config
Public Environment
The Public Env will be hosted at Hetzner Cloud. For Rollout the public environment, you must execute the Terraform and the Ansible sources.
export HCLOUD_TOKEN_STORAGE_PROJECT=$(pass internet/hetzner.com/projects/personal_storage/token) && \
export HCLOUD_TOKEN=$(pass internet/hetzner.com/projects/minecraft/terraform-token) && \
export AWS_ACCESS_KEY_ID=$(pass internet/project/mystoragebox/minio_access_key) && \
export AWS_SECRET_ACCESS_KEY=$(pass internet/project/mystoragebox/minio_secret_key) && \
export AWS_S3_ENDPOINT=https://$(curl -s -H "Authorization: Bearer $HCLOUD_TOKEN_STORAGE_PROJECT" 'https://api.hetzner.cloud/v1/servers?name=storagenode' | jq -r '.servers[0].public_net.ipv4.dns_ptr')
Infrastructure
For the Terraform State File, we use a Self Hosted Remote S3 Bucket, Personal Storage.
Long Term Elements
The Long Term Infrastructure Elements, are Ressources like SSH Key, or the Backup Volume. Keep this Part from the Infrastructure, for restore some backup costs ~1€/mon.
Warning
Be carefull with terraform destroy
, you will be lost all your restic Backups!
The Ressources from provisioning/infrastructure/longterm
will be handled in a seperated Terraform State File.
Computing Elements
The Computing Infrastructure Elements, are Ressources like Attatch Volume and Create the Server.
Note
Do terraform destroy
, to save money (~5,83€/mon), or when the Server makes Problems! You can use the Last Backup from the Long Term Infrastructure Elements for restoring, look Restore on Create.
So it makes fun to destroy the Ressources from provisioning/infrastructure/computing
.
Configure the Server
For this we use the Ansible Playbooks from the Maintenance Directory and the dependencies from the Preconditions.
ansible-galaxy install -r provisioning/ext_debs/ansible_playbook-baseline-online-server/requirements.yml && \
ansible-galaxy install -r provisioning/maintenance/requirements.yml
When you use a external Inventory Folder, define the Environment Variable: export ANSIBLE_INVENTORY=$(pwd)/storagebox/prod/
, now you can execute the Master Playbook:
ansible-playbook provisioning/maintenance/master-configure-system.yml
After the successfull configuration execute the “Testinfra Acception Tests”.
Maintenance
Testing
For a Quick feedback you can execute the Testinfra Acception Tests. Ensure that you have set the Required Environment Variables are configured, if requred.
py.test --hosts='ansible://minecraftgameserver*' ext_debs/ansible_playbook-baseline-online-server/test/test_base_acception_test.py
py.test --hosts='ansible://minecraftgameserver*' maintenance-test/*
Backub and Restore
We splitting the Backup in two different parts, the World Data Backups and the Plugins Data Backups. The Structure configuration is located at /provisioning/vars/facts_mc_node.yml
.
---
backup_dirs:
plugins:
source:
basedir: /opt/minecraft/plugins
excludes:
- shared/dynmap/*
dest:
local:
restic_job_cron: '15 1 * * *'
RESTIC_REPOSITORY: "{{ restic_repos_localbackup_url }}/plugindata"
RESTIC_PASSWORD: "{{ mc_backup_restic_repository_password | default('dolphins') }}"
b2:
RESTIC_REPOSITORY: "b2:mcbackup-plugindata"
s3:
restic_job_cron: '40 2 * * *'
RESTIC_S3_URL: "{{ restic_s3_endpoint | default('') }}"
RESTIC_S3_BUCKET: backup
RESTIC_S3_PATH: "/{{ backup_s3_subpath | default('') }}/restic/plugindata"
RESTIC_PASSWORD: "{{ mc_backup_restic_repository_password | default('dolphins') }}"
AWS_ACCESS_KEY_ID: "{{ s3_access_key | default('') }}"
AWS_SECRET_ACCESS_KEY: "{{ s3_secret_key | default('') }}"
worlddata:
source:
basedir: /opt/minecraft/server/shared
excludes:
- plugins/*
- logs/*
- crash-reports/*
dest:
local:
restic_job_cron: '0 1 * * *'
RESTIC_REPOSITORY: "{{ restic_repos_localbackup_url }}/gamedata"
RESTIC_PASSWORD: "{{ mc_backup_restic_repository_password | default('dolphins') }}"
b2:
RESTIC_REPOSITORY: "b2:mcbackup-worlddata"
s3:
# restic_job_cron: '*/10 * * * *'
restic_job_cron: '0 2 * * *'
RESTIC_S3_URL: "{{ restic_s3_endpoint | default('') }}"
RESTIC_S3_BUCKET: backup
RESTIC_S3_PATH: "/{{ backup_s3_subpath | default('') }}/restic/gamedata"
RESTIC_PASSWORD: "{{ mc_backup_restic_repository_password | default('dolphins') }}"
AWS_ACCESS_KEY_ID: "{{ s3_access_key | default('') }}"
AWS_SECRET_ACCESS_KEY: "{{ s3_secret_key | default('') }}"
You will find the implementation at a ugly but functional Ansible Role, located at provisioning/maintenance/roles/backup
.
World Data Backups
Plugins Data Backups
Working with Backups
Daily Backups
#!/bin/bash
backup_excludes=$1
backup_source=$2
# exit with this by default, if it is not set later
exit_code=0
rcon_password=$(awk -F "=" '/rcon.password/ {print $2}' /opt/minecraft/server/shared/server.properties)
rcon_port=$(awk -F "=" '/rcon.port/ {print $2}' /opt/minecraft/server/shared/server.properties)
# the cleanup function will be the exit point
cleanup () {
# ignore stderr from rm incase the hook is called twice
/usr/local/bin/rcon-cli --port $rcon_port --password $rcon_password save-on
/usr/local/bin/rcon-cli --port $rcon_port --password $rcon_password say "backup finished $backup_source"
# exit(code)
exit $exit_code
}
# register the cleanup function for all these signal types (see link below)
trap cleanup EXIT ERR INT TERM
/usr/local/bin/rcon-cli --port $rcon_port --password $rcon_password save-off
/usr/local/bin/rcon-cli --port $rcon_port --password $rcon_password say "backup starts $backup_source"
restic backup $backup_excludes $backup_source
# set the exit_code with the real result, used when cleanup is called
exit_code=$?
You can controll the the Jobs by define Variabels at the Inventory:
backup_dests:
- "s3"
- "local"
backup_parts:
- "plugins"
- "worlddata"
Adhoc Backups
You can execute Adhoc Restic Backup when you destroying the Computing Elements, and will be safe that all gamedata saved! Or You can use the Adhoc Archiving Backup function to share your Server over GDrive, or a local archive. For executing be ensure that you have set the Required Environment Variables.
The Adhoc Restic Backup are usefull for store the current state before you change any configruations.
ansible-playbook maintenance/playbook-execute-backup.yml --extra-vars '{"backup_dests":["s3","local"]}' --extra-vars '{"backup_parts":["plugins","worlddata"]}'
when you Call the playbook playbook-execute-backup.yml
without any --extra-vars
we use the configuration from, Configure Backup Jobs.
ansible-playbook maintenance/playbook-execute-backup.yml --extra-vars 'backup_type=archive' --extra-vars 'backup_adhoc_publish=local'
For Upload to GDrive you must define your gdrive_refresh_token
as Variable.
Restore on Create
At the moment you can only use restic (local or remote) Repositories for automatical restore.
minecraft_restore_src: "s3" minecraft_restore_elements: - "worlddata" # - "plugins"
The Values must match with entries from the Static Backup Config.
Development
Tools
For Handle the Environment we use different OpenSource Tools, Terraform and Ansible for provisioning and configuration.
Projekte |
Type |
description |
---|---|---|
Ansible Role |
Controll the Server |
|
Ansible Role |
für den Minecraft Region Fixer |
|
commandline tool |
Go RCon Commandline Interface |
|
Ansible Role |
Harder the OS, set file permissions. |
|
Ansible Role |
Harder the SSH Service, disable password login etc. |
|
Ansible Role |
Used for install a Open JDK |
|
Ansible Role |
||
Ansible Role |
used for activate the Extra Packages for Enterprise Linux repository |
|
Ansible Role |
used for configure fail2ban |
|
Ansible Role |
used for configure the restic backups |
|
Ansible Role |
used for handle the logrotate configuration |
|
commandline tool |
pass is a Commandline Passwordstore |
|
Terraform Provider |
Used for pass secret lookups. |
|
Ansible Extention |
Generate a Simple system Overview, see: ref-maintenance-process-cmdb |
Directory Structure
This Git Repository is Structured in different folders, the Documentation Directory and the Provision Directory with the Infrastructure Providing Code.
Documentation Directory
The Sphinx Documentation is located at the docs
subdirectory.
Local Directory
Working Directory for local Vagrant Box, more information at Local Environment.
Provision Directory
The Scripts for configure the ref-architecture-env-local-test and ref-architecture-env-public are structured under provisioning
.
Infrastructure Directory
provisioning/infrastructure
provisioning/infrastructure/longterm
provisioning/infrastructure/computing
Maintenance Directory
provisioning/maintenance
provisioning/maintenance/build
Maintenance Test Directory
provisioning/maintenance-test
Inventories Directory
provisioning/inventories
provisioning/inventories/pluginconfigs
The Inventory Structure under provisioning/inventories/test
, will be used for creating the Vagrant box.
---
mc_version: 1.18.1
jdk_used_version: 17
pluginset: "server"
restic_repos_localbackup_url: /tmp/localbackup
restic_active: true
mc_accept_eula: true
mc_plugins_file: "{{ inventory_dir }}/../pluginlist.yml"
plugin_config_dir: "{{ inventory_dir }}/../pluginconfigs"
mc_rcon_password: "password"
minecraft_port_gameserver: 25565
minecraft_max_memory: 1024M
minecraft_initial_memory: 1024M
mc_backup_restic_repository_password: dolphins
minecraft_motd: '\u00A7cAmazi\u00A76neg Mi\u00A7anecr\u00A7aft S\u00A79erv\u00A71er\n\u00A78(\u00A7nhttps\://nolte.github.io/minecraft-gameserver\u00A78\u00A7r)'
# Disable Sel linux for the moment, will be reactivate later ...
selinux_state: disabled
minecraft_ops:
- nolte07
minecraft_whitelist:
- nolte07
# minecraft_restore_src: "s3"
# minecraft_restore_elements:
# - "worlddata"
# - "plugins"
backup_dests:
- "local"
backup_parts:
- "plugins"
- "worlddata"
Not a Part of this Repostiory, moved to an private GithHub Project.
Local Development
For the Local develop process we use a combination of Molecule and Vagrant it is recommendet to manage the dependencies in a own Virtualenv.
virtualenv -p python3 ~/venvs/develop-ansible_role-vagrant/
source ~/venvs/develop-ansible_role-vagrant/bin/activate
pip install requirementsDev.txt
Starting a local Vagrant Box
Molecule handle the Vagrant box and execute the Ansible playbooks.
molecule test
for reexecuting the playbook use the converge
command.
molecule converge
Package a new Vagrant Box
For creating new Vagrant box Versions we use Packer.
Navigate to the Maintenance Build Directory (provisioning/maintenance/build
) for packaging a new box.
packer build minecraft_box.json
this will be need some minutes, but you got a reuseable and shareable Vagrant box, with a preconfigured ready to used Server.
vagrant box add mc-gameserver-spigot file:///$(pwd)/output-vagrant/package.box --force
know when you have imported the new box Version, got to the Maintenance Directory (provisioning/maintenance
) Directory and create the Virtual Machine.
vagrant up
export ANSIBLE_INVENTORY=$(pwd)/inventories/test/
ansible all -m ping
Naming Strategy
Naming |
Example |
description |
---|---|---|
|
|
Name of Classic Playbooks |
|
|
Name of Ansile Master Playbooks |
Generate the Docs
For the Docs we use Sphinx with the reStructuredText. The simpleste way for generate, is using the tox Build framework.
tox -e docs
glossary
- Terraform
With Terraform we Create the Infrastructure like Volumes, FloatingIP and Virtual Machines. for the Hetzner Intergration wie use the ref-env-provider-hetzner-integration-terraform
- Ansible
Ansible is used for System configuration.
- restic
restic is a backup tool.
- Vagrant
Vagrant, is used for the local Environment.
- Minecraft-Region-Fixer
- RCON
- logrotate
Remove old, and rotate the logs with logrotate.
- fail2ban
Usig fail2ban for block brute force attacks.
- Extra Packages for Enterprise Linux
The EPEL repository is used for install extra packages like restic.
- Open JDK
- pass
The Commandline based passwordstore, can integrated to Ansible and Terraform,
- pass ansible plugin
Used for Secrets lookups passwordstore plugin
- pass Terraform Provider
For combinate Terraform and pass we use the custom provider camptocamp/terraform-provider-pass.
- Ansile Master Playbooks
- Hetzner Cloud
- firewall
hier wird der klassiker FirewallD verwendet.
- Advanced Intrusion Detection Environment (aide)
Store file see install-aide-centos-7. (umsetzung offen)
- OpenSCAP
System vulnerability scans, see (open-scap)
- backblaze.com
Cold Storage Provider, see ref-architecture-provider-backblaze
- Sphinx
Sphinx, is a tool that makes it easy to create documentation
- reStructuredText
reStructuredText Markdown alternative.
- Molecule
Molecule used for automatical Ansible Tests.
- Testinfra
Testinfra Testing infrastructure with Ansible and Pytest.
- Virtualenv
Virtualenv create isolated Python environments.
- Personal Storage
At Hetzner Cloud hosted MinIO S3 Object Storage (look nolte/personal-storage-infrastructure).
- MinIO
- gilt
gilt, a GIT layering tool (deprecated).
- vendir
vendir, a GIT layering tool.
- tox
tox, python automation project.