Minecraft Infrastructure

Infrastructure Repostory for hosting a Minecraft Server at hetzner.de, used a combination of Terraform, Ansible and Vagrant.

Motivation

We play Minecraft since many years, on the same world, mostly on the latest Spigot Version. Sometimes with more players, othertimes with not so many. So we need a scalable environment with mininmal cost, at not used Time, lets go in the Cloud! ;).

Features

Install a ready to use Minecraft Server based on nolte/ansible-minecraft
Provide Different Stages
- a Reuseable Vagrant Box with build with Packer.
- a Local Test Environment, Vagrant based.
- a Production Environment, controlled with Terraform and Ansible, hosted at the hetzner.de/cloud
Handle different Type of Backup Solutions
- restic, for local or remote Backups to destinations like minio-server, amazon-s3 etc. (Useable for Restore Servers)
- archive as tar, and optional downloading or publish to GDrive.
Minimize the Environment, at not used time like the Summer, so minimize the costs to the storage part (~0,90€/Mon).
Combine different Ansible roles for os hardening, like (sshd, fail2ban, logrotate, restic) (nolte/ansible_playbook-baseline-online-server).

Usage

This Repository can be used to Provide your Server on two different Platforms Vagrant and Hetzner Cloud. For more information take al kook to the Documentation nolte.github.io/minecraft-infrastructure.

Local (Vagrant)

For the Local usage you need a runnable Vagrant Installation. The Local Vagrant part skip the Terraform installation tasks, like backup volumen handling etc. Only the Server Configuration Part provisioning/maintenance/master_playbook-configure-system.yml will executed, with the inventory provisioning/inventories/test.

Hetzner Cloud

As Cloud Provide for the Production we use hetzner.de, it exists a nice RestAPI, and a good Terraform Provider for configure the Infrastructure, i love it ;).

Advanced Informations

This Repository is part of the “Host your own Minecraft Server” Project, other parts of the project are nolte/minecraft-gameserver for public WebPresentation and nolte/ansible-minecraft a Ansible Role for install and configure the Minecraft Server.

The reuseable Base is extracted to nolte/ansible_playbook-baseline-online-server/ and nolte/terraform-infrastructure-modules.

Our Production Ansible Inventory is located add a private GitRepository, for protecting Player Informations.

License

Getting Started

Note

For all steps (Development,starting the Server and executing tests) the User must accept the Minecraft EULA, by own configured property!

by example setting a envierment property like export mc_accept_eula=true

For A Better Python Depenendency Mananagement it is recommended to configure a own Virtualenv.

Configure Python Virtual Env

 virtualenv -p python3 ~/venvs/ansible-vagrant
 source ~/venvs/ansible-vagrant/bin/activate
 pip install -r requirements.txt
 pre-commit install

Preconditions

For a full configuration you need some “Commons Scripts” from other Repositories, for example the nolte/ansible_playbook-baseline-online-server Project. The Dependencies will be managed by vendir and can be installed with the overlay command.

Install required Dependencies

 vendir sync

The Configuration file for the Dependencies is the vendir.yml.

---
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
  - path: provisioning/ext_debs
    contents:
      - path: ansible_playbook-baseline-online-server
        git:
          url: https://github.com/nolte/ansible_playbook-baseline-online-server.git
          ref: v1.0.4

Local Environment

For local Usage you can use the Vagrant file, located at the Maintenance Directory.

Start the Local Vagrant Machine

 cd local
 vagrant up

The Vagrant Box are configured with a minimal set of features, from the Test Inventory. After successfull starting you can join the server at localhost:25567

Note

The original gameport will mapped in the Vagrant file from 25565 to 25567.

Usefull Commands

vagrant ssh (SSH Connect to the Local Vagrant Box)
vagrant provision (Reexecute the Installation Steps)
vagrant global-status
vagrant ssh-config

Public Environment

The Public Env will be hosted at Hetzner Cloud. For Rollout the public environment, you must execute the Terraform and the Ansible sources.

Required Environment Variables

 export HCLOUD_TOKEN_STORAGE_PROJECT=$(pass internet/hetzner.com/projects/personal_storage/token) && \
   export HCLOUD_TOKEN=$(pass internet/hetzner.com/projects/minecraft/terraform-token) && \
   export AWS_ACCESS_KEY_ID=$(pass internet/project/mystoragebox/minio_access_key) && \
   export AWS_SECRET_ACCESS_KEY=$(pass internet/project/mystoragebox/minio_secret_key) && \
   export AWS_S3_ENDPOINT=https://$(curl -s -H "Authorization: Bearer $HCLOUD_TOKEN_STORAGE_PROJECT" 'https://api.hetzner.cloud/v1/servers?name=storagenode' | jq -r '.servers[0].public_net.ipv4.dns_ptr')

Infrastructure

For the Terraform State File, we use a Self Hosted Remote S3 Bucket, Personal Storage.

Long Term Elements

The Long Term Infrastructure Elements, are Ressources like SSH Key, or the Backup Volume. Keep this Part from the Infrastructure, for restore some backup costs ~1€/mon.

Warning

Be carefull with terraform destroy, you will be lost all your restic Backups!

The Ressources from provisioning/infrastructure/longterm will be handled in a seperated Terraform State File.

Computing Elements

The Computing Infrastructure Elements, are Ressources like Attatch Volume and Create the Server.

Note

Do terraform destroy, to save money (~5,83€/mon), or when the Server makes Problems! You can use the Last Backup from the Long Term Infrastructure Elements for restoring, look Restore on Create. So it makes fun to destroy the Ressources from provisioning/infrastructure/computing.

Configure the Server

For this we use the Ansible Playbooks from the Maintenance Directory and the dependencies from the Preconditions.

Install Ansible Galaxy Dependencies

ansible-galaxy install -r provisioning/ext_debs/ansible_playbook-baseline-online-server/requirements.yml && \
 ansible-galaxy install -r provisioning/maintenance/requirements.yml

When you use a external Inventory Folder, define the Environment Variable: export ANSIBLE_INVENTORY=$(pwd)/storagebox/prod/, now you can execute the Master Playbook:

Execute the full Installation

 ansible-playbook provisioning/maintenance/master-configure-system.yml

After the successfull configuration execute the “Testinfra Acception Tests”.

Maintenance

Testing

For a Quick feedback you can execute the Testinfra Acception Tests. Ensure that you have set the Required Environment Variables are configured, if requred.

“Testinfra Acception Tests”

  py.test --hosts='ansible://minecraftgameserver*' ext_debs/ansible_playbook-baseline-online-server/test/test_base_acception_test.py
  py.test --hosts='ansible://minecraftgameserver*' maintenance-test/*

Backub and Restore

We splitting the Backup in two different parts, the World Data Backups and the Plugins Data Backups. The Structure configuration is located at /provisioning/vars/facts_mc_node.yml.

Static Backup Config

---
backup_dirs:
  plugins:
    source:
      basedir: /opt/minecraft/plugins
      excludes:
        - shared/dynmap/*
    dest:
      local:
        restic_job_cron: '15 1  * * *'
        RESTIC_REPOSITORY: "{{ restic_repos_localbackup_url }}/plugindata"
        RESTIC_PASSWORD: "{{ mc_backup_restic_repository_password | default('dolphins') }}"
      b2:
        RESTIC_REPOSITORY: "b2:mcbackup-plugindata"
      s3:
        restic_job_cron: '40 2  * * *'
        RESTIC_S3_URL: "{{ restic_s3_endpoint | default('') }}"
        RESTIC_S3_BUCKET: backup
        RESTIC_S3_PATH: "/{{ backup_s3_subpath | default('') }}/restic/plugindata"
        RESTIC_PASSWORD: "{{ mc_backup_restic_repository_password | default('dolphins') }}"
        AWS_ACCESS_KEY_ID: "{{ s3_access_key | default('') }}"
        AWS_SECRET_ACCESS_KEY: "{{ s3_secret_key | default('') }}"
  worlddata:
    source:
      basedir: /opt/minecraft/server/shared
      excludes:
        - plugins/*
        - logs/*
        - crash-reports/*
    dest:
      local:
        restic_job_cron: '0 1  * * *'
        RESTIC_REPOSITORY: "{{ restic_repos_localbackup_url }}/gamedata"
        RESTIC_PASSWORD: "{{ mc_backup_restic_repository_password | default('dolphins') }}"
      b2:
        RESTIC_REPOSITORY: "b2:mcbackup-worlddata"
      s3:
        # restic_job_cron: '*/10 * * * *'
        restic_job_cron: '0 2  * * *'
        RESTIC_S3_URL: "{{ restic_s3_endpoint | default('') }}"
        RESTIC_S3_BUCKET: backup
        RESTIC_S3_PATH: "/{{ backup_s3_subpath | default('') }}/restic/gamedata"
        RESTIC_PASSWORD: "{{ mc_backup_restic_repository_password | default('dolphins') }}"
        AWS_ACCESS_KEY_ID: "{{ s3_access_key | default('') }}"
        AWS_SECRET_ACCESS_KEY: "{{ s3_secret_key | default('') }}"

You will find the implementation at a ugly but functional Ansible Role, located at provisioning/maintenance/roles/backup.

World Data Backups

Plugins Data Backups

Working with Backups

Daily Backups

#!/bin/bash

backup_excludes=$1
backup_source=$2
# exit with this by default, if it is not set later
exit_code=0
rcon_password=$(awk -F "=" '/rcon.password/ {print $2}' /opt/minecraft/server/shared/server.properties)
rcon_port=$(awk -F "=" '/rcon.port/ {print $2}' /opt/minecraft/server/shared/server.properties)

# the cleanup function will be the exit point
cleanup () {
  # ignore stderr from rm incase the hook is called twice
  /usr/local/bin/rcon-cli --port $rcon_port --password $rcon_password save-on
  /usr/local/bin/rcon-cli --port $rcon_port --password $rcon_password say "backup finished $backup_source"
  # exit(code)
  exit $exit_code
}

# register the cleanup function for all these signal types (see link below)
trap cleanup EXIT ERR INT TERM

/usr/local/bin/rcon-cli --port $rcon_port --password $rcon_password save-off
/usr/local/bin/rcon-cli --port $rcon_port --password $rcon_password say "backup starts $backup_source"

restic backup $backup_excludes $backup_source

# set the exit_code with the real result, used when cleanup is called
exit_code=$?

You can controll the the Jobs by define Variabels at the Inventory:

Configure Backup Jobs

 backup_dests:
   - "s3"
   - "local"
 backup_parts:
   - "plugins"
   - "worlddata"

Adhoc Backups

You can execute Adhoc Restic Backup when you destroying the Computing Elements, and will be safe that all gamedata saved! Or You can use the Adhoc Archiving Backup function to share your Server over GDrive, or a local archive. For executing be ensure that you have set the Required Environment Variables.

Adhoc Restic Backup

The Adhoc Restic Backup are usefull for store the current state before you change any configruations.

Start Restic Adhoc Backup

ansible-playbook maintenance/playbook-execute-backup.yml --extra-vars '{"backup_dests":["s3","local"]}' --extra-vars '{"backup_parts":["plugins","worlddata"]}'

when you Call the playbook playbook-execute-backup.yml without any --extra-vars we use the configuration from, Configure Backup Jobs.

Adhoc Archiving Backup

Local Archive

Start Restic Adhoc Backup

ansible-playbook maintenance/playbook-execute-backup.yml --extra-vars 'backup_type=archive' --extra-vars 'backup_adhoc_publish=local'

Gdrive Upload

For Upload to GDrive you must define your gdrive_refresh_token as Variable.

Restore on Create

At the moment you can only use restic (local or remote) Repositories for automatical restore.

Restore Configuration Attributes.

 minecraft_restore_src: "s3"
 minecraft_restore_elements:
   - "worlddata"
 #  - "plugins"

The Values must match with entries from the Static Backup Config.

Development

Tools

For Handle the Environment we use different OpenSource Tools, Terraform and Ansible for provisioning and configuration.

used projects
Projekte	Type	description
nolte/ansible-minecraft	Ansible Role	Controll the Server
nolte/ansible-minecraft-region-fixer	Ansible Role	für den Minecraft Region Fixer
itzg/rcon-cli	commandline tool	Go RCon Commandline Interface
dev-sec/ansible-os-hardening	Ansible Role	Harder the OS, set file permissions.
ypsman/ansible-sshd-config	Ansible Role	Harder the SSH Service, disable password login etc.
geerlingguy.java	Ansible Role	Used for install a Open JDK
robertdebock.bootstrap	Ansible Role
robertdebock.epel	Ansible Role	used for activate the Extra Packages for Enterprise Linux repository
robertdebock.fail2ban	Ansible Role	used for configure fail2ban
paulfantom.restic	Ansible Role	used for configure the restic backups
arillso.logrotate	Ansible Role	used for handle the logrotate configuration
passwordstore	commandline tool	pass is a Commandline Passwordstore
camptocamp/terraform-provider-pass	Terraform Provider	Used for pass secret lookups.
fboender/ansible-cmdb	Ansible Extention	Generate a Simple system Overview, see: ref-maintenance-process-cmdb

Directory Structure

This Git Repository is Structured in different folders, the Documentation Directory and the Provision Directory with the Infrastructure Providing Code.

Documentation Directory

The Sphinx Documentation is located at the docs subdirectory.

Local Directory

Working Directory for local Vagrant Box, more information at Local Environment.

Provision Directory

The Scripts for configure the ref-architecture-env-local-test and ref-architecture-env-public are structured under provisioning.

Infrastructure Directory

provisioning/infrastructure

Long Term Infrastructure Elements

provisioning/infrastructure/longterm

Computing Infrastructure Elements

provisioning/infrastructure/computing

Maintenance Directory

provisioning/maintenance

Maintenance Build Directory

provisioning/maintenance/build

Maintenance Test Directory

provisioning/maintenance-test

Inventories Directory

provisioning/inventories

Plugindata

provisioning/inventories/pluginconfigs

Test Inventory

The Inventory Structure under provisioning/inventories/test, will be used for creating the Vagrant box.

Test Infrastructure Server Config

---
mc_version: 1.18.1
jdk_used_version: 17
pluginset: "server"
restic_repos_localbackup_url: /tmp/localbackup
restic_active: true
mc_accept_eula: true
mc_plugins_file: "{{ inventory_dir }}/../pluginlist.yml"
plugin_config_dir: "{{ inventory_dir }}/../pluginconfigs"
mc_rcon_password: "password"
minecraft_port_gameserver: 25565
minecraft_max_memory: 1024M
minecraft_initial_memory: 1024M
mc_backup_restic_repository_password: dolphins
minecraft_motd: '\u00A7cAmazi\u00A76neg Mi\u00A7anecr\u00A7aft S\u00A79erv\u00A71er\n\u00A78(\u00A7nhttps\://nolte.github.io/minecraft-gameserver\u00A78\u00A7r)'

# Disable Sel linux for the moment, will be reactivate later ...
selinux_state: disabled

minecraft_ops:
  - nolte07
minecraft_whitelist:
  - nolte07

# minecraft_restore_src: "s3"
# minecraft_restore_elements:
#   - "worlddata"
#   - "plugins"

backup_dests:
  - "local"
backup_parts:
  - "plugins"
  - "worlddata"

Prod Inventory

Not a Part of this Repostiory, moved to an private GithHub Project.

Local Development

For the Local develop process we use a combination of Molecule and Vagrant it is recommendet to manage the dependencies in a own Virtualenv.

Configure a virtual env

 virtualenv -p python3 ~/venvs/develop-ansible_role-vagrant/
 source ~/venvs/develop-ansible_role-vagrant/bin/activate
 pip install requirementsDev.txt

Starting a local Vagrant Box

Molecule handle the Vagrant box and execute the Ansible playbooks.

Execute a full Tests Cycle

molecule test

for reexecuting the playbook use the converge command.

Starting a and configure a Vagrant Box

molecule converge

Package a new Vagrant Box

For creating new Vagrant box Versions we use Packer. Navigate to the Maintenance Build Directory (provisioning/maintenance/build) for packaging a new box.

build a reuseable box with packer

packer build minecraft_box.json

this will be need some minutes, but you got a reuseable and shareable Vagrant box, with a preconfigured ready to used Server.

Import the Box

 vagrant box add mc-gameserver-spigot file:///$(pwd)/output-vagrant/package.box --force

know when you have imported the new box Version, got to the Maintenance Directory (provisioning/maintenance) Directory and create the Virtual Machine.

Import the Box

 vagrant up

Ping the Box

export ANSIBLE_INVENTORY=$(pwd)/inventories/test/
ansible all -m ping

Naming Strategy

naming patterns
Naming	Example	description
`playbook-{title}.yml`	`playbook-execute-backup.yml`	Name of Classic Playbooks
`master_playbook-{title}.yml`	`master_playbook-mc-server.yml`	Name of Ansile Master Playbooks

Generate the Docs

For the Docs we use Sphinx with the reStructuredText. The simpleste way for generate, is using the tox Build framework.

Execute create Docs

 tox -e docs

glossary

Terraform: With Terraform we Create the Infrastructure like Volumes, FloatingIP and Virtual Machines. for the Hetzner Intergration wie use the ref-env-provider-hetzner-integration-terraform
Ansible: Ansible is used for System configuration.
restic: restic is a backup tool.
Vagrant: Vagrant, is used for the local Environment.
Minecraft-Region-Fixer: Fenixin/Minecraft-Region-Fixer
RCON: RCON RCON Protocol
logrotate: Remove old, and rotate the logs with logrotate.
fail2ban: Usig fail2ban for block brute force attacks.
Extra Packages for Enterprise Linux: The EPEL repository is used for install extra packages like restic.
Open JDK
pass: The Commandline based passwordstore, can integrated to Ansible and Terraform,
pass ansible plugin: Used for Secrets lookups passwordstore plugin
pass Terraform Provider: For combinate Terraform and pass we use the custom provider camptocamp/terraform-provider-pass.
Ansile Master Playbooks: importing-playbooks
Hetzner Cloud: Hetzner Cloud
firewall: hier wird der klassiker FirewallD verwendet.
Advanced Intrusion Detection Environment (aide): Store file see install-aide-centos-7. (umsetzung offen)
OpenSCAP: System vulnerability scans, see (open-scap)
backblaze.com: Cold Storage Provider, see ref-architecture-provider-backblaze
Sphinx: Sphinx, is a tool that makes it easy to create documentation
reStructuredText: reStructuredText Markdown alternative.
Molecule: Molecule used for automatical Ansible Tests.
Testinfra: Testinfra Testing infrastructure with Ansible and Pytest.
Virtualenv: Virtualenv create isolated Python environments.
Personal Storage: At Hetzner Cloud hosted MinIO S3 Object Storage (look nolte/personal-storage-infrastructure).
MinIO: min.io
gilt: gilt, a GIT layering tool (deprecated).
vendir: vendir, a GIT layering tool.
tox: tox, python automation project.