Welcome to stackd.io!¶
stackd.io is a modern cloud deployment and provisioning framework for everyone. Its purpose is to provide a common platform for deploying and configuring hardware on any cloud platform. We currently only support AWS EC2, but the driver framework is expandable to support other cloud providers.
stackd.io is a Django project, and uses Salt for its back-end configuration management.
Installation Overview¶
There are two main options for installation:
Amazon AMI¶
Reading through long install guides and executing each and every command can be time consuming and error prone. If you would rather just run a script to do a lot of this for you, we have a script to build an AMI for you. Keep in mind that the script is somewhat opinionated and won’t let you make many decisions (you’re free to modify it to suit your needs though!) Here’s a list of things it will do:
- Install all of the necessary stuff (MySQL, python, virtualenv, tons of packages, etc)
- Create a
stackdio
virtualenv at/usr/share/stackdio
- Install stackdio and its python dependencies
- Install and configure Nginx
- Install and configure supervisord to run gunicorn, celery, and salt-master
- Create an
admin
stackdio user
Check it out here: Using the Amazon AMI
Manual Install¶
If you’d rather have a more custom-fitted installation that fits your needs, check out Manual Install instead.
Manual Install¶
This guide is intended to quickly march you through the steps of installing and running stackd.io and its dependencies. We’re not intending to be complete or provide you with everything needed for a production-ready install, we may make some assumptions you don’t agree with, and there may be things we missed. If you feel anything is out of the ordinary, a bit confusing, or just plain missing, please contact us.
1. A database¶
stackd.io needs a relational database to store internal information. Since it’s built on Django, it inherently supports many different database servers. Mysql will do fine, but if you’d prefer something else, it is beyond the scope of this guide to install it. For more information on Django’s database support, see: https://docs.djangoproject.com/en/1.8/ref/databases/
The OS-specific prep of your choice (below) will walk you through installing mysql.
2. OS-specific preparation¶
Warning
You must follow the steps in one of the following prep guides for the OS you’re installing stackd.io in.
Follow one of the individual guides below to prepare your particular environment for stackd.io. Once you finish, come back here and continue on.
Preparing Ubuntu for stackd.io installation¶
The steps below were written using Ubuntu 13.10 from a Ubuntu-provided
AMI on Amazon Web Services (AWS). The exact AMI we used is
ami-2f252646
, and you should be able to easily launch an EC2
instance using this AMI from the
AWS EC2 Console.
Prerequisites¶
root
or sudo
access. Before installing anythingapt-get
you should run apt-get update
first.MySQL¶
Note
Please skip this section if you are using a different database or already have a supported database server running elsewhere.
Install MySQL server:
sudo apt-get install mysql-server mysql-client
# When prompted, provide a password for the root user to access the MySQL server.
Below we’ll create a stackdio
database and grant permissions to the
stackdio
user for that database.
WARNING: we’re not focusing on security here, so the default MySQL setup definitely needs to be tweaked, passwords changed, etc., but for a quick-start guide this is out of scope. Please, don’t run this as-is in production :)
echo "create database stackdio; \
grant all on stackdio.* to stackdio@'localhost' identified by 'password';" | \
mysql -hlocalhost -uroot -ppassword
virtualenvwrapper¶
# install the package
sudo apt-get install virtualenvwrapper
# post-install step for virtualenvwrapper shortcuts
source /etc/bash_completion.d/virtualenvwrapper
Core requirements¶
- gcc and other development tools
- git
- mysql-devel
- swig
- python-devel
- rabbitmq-server
To quickly get up and running, you can run the following to install the required packages.
# Install requirements needed to install stackd.io
sudo apt-get install python-dev libssl-dev libncurses5-dev libyaml-dev swig nodejs npm \
libmysqlclient-dev rabbitmq-server git nginx libldap2-dev libsasl2-dev
# Link nodejs over to node - bower will complain otherwise
sudo ln -s /usr/bin/nodejs /usr/bin/node
Next Steps¶
You’re now finished with the Ubuntu-specific requirements for stackd.io. You can head back over to the Manual Install and continue the installation of stackd.io.
Preparing CentOS for stackd.io installation¶
The steps below were written using CentOS 6.4 from a CentOS-provided AMI
on Amazon Web Services (AWS). The exact AMI we used is ami-bf5021d6
,
and you should be able to easily launch an EC2 instance using this AMI from the
AWS Marketplace.
Prerequisites¶
All of the CentOS-provided AMIs have SELinux and iptables enabled. We disabled both of these to be as straight forward as possible during this guide. SELinux causes issues that are beyond the scope of the guide, and we disabled iptables because we leverage EC2’s security groups for firewall access.
iptables¶
Let’s just turn it off for now. Please note, if you’re using EC2 or some other cloud provider that has firewall rules enabled by default, you will need to configure the particular firewall rules to gain access to the web server we’ll start in the guide. The default port for the webserver is 8000, so open this port up at a minimum. (Port 22 for SSH will obviously be needed as well.)
sudo service iptables stop
SELinux¶
Getting things working using SELinux could be an entirely separate guide. For our purposes, it’s completely out of scope, so we’re going to disable it.
Note
You will be required to restart the machine during this step.
# Edit /etc/sysconfig/selinux and make sure the line beginning
# with SELINUX looks like:
SELINUX=disabled
# If it was already disabled you can skip the following, however
# if you switched the policy from anything other than 'disabled'
# you need to relabel the filesystem to remove the garbage that
# SELinux has added. This *requires* a restart to take effect.
touch /.autorelabel
reboot
# When the machine is back up, you can confirm SELinux is not
# running
>>> selinuxenabled
>>> echo $?
>>> 1
# If the output is 1 you're good to go.
EPEL¶
sudo rpm -Uvh http://mirror.steadfast.net/epel/6/i386/epel-release-6-8.noarch.rpm
MySQL¶
Note
Please skip this section if you are using a different database or already have a supported database server running elsewhere.
Install MySQL server:
sudo yum install mysql-server
Start MySQL server:
sudo service mysqld start
Below we’ll create a stackdio
database and grant permissions to the
stackdio
user for that database.
Warning
We’re not focusing on security here, so the default MySQL setup definitely needs to be tweaked, passwords changed, etc., but for a quick-start guide this is out of scope. Please, don’t run this as-is in production :)
echo "create database stackdio; \
grant all on stackdio.* to stackdio@'localhost' identified by 'password';" | \
mysql -h localhost -u root
virtualenvwrapper¶
# install the package
sudo yum install python-virtualenvwrapper
# Update the user's ~/.bash_profile to enable virtualenvwrapper
# You're using the stackdio user, right? :)
echo "source /usr/bin/virtualenvwrapper.sh" >> ~/.bash_profile
# re-source the .bash_profile
. ~/.bash_profile
Core requirements¶
- gcc and other development tools
- git
- mysql-devel
- swig
- python-devel
- rabbitmq-server
- nginx
To quickly get up and running, you can run the following to install the required packages.
# Install the development tools group
sudo yum groupinstall "Development Tools"
# Install the other requirements needed to install stackd.io
sudo yum install git mysql-devel swig python-devel rabbitmq-server nginx nodejs npm
Next Steps¶
You’re now finished with the CentOS-specific requirements for stackd.io. You can head back over to the Manual Install and continue the installation of stackd.io.
3. Create a virtualenv¶
Let’s create a virtualenv to install stackd.io into:
mkvirtualenv stackdio
The virtualenv should automatically activate when you create it. If you
exit your current shell and come back later to work on stackdio and find
things not working as expected you probably need to activate the
virtualenv again. To do this, virtualenvwrapper gives you the workon
command:
workon stackdio
4. Install bower¶
In your terminal, run the following command to install bower:
Note
You must have previously installed npm/node from the OS specific preparation
sudo npm install -g bower
5. Install stackd.io¶
Note
Double-check that your virtualenv is activated or else this will probably complain that you don’t have permissions to install (because it’s trying to install into the global python site-packages directory which we don’t want!)
There’s two options for installing here. We recommend pulling the latest version from pypi with pip, like this:
workon stackdio # Activate the virtualenv
pip install stackdio-server[production,mysql]
6. Configuration¶
After the install, you’ll have a stackdio
command available to
interact with much of the platform. First off, we need to configure
stackd.io a bit. The stackdio init
command will prompt you for
several pieces of information. If you followed all steps above verbatim,
then all defaults may be accepted, but if you deviated from the path you
will need to provide the following information:
- an existing user on the system that will run everything (it will
default to the
stackdio
user) - an existing location where stackd.io can store its data (the default
is
$HOME/.stackdio/storage
and will be created for you if permissions allow) - a database DSN that points to a running database you have access to
(if you’re using the MySQL install from above, the default
mysql://stackdio:password@localhost:3306/stackdio
is appropriate)
stackdio init
Now, let’s populate are database with a schema:
stackdio manage.py migrate
7. stackd.io users¶
LDAP¶
stackd.io can easily integrate with an LDAP server. See our LDAP Guide for more information on configuring stackd.io to work with LDAP. If you choose to go the LDAP route, you can skip this entire section because users who successfully authenticate and are members of the right groups via LDAP will automatically be created in stackd.io.
Non-LDAP admin user¶
Admin users in stackd.io have less restriction to various pieces of the platform. For example, only admin users are allowed to create and modify cloud providers and profiles that other users can use to spin up their stacks.
Note
You will need at least one admin user to configure some key areas of the system.
stackdio manage.py createsuperuser
# and follow prompts...
Non-LDAP regular users¶
When not using LDAP, the easiest way to create new non-admin users is to use the built-in Django admin interface. First we need the server to be up and running so keep following the steps below and we’ll come back to adding users later.
8. Web server configuration¶
For this guide, we’ll use the stackdio
command to generate the
necessary configuration for Nginx to serve our static content as well as
proxying the Python app through gunicorn.
To configure Nginx for CentOS:
# CENTOS
# add execute permissions to the user's home directory for static content to serve correctly
chmod +x ~/
stackdio config nginx | sudo tee /etc/nginx/conf.d/stackdio.conf > /dev/null
# rename the default server configuration
sudo mv /etc/nginx/conf.d/default.conf /etc/nginx/conf.d/default.conf.bak
To configure Nginx for Ubuntu:
# UBUNTU ONLY
stackdio config nginx | sudo tee /etc/nginx/sites-available/stackdio > /dev/null
sudo ln -s /etc/nginx/sites-available/stackdio /etc/nginx/sites-enabled
# remove the default configuration symlink
sudo rm /etc/nginx/sites-enabled/default
After this, generate the static content we’ll need to serve:
stackdio manage.py collectstatic --noinput
and finally, start Nginx:
sudo service nginx restart
9. RabbitMQ, celery, and salt¶
Start the rabbitmq server:
sudo service rabbitmq-server start
For celery and salt-master, we’ll be using supervisord. The required packages should already be installed, so we’ll just need to configure supervisor and start the services.
# generate supervisord configuration that controls gunicorn, celery, and salt-master and store it in the .stackdio directory.
stackdio config supervisord > ~/.stackdio/supervisord.conf
# launch supervisord and start the services
supervisord -c ~/.stackdio/supervisord.conf
supervisorctl -c ~/.stackdio/supervisord.conf start all
10. Try it out!¶
At this point, you should have everything configured and running, so
fire up a web browser and point it to your hostname and you should see
the stackd.io login page. If you’re using LDAP, try logging in with a
user that is a member of the stackdio-admin
and stackdio-user
groups, or login with the admin user you created earlier.
11. Creating additional users¶
Note
If you’re using LDAP, you can skip this step.
The superuser we created earlier will give us admin access to stackd.io, however, you probably want at least one non-superuser. Point your browser to http://hostname:8000/__private/admin and use the username and password for the super user you created earlier. You should be presented with the Django admin interface. To create additional users, follow the steps below.
- click Users
- click Add user in the top right of the page
- set the username and password of the user and click save
- optionally provide first name, last name, and email address of the user and click save
The newly created users will now have access to stackd.io. Test this by logging out and signing in with one of the non-admin users.
Using the Amazon AMI¶
To make installation easier, we provide an packer build that creates an Amazon Machine Image (AMI). This AMI is built from an Ubuntu 14.04 LTS image, and is HVM based. While our ultimate plan is to provide this AMI on the AWS Marketplace, we don’t have this set up yet.
Building an AMI¶
Note
The build script that runs packer requires you to have python installed locally.
1. Install Packer¶
If you haven’t already, install packer using their documentation here. We recommend using homebrew for the installation if you’re using OSX.
2. Accept License¶
Before building with packer, you must accept the license agreement for the base Ubuntu AMI: http://aws.amazon.com/marketplace/pp?sku=b3dl4415quatdndl4qa6kcu45
3. Clone Repository¶
git clone https://github.com/stackdio/stackdio.git
cd stackdio
4. Export AWS Credentials¶
Ensure packer knows about your aws credentials:
export AWS_ACCESS_KEY='<YOUR_ACCESS_KEY>'
export AWS_SECRET_KEY='<YOUR_SECRET_KEY>'
5. Run the packer build¶
Finally, run the packer build, where <version>
is the version you want to build:
./packer/build.py <version>
After a few minutes, you should have a usable AMI.
Using the AMI¶
After you’ve built the AMI, you can launch an instance from it. Once the instance is running,
you can navigate to http://<instance-ip>/
and login using the following credentials:
username: admin
password: stackdio
We recommend changing this password immediately after logging in the first time.
LDAP Guide¶
django-auth-ldap¶
Under the hood, we use django-auth-ldap for all our interaction with an LDAP server. It’s a very useful library that allows us to integrate with LDAP in just a few lines of python configuration.
LDAP and stackd.io¶
If you’d like to integrate with LDAP, all you need to do is create an ldap_settings.py
file
in the stackdio.server.settings
package. If this file exists, stackd.io will automatically
pick it up and begin authenticating users to your ldap server. Inside that settings package,
there is an ldap_settings.py.template
file that you may rename to ldap_settings.py
and update to match your needs. It’s contents are displayed below.
The template contains most of the relevant bits you may want to use, but we defer anything deeper to the django-auth-ldap documentation.
Note
If you’re using the packer-built AMI described in Using the Amazon AMI, your ldap settings should be placed here:
/usr/share/stackdio/lib/python2.7/site-packages/stackdio/server/settings/ldap_settings.py
##
# LDAP configuration
# We are using django-auth-ldap to enable stackd.io to use LDAP for
# authentication. The settings below correspond to our internal
# LDAP and we don't guarantee this to work for all. Please see
# the docs at http://pythonhosted.org/django-auth-ldap/ for more
# information.
##
import ldap
from django_auth_ldap.config import LDAPSearch, GroupOfNamesType
# We use direct binding with a dedicated user. If you have anonymous
# access available or can bind with any user, you'll want to change
# this.
AUTH_LDAP_BIND_AS_AUTHENTICATING_USER = False
AUTH_LDAP_SERVER_URI = 'YOUR_LDAP_SERVER_URI'
AUTH_LDAP_BIND_DN = 'uid=YOUR_BIND_USER,ou=People,dc=yourcompany,dc=com'
AUTH_LDAP_BIND_PASSWORD = 'YOUR_BIND_USER_PASSWORD'
AUTH_LDAP_REQUIRE_GROUP = ('cn=stackdio-user,ou=Group,dc=yourcompany,dc=com')
AUTH_LDAP_USER_SEARCH = LDAPSearch('ou=People,dc=yourcompany,dc=com',
ldap.SCOPE_SUBTREE,
'(&(objectClass=Person)(uid=%(user)s))')
# Group handling. Read the django_auth_ldap documentation for more info.
AUTH_LDAP_FIND_GROUP_PERMS = False
AUTH_LDAP_MIRROR_GROUPS = False
AUTH_LDAP_GROUP_TYPE = GroupOfNamesType()
AUTH_LDAP_GROUP_SEARCH = LDAPSearch(
'ou=Group,dc=yourcompany,dc=com',
ldap.SCOPE_SUBTREE,
'(objectClass=groupOfNames)'
)
AUTH_LDAP_USER_ATTR_MAP = {
'first_name': 'givenName',
'last_name': 'sn',
'email': 'mail',
}
AUTH_LDAP_USER_FLAGS_BY_GROUP = {
'is_superuser': 'cn=stackdio-admin,ou=Group,dc=yourcompany,dc=com',
'is_staff': 'cn=stackdio-admin,ou=Group,dc=yourcompany,dc=com',
}
AUTH_LDAP_CONNECTION_OPTIONS = {
ldap.OPT_X_TLS_REQUIRE_CERT: ldap.OPT_X_TLS_NEVER,
ldap.OPT_X_TLS_NEWCTX: 0,
}
Webserver Guide¶
This guide will help you quickly get the web portion of stackd.io running behind either Apache or Nginx. You should’ve already worked through this manual install guide before running through the steps below. As with this guide, our focus is not entirely on building out a production-ready system, but merely helping you quickly get a system stood up to become familiar with stackd.io. Once you understand how it works, then we can start hardening the system for production use.
So, with that said, there are two paths to take: Apache or Nginx. We recommend using whichever you feel more comfortable with. Some of us here like Apache, while others like Nginx. Your mileage may vary :)
Common Steps¶
To do some of the steps below you will need to have already installed stackdio and be in the virtual environment. To make sure you’re in the virtualenv:
workon stackdio
Both Apache and Nginx installs need a place to store logs and some static files to serve up. These two steps are common to both and should be run before proceeding with configuring the web server.
# Create the log directory that we'll have Apache or Nginx log to
mkdir -p ~/.stackdio/var/log/web
# And tell Django to collect its static files into a common directory for the webserver to serve up
stackdio manage.py collectstatic --noinput
Apache¶
CentOS Installation¶
Install required packages:
sudo yum install httpd mod_wsgi mod_ssl
Followed by having stackd.io generate a simple Apache configuration file for serving up the Django-based API and static assets and store the output into the appropriate location.
stackdio config apache | sudo tee /etc/httpd/conf.d/stackdio.conf > /dev/null
Fix a permissions problem with the user’s home directory not having execute permissions. This is needed because of httpd v2.2 needing directory execute permissions from the web directory up to the root directory.
chmod +x ~/
Note
You may pass –with-ssl to generate boilerplate for serving over SSL, but you will need to add your certs and point to them in the configuration file. You may also need to remove the existing ssl.conf from within conf.d.
And that’s it...let’s start the server and then point your browser to the hostname on port 80 (use https if you decided to serve over SSL.)
sudo service httpd restart
Ubuntu Installation¶
Install required packages:
sudo apt-get install apache2 libapache2-mod-wsgi
and just like the CentOS instructions, generate and store the Apache configuration file into the correct location:
stackdio config apache | sudo tee /etc/apache2/sites-enabled/stackdio.conf > /dev/null
Note
You may pass –with-ssl to generate boilerplate for serving over SSL, but you will need to add your certs and point to them in the configuration file.
and finally, start the server:
sudo service apache2 restart
Nginx¶
In our configuration, Nginx will be used to serve static files and as a proxy to send requests down to the Django application running via gunicorn on port 8000. The configuration we’ll generate is useful to use a quick start mechanism to get you up and running behind Nginx/gunicorn very quickly.
CentOS Installation¶
Install required packaged, generate and write configuration file, and restart server:
sudo yum install nginx
stackdio config nginx | sudo tee /etc/nginx/conf.d/stackdio.conf > /dev/null
# rename the default server configuration
sudo mv /etc/nginx/conf.d/default.conf /etc/nginx/conf.d/default.conf.bak
sudo service nginx restart
Ubuntu Installation¶
sudo apt-get install nginx
stackdio config nginx | sudo tee /etc/nginx/sites-enabled/stackdio.conf > /dev/null
# remove the default configuration symlink
sudo rm /etc/nginx/sites-enabled/default
sudo service nginx restart
Contact Information¶
- Static site - http://stackd.io
- IRC - #stackdio on Freenode
- Twitter - @stackdio or #stackdio
- Github - https://github.com/stackdio/stackdio
Filing issues and feature requests¶
We want stackd.io to be solid, but there’s always going to be issues to fix and new features to enhance the product. If you find any problems or would like to request a particular feature, feel free to head over to stackd.io’s issue tracker. Below are some general guidelines.
Feature Requests¶
We definitely want this project to be better in every way, but sometimes we just can’t see the forest because of those trees. If you think of something that would make stackd.io be more intuitive, easier, faster, or efficient, we encourage you to file a feature request, but please keep the following in mind:
- Double-check that the feature hasn’t already made its way into the tracker and/or been completed
- After filing it, please be patient – sometimes it might take a while for things to get prioritized, so don’t be bummed that it’s not available tomorrow ;)
- As always, the quickest way to get your shiny new feature is to get your hands dirty and implement it yourself!
Bug Reports¶
If you think you’ve found a bug, here’s what we ask of you:
- Check the issue tracker and/or search for your issue to make sure we aren’t already tracking the problem
- See if you can reproduce the issue on the develop branch of the project to confirm we haven’t already fixed the problem
- Try to be as descriptive as possible when filing your issues. We’d like to know the operating system you’re on, any stack traces you’ve seen in the logs, a good account of the steps required to reproduce the problem along with what you feel is the expected outcome. Providing more information is almost always better so we can quickly identify and fix the issue.
- Be patient :)
- Optionally, it would be fantastic if you could help us out and fix the problem yourself...if you’re brave enough, see the Contributor Guidelines on contributing to the project.
Usage Questions¶
Usage questions are best handled in one of the following ways:
- Asking a question on the IRC channel
- Emailing the contributors, especially those who have been recently active on the project.
Long term development comments¶
Long term development tasks will usually have a branch opened on the main repository, along with a pull request back into develop. This is the place to make comments on the development cycle for that feature.
Contributor Guidelines¶
We’re hopeful that you’ll find value in using stackd.io along with your existing tools to manage your infrastructure. We’re also hopeful that you’ll find ways to help contribute. Either through finding and reporting bugs, providing new features, or even getting your hands dirty and contributing some code or docs. However you feel comfortable contributing, we offer a few helpful guidelines to make it that much easier.
Note that stackd.io is built on SaltStack, and we believe that its community will find stackd.io useful. As such, we’re trying to stay close to the conventions and guidelines they’ve adopted to make it easier for folks in that community to help out – and we’ve borrowed from the SaltStack Development guide :)
Filing issues and feature requests¶
The process for filing issues and feature requests is described in the Contact Information page.
Contributing Code¶
Since we’re using Github, the recommended workflow for fixing bugs, adding features, or modifying documentation is to fork and submit pull requests. The process is pretty straightforward, but if you’re unfamiliar with Github, take some time to browse through Github’s Help.
In a nutshell, we’ll need you to:
fork the stackd.io project into your personal account [Tutorial]
make the necessary changes to the code/docs and issue a pull request. [Tutorial]
keep your local fork in sync with the parent stackd.io repository to minimize the chance of merge conflicts. [Tutorial]
and, if you’re working on multiple things or your changes are going to be somewhat large, it’s generally recommended to create a branch for each piece of work you’re doing. [Tutorial]
NOTE: SaltStack has a great guide on how to work within their project and it mostly applies to stackd.io as well
Pull request guidelines¶
CLA¶
Contribution to stackd.io requires a CLA before pull requests will be merged. This is currently handled manually by the repo admins, but may be handled by a bot in the future.
Branch naming¶
The branch name that the pull request originates from should start with
either feature/
or bugfix/
, depending on its contents. The rest
of the branch name should describe the contents of the patch, preferably
by being an Issue#. Issue#’s are required for bugfix/
branches.
Code style and quality¶
PEP8 compatibility¶
All pull requests must meet PEP8 compatibility
Tests¶
Pull requests will be easier to review and understand if they contain automated tests for the functionality changed. As such, pull requests with tests are more likely to be accepted more quickly.