Welcome to Clearwater¶

Bono (Edge Proxy)¶

The Bono nodes form a horizontally scalable SIP edge proxy providing both a SIP IMS Gm compliant interface and a WebRTC interface to clients. Client connections are load balanced across the nodes. The Bono node provides the anchor point for the client’s connection to the Clearwater system, including support for various NAT traversal mechanisms. A client is therefore anchored to a particular Bono node for the duration of its registration, but can move to another Bono node if the connection or client fails.

Sprout (SIP Router)¶

The Sprout nodes act as a horizontally scalable, combined SIP registrar and authoritative routing proxy, and handle client authentication and the ISC interface to application servers. The Sprout nodes also contain the in-built MMTEL application server. The Sprout cluster includes a memcached cluster storing client registration data. SIP transactions are load balanced across the Sprout cluster, so there is no long-lived association between a client and a particular Sprout node. Sprout uses web services interfaces provided by Homestead and Homer to retrieve HSS configuration such as authentication data/user profiles and MMTEL service settings.

Homestead (HSS Cache)¶

Homestead provides a web services interface to Sprout for retrieving authentication credentials and user profile information. It can either master the data (in which case it exposes a web services provisioning interface) or can pull the data from an IMS compliant HSS over the Cx interface. The Homestead nodes run as a cluster using Cassandra as the store for mastered/cached data.

Homer (XDMS)¶

Homer is a standard XDMS used to store MMTEL service settings documents for each user of the system. Documents are be created, read, updated and deleted using a standard XCAP interface. As with Homestead, the Homer nodes runs as a cluster using Cassandra as the data store.

Ralf (CTF)¶

Ralf provides an HTTP API that both Bono and Sprout can use to report billable events that should be passed to the CDF (Charging Data Function) over the Rf billing interface. Ralf uses a cluster of Memcached instances and a cluster of Chronos instances to store and manage session state, allowing it to conform to the Rf protocol.

Ellis¶

Ellis is a sample provisioning portal providing self sign-up, password management, line management and control of MMTEL service settings. It is not intended to be a part of production Clearwater deployments (it is not easy to horizontally scale because of the MySQL underpinnings for one thing) but to make the system easy to use out of the box.

Load Balancing¶

In a cloud scalable system like Clearwater load balancing is an important part of making the system horizontally scale in a robust way. Clearwater uses a variation on DNS load balancing to ensure even loading when clusters are being elastically resized to adapt to changes in total load.

As an example, a single domain name is configured for all the Sprout nodes. Each Bono node maintains a pool of SIP connections to the Sprout nodes, with the target node for each connection selected at random from the list of addresses returned by DNS. Bono selects a connection at random for each SIP transaction forwarded to Sprout. The connections in the pool are recycled on failure and periodically, selecting a different address from the list returned by the DNS server each time.

A similar technique is used for the HTTP connections between Sprout and Homer/Homestead - each Sprout maintains a pool of connections load balanced across the Homer/Homestead clusters and periodically forces these connections to be recycled.

Reliability and Redundancy¶

Traditional telco products achieve reliability using low-level data replication, often in a one-to-one design. This is both complex and costly - and does not adapt well to a virtualized/cloud environment.

The Clearwater approach to reliability is to follow common design patterns for scalable web services - keeping most components largely stateless and storing long-lived state in specially design reliable, scalable clustered data stores.

Both Bono and Sprout operate as transaction-stateful rather than dialog-stateful proxies - transaction state is typically short-lived compared to dialog state. As the anchor point for client connections for NAT traversal, the Bono node used remains on the signaling path for the duration of a SIP dialog. Any individual Sprout node is only in the signaling path for the initial transaction, and subsequent requests are routed through the entire Sprout cluster, so failure of a Sprout node does not cause established SIP dialogs to fail. Long-lived SIP state such as registration data and event subscription state is stored in a clustered, redundant shared data store (memcached) so is not tied to any individual Sprout node.

Homer and Homestead similar only retain local state for pending requests - all long lived state is stored redundantly in the associated Cassandra cluster.

Cloud Security¶

SIP communications are divided into a trusted zone (for flows between Sprout nodes, Bono nodes and trusted application servers) and an untrusted zone (for message flows between Bono nodes and external clients or other systems). These zones use different ports allowing the trusted zone to be isolated using security groups and/or firewall rules, while standard SIP authentication mechanisms are used to protect the untrusted ports.

Other interfaces such as the XCAP and Homestead interfaces use a combination of locked down ports, standard authentication schemes and shared secret API keys for security.

Self-service account and line creation¶

In a browser, go to the Account Signup tab at http://ellis.<domain>/signup.html and supply details as requested:

• Use whatever name you like.
• Use a real email address, or the password reset function won’t work.
• The signup code to enter in the final box should have been given to you by the deployment administrator.
• Hit “Sign up”.

You are now shown a new account with a phone number (which may take a second or two to show up).

• The password for each number is only shown once - it is only stored encrypted in the system, for security, so cannot be retrieved by the system later. It can be reset by the user if they forget it (see button towards the right).

Try adding and deleting extra lines. Lines can have non-routable numbers for a purely on-net service, or PSTN numbers. Make sure you end up with just a PSTN number assigned to the account.

Address book: for this system there is a single address book which all numbers are by default added to - see the checkbox by the line.

Connecting an X-Lite client¶

Now register for the new PSTN line on your newly created account with your as yet unregistered X-Lite phone.

• Download X-Lite.
• Install and start X-Lite
• Navigate to Softphone / Account Settings
• Account tab:
  • Account name: Whatever you like.
  • User ID: “SIP Username” from the line you’re adding
  • Domain: <deployment domain>, e.g., ngv.example.com
  • Password: “SIP Password” from the line you’re adding
  • Display name: Whatever you like, or omit
  • Authorization name: <SIP Username>@<domain> e.g. 6505550611@ngv.example.com
  • “Register with domain and receive calls” should be selected
  • Send outbound via: Domain
• Topology tab:
  • Select “Auto-detect firewall traversal method using ICE (recommended)”
  • Server address: <domain>
  • User name: <SIP Username>@<domain>
  • Password: “SIP Password” from the line
• Transport tab:
  • Signaling transport: TCP
• hit “OK”
• phone should register.

We have now registered for the new line.

Connecting an Android phone¶

See here for instructions.

Making calls¶

• Make a call using your new line to a PSTN number (eg your regular mobile phone number). Clearwater performs an ENUM lookup to decide how to route the call, and rings the number.
• Make a call using a PSTN number (eg your regular mobile phone) to your new line. Use the online global address book in Ellis to find the number to call.
• Make a call between X-Lite and the Android client, optionally using the address book.

We’ve gone from no user account, no line, and no service to a configured user and working line in five minutes. All of this works without any on-premise equipment, without dedicated servers in a data centre, without any admin intervention at all.

WebRTC support¶

See WebRTC support in Clearwater for how to use a browser instead of a SIP phone as a client.

VoLTE call services¶

Integrating Clearwater with existing services¶

An install of Clearwater will, by default, run with no external requirements and allow simple on-net calls between lines provisioned in Ellis. Clearwater also supports interoperating with various other SIP core devices to add more functionality and integrate with an existing SIP core.

• IBCF function
• Application Server
• External HSS Integration
• CDF Integration
• SIP RFCs supported

Call features¶

Clearwater has a number of call features provided by its built-in TAS, including:

• Call barring
• Call diversion
• Privacy

Clearwater’s support for all IR.92 Supplementary Services is discussed in this document.

Scaling and Redundancy¶

Clearwater is designed from the ground up to scale horizontally across multiple instances to allow it to handle large subscriber counts and to support a high level of redundancy.

• Scaling Numbers
• The local redundancy story is described in the Clearwater Architecture page.
• Geographic redundancy

Operational support¶

To help you manage your deployment, Clearwater provides:

• Command-line/scriptable provisioning tools
• Support for separated management networks
• Deployment monitoring
• Backup
• A troubleshooting guide with instructions for viewing the database entries on Sprout and Homestead/Homer.

Mailing List¶

The mailing list is managed at lists.projectclearwater.org. When posting, please try to

• provide a clear subject line
• provide as much diagnostic information as possible.

You’re more likely to get your question answered quickly and correctly that way!

Issues¶

Clearwater uses github’s Issues system to track problems. To access the Issues system, first determine which project the issue is with - ask in the forums if you’re not sure. Then go to that component’s repository page in github and click the Issues tab at the top.

Remember to do a search for existing issues covering your problem before raising a new one!

Pull Requests¶

Clearwater follows the “Fork & Pull” model of collaborative development, with changes being offered to the main Clearwater codebase via pull requests. If you’re interested in contributing,

• thanks!
• see the github docs for how to create a pull request
• before creating a pull request, please make sure you’ve successfully
  • ensured the code matches our Ruby or C++ coding guidelines
  • compiled the code
  • run the unit tests for the component
  • run the live tests against your changes.

General¶

• Clearwater components are monitored by monit and should be restarted by it if the component fails or hangs. You can check that components are running by issuing monit status. If components are not being monitored as expected, run monit monitor <component> to monitor it. To restart a component, we recommend using service <component> stop to stop the component, and allowing monit to automatically start the component again.
• The Clearwater diagnostics monitor detects crashes in native clearwater processes (Bono, Sprout and Homestead) and captures a diagnostics bundle containing a core file (among other useful information). A diagnostics bundle can also be created by running a command line script (sudo /usr/share/clearwater/bin/gather_diags).
• By default each component logs to /var/log/<service>/, at log level 2 (which only includes errors and very high level events). To see more detailed logs you can enable debug logging; details for how to do this for each component are below. Note that if you want to run stress through your deployment, you should revert the log levels back to the default level.

Ellis¶

The most common problem from Ellis is it reporting “Failed to update server”. This can happen for several reasons.

• If Ellis reports “Failed to update server” when allocating a new number (either after an explicit request or as part of creating a whole new account), check that ellis has free numbers to allocate. The create_numbers.py script is safe to re-run, to ensure that numbers have been allocated.
• Check the /var/log/ellis/ellis-*.log files. If these indicate that a timeout occurred communicating with Homer or Homestead-prov, check that the DNS entries for Homer and Homestead-prov exist and are configured correctly. If these are already correct, check Homer or Homestead-prov to see if they are behaving incorrectly.
• To turn on debug logging for Ellis, write LOG_LEVEL = logging.DEBUG to the local_settings.py file (at /usr/share/clearwater/ellis/local_settings.py). Then restart clearwater-infrastructure (sudo service clearwater-infrastructure restart), and restart Ellis (sudo service ellis stop - it will be restarted by monit).

To examine Ellis’ database, run mysql (as root), then type use ellis; to set the correct database. You can then issue standard SQL queries on the users and numbers tables, e.g. SELECT * FROM users WHERE email = '<email address>'.

Homer and Homestead¶

The most common problem on Homer and Homestead is failing to read or write to the Cassandra database.

• Check that Cassandra is running (sudo monit status). If not, check its /var/log/cassandra/*.log files.
• Check that Cassandra is configured correctly. First access the command-line CQL interface by running cqlsh.
  • If you’re on Homer, type use homer; to set the correct database and then describe tables; - this should report simservs. If this is missing, recreate it by running /usr/share/clearwater/cassandra-schemas/homer.sh.
  • If you’re on Homestead, there are 2 databases.
  • Type use homestead_provisioning; to set the provisioning database and then describe tables; - this should report service_profiles, public, implicit_registration_sets and private.
  • Type use homestead_cache; to set the cache database and then describe tables; as before - this should report impi, impi_mapping and impu.
  • If any of these are missing, recreate them by running /usr/share/clearwater/cassandra-schemas/homestead_cache.sh and /usr/share/clearwater/cassandra-schemas/homestead_provisioning.sh.
• Check that Cassandra is clustered correctly (if running a multi-node system). nodetool status tells you which nodes are in the cluster, and how the keyspace is distributed among them.
• If this doesn’t help, Homer logs to /var/log/homer/homer-*.log and Homestead logs to /var/log/homestead/homestead-*.log and /var/log/homestead-prov/homestead-*.log. To turn on debug logging for Homer or Homestead-prov, write LOG_LEVEL = logging.DEBUG to the local_settings.py file (at /usr/share/clearwater/<homer|homestead>/local_settings.py). Then restart clearwater-infrastructure (sudo service clearwater-infrastructure restart), and restart Homer/Homestead-prov (sudo service <homer|homestead-prov> stop - they will be restarted by monit). To turn on debug logging for Homestead write log_level=5 to /etc/clearwater/user_settings (creating it if it doesn’t exist already), then restart Homestead (sudo service homestead stop - it will be restarted by monit).

To examine Homer or Homestead’s database, run cqlsh and then type use homer;, use homestead_provisioning; or use homestead_cache to set the correct database. You can then issue CQL queries such as SELECT * FROM impi WHERE private_id = '<private user ID>'.

Sprout¶

The most common problem on Sprout is lack of communication with other nodes and processes, causing registration or calls to fail. Check that Homer, Homestead and memcached are reachable and responding.

Sprout maintains registration state in a memcached cluster. It’s a little clunky to examine this data but you can get some basic information out by running . /etc/clearwater/config ; telnet $local_ip 11211 to connect to memcached, issuing stats items. This returns a list of entries of the form STAT items:<slab ID>:.... You can then query the keys in each of the slabs with stats cachedump <slab ID> 0.

Memcached logs to /var/log/memcached.log. It logs very little by default, but it is possible to make it more verbose by editing /etc/memcached_11211.conf, uncommenting the -vv line, and then restarting memcached.

To turn on debug logging for Sprout write log_level=5 to /etc/clearwater/user_settings (creating it if it doesn’t exist already), then restart Sprout (sudo service sprout stop - it will be restarted by monit).

Sprout also uses Chronos to track registration, subscription and authorization timeouts. Chronos logs to /var/log/chronos/chronos*. Details of how to edit the Chronos configuration are here.

If you see Sprout dying/restarting with no apparent cause in /var/log/sprout/sprout*.txt, check /var/log/monit.log and /var/log/syslog around that time - these can sometimes give clues as to the cause.

Bono¶

The most common problem on Bono is lack of communication with Sprout. Check that Sprout is reachable and responding.

To turn on debug logging for Bono write log_level=5 to /etc/clearwater/user_settings (creating it if it doesn’t exist already), then restart Bono (sudo service bono stop - it will be restarted by monit).

If you see Bono dying/restarting with no apparent cause in /var/log/bono/bono*.txt, check /var/log/monit.log and /var/log/syslog around that time - these can sometimes give clues as to the cause.

Ralf¶

The most common problem on Ralf is lack of communication with a CCF. Check that your CCF is reachable and responding (if you don’t have a CCF, you don’t need a Ralf).

To turn on debug logging for Ralf write log_level=5 to /etc/clearwater/user_settings (creating it if it doesn’t exist already), then restart Ralf (sudo service ralf stop - it will be restarted by monit).

Ralf also uses Chronos to track call timeouts. Chronos logs to /var/log/chronos/chronos*. Details of how to edit the Chronos configuration are here.

If you see Ralf dying/restarting with no apparent cause in /var/log/ralf/ralf*.txt, check /var/log/monit.log and /var/log/syslog around that time - these can sometimes give clues as to the cause.

Deployment Management¶

Clearwater comes with a system that automate clustering and configuration sharing. If you cannot scale your deployment up or down, or if configuration changes are not being applied, this system may not be working.

• The management system logs to /var/log/clearwater-etcd, /var/log/clearwater-cluster-manager and /var/log/clearwater-config-manager. To turn on debug logging write log_level=5 to etc/clearwater/user_settings (creating it if it doesn’t exist already), then restart the etcd processes (sudo service <clearwater-config-manager|clearwater-cluster-manager> stop - they will be restarted by monit)

• /usr/share/clearwater/clearwater-cluster-manager/scripts/check_cluster_state will display information about the state of the various data-store clusters used by Clearwater.

• sudo /usr/share/clearwater/clearwater-config-manager/scripts/check_config_sync will display whether the node has learned shared configuration.

• The following commands can be useful for inspecting the state of the underlying etcd cluster used by the management system:

  clearwater-etcdctl cluster-health
  clearwater-etcdctl member list
  

Getting Help¶

If none of the above helped, please try the mailing list.

Installation Methods¶

Installation Steps¶

The installation process for a full Clearwater deployment (i.e. not an all-in-one) can be described at a high level as follows:

• Sourcing enough machines to host the deployment (minimum is 6) and installing an OS on them all. Ubuntu 14.04 is the recommended and tested OS for hosting Clearwater.
• Preparing each machine to allow installation of the Clearwater software.
• Installing the Clearwater software onto the machines.
• Configuring firewalls to allow the various machines to talk to each other as required.
• Configuring DNS records to allow the machines to find each other and to expose the deployment to clients.

Next Steps¶

Once you have installed your deployment, you’ll want to test that it works.

• Making your first call
• Running the live test suite

Images¶

All-in-one images consist of

• Ubuntu 14.04, configured to use DHCP
• bono, sprout, homestead, homer and ellis
• Clearwater auto-configuration scripts.

On boot, the image retrieves its IP configuration over DHCP and the auto-configuration scripts then configure the bono, sprout, homestead, homer and ellis software to match.

The image is designed to run on a virtual machine with a single core, 2GB RAM and 8GB of disk space. On EC2, this is an t2.small.

Capabilities and Restrictions¶

Since the all-in-one images include all of bono, sprout, homestead, homer and ellis, they are capable of almost anything a full deployment is capable of.

The key restrictions of all-in-one images are

• hard-coded realm - the all-in-one image uses a hard-coded realm of example.com so your SIP URI might be sip:6505551234@example.com - by default, SIP uses this realm for routing but example.com won’t resolve to your host so you need to configure an outbound proxy on all your SIP clients (more details later)
• performance - since all software runs on a single virtual machine, performance is significantly lower than even the smallest scale deployment
• scalability - there is no option to scale up and add more virtual machines to boost capacity - for this, you must create a normal deployment
• fault-tolerance - since everything runs on a single virtual machine, if that virtual machine fails, the service as a whole fails.

Installation Options¶

All-in-one images can be installed on EC2 or on your own virtualization platform, as long as it supports OVF (Open Virtualization Format).

• To install on EC2, follow the all-in-one EC2 AMI installation instructions.
• To install on your own virtualization platform, follow the all-in-one OVF installation instructions.

Manual Build¶

If your virtualization platform is not EC2 and doesn’t support OVF, you may still be able to manually build an all-in-one node. To do so,

1. install Ubuntu 14.04 - 64bit server edition

2. find the preseed/late_command entry in the all-in-one image’s install script - as of writing this is as follows, but please check the linked file for the latest version

   d-i preseed/late_command string in-target bash -c '{ echo "#!/bin/bash" ; \
                                               echo "set -e" ; \
                                               echo "repo=... # filled in by make_ovf.sh" ; \
                                               echo "curl -L https://raw.githubusercontent.com/Metaswitch/clearwater-infrastructure/master/scripts/clearwater-aio-install.sh | sudo bash -s clearwater-auto-config-generic $repo" ; \
                                               echo "python create_numbers.py --start 6505550000 --count 1000" ; \
                                               echo "rm /etc/rc2.d/S99zclearwater-aio-first-boot" ; \
                                               echo "poweroff" ; } > /etc/rc2.d/S99zclearwater-aio-first-boot ; \
                                             chmod a+x /etc/rc2.d/S99zclearwater-aio-first-boot'
   

3. run the command (stripping the d-i preseed/late_command string in-target prefix, filling in the repo variable - the default is http://repo.cw-ngv.com/stable, and stripping the \) - this will create an /etc/rc2.d/S99zclearwater-aio-first-boot script

4. run the /etc/rc2.d/S99zclearwater-aio-first-boot script - this will install the all-in-one software and then shutdown (ready for an image to be taken)

5. restart the node.

The Install Process¶

The automated install system is made up of two phases:

• A series of steps that need to be performed once, before any deployment is created.
• A simpler set of instructions that will actually create the deployment.

Once the first phase has been completed, multiple deployments (of various sizes) can be created by repeating the second phase multiple times.

The first phase:

• Installing a Chef server
• This server will track the created Clearwater nodes and allow the client access to them.
• Configuring a Chef workstation
• This machine will be the one on which deployments will be defined and managed.

The second phase:

• Creating a deployment environment
• The automated install supports the existence and management of multiple deployments simultaneously, each deployment lives in an environment to keep them separate.
• Creating the deployment
• Actually provisioning the servers, installing the Clearwater software and configuring DNS.

Next steps¶

Once you’ve followed the instructions above, your Clearwater deployment is ready for use. Be aware that the newly created DNS entries may take some time to propagate fully through your network and until propagation is complete, your deployment may not function correctly. This propagation usually takes no more than 5 minutes.

To test the deployment, you can try making some real calls, or run the provided live test framework.

• Making your first call
• Running the live test framework

Prerequisites¶

Before starting this process you will need the following:

• Six machines running clean installs of Ubuntu 14.04 - 64bit server edition.
  • The software has been tested on Amazon EC2 t2.small instances (i.e. 1 vCPU, 2 GB RAM), so any machines at least as powerful as one of them will be sufficient.
  • Each machine will take on a separate role in the final deployment. The system requirements for each role are the same thus the allocation of roles to machines can be arbitrary.
  • The firewalls of these devices must be independently configurable. This may require some attention when commissioning the machines. For example, in Amazon’s EC2, they should all be created in separate security groups.
  • On Amazon EC2, we’ve tested both within a VPC and without. If using a VPC, we recommend using the “VPC with a Single Public Subnet” model (in the “VPC Wizard”) as this is simplest.
• A publicly accessible IP address of each of the above machines and a private IP address for each of them (these may be the same address depending on the machine environment). These will be referred to as <publicIP> and <privateIP> below. (If running on Amazon EC2 in a VPC, you must explicitly add this IP address by ticking the “Automatically assign a public IP address” checkbox on creation.)
• The FQDN of the machine, which resolves to the machine’s public IP address (if the machine has no FQDN, you should instead use the public IP). Referred to as <hostname> below.
• SSH access to the above machines to a user authorised to use sudo. If your system does not come with such a user pre-configured, add a user with sudo adduser <username> and then authorize them to use sudo with sudo adduser <username> sudo.
• A DNS zone in which to install your deployment and the ability to configure records within that zone. This zone will be referred to as <zone> below.
• If you are not using the Project Clearwater provided Debian repository, you will need to know the URL (and, if applicable, the public GPG key) of your repository.

Configure the APT software sources¶

Configure each machine so that APT can use the Clearwater repository server.

Under sudo, create /etc/apt/sources.list.d/clearwater.list with the following contents:

deb http://repo.cw-ngv.com/stable binary/

Note: If you are not installing from the provided Clearwater Debian repository, replace the URL in this file to point to your Debian package repository.

Once this is created install the signing key used by the Clearwater server with:

curl -L http://repo.cw-ngv.com/repo_key | sudo apt-key add -

You should check the key fingerprint with:

sudo apt-key finger

The output should contain the following - check the fingerprint carefully.

pub   4096R/22B97904 2013-04-30
      Key fingerprint = 9213 4604 DE32 7DF7 FEB7  2026 111D BE47 22B9 7904
uid                  Project Clearwater Maintainers <maintainers@projectclearwater.org>
sub   4096R/46EC5B7F 2013-04-30

Once the above steps have been performed, run the following to re-index your package manager:

sudo apt-get update

Determine Machine Roles¶

At this point, you should decide (if you haven’t already) which of the six machines will take on which of the Clearwater roles.

The six roles are:

• ellis
• bono - This role also hosts a restund STUN server
• sprout
• homer
• homestead
• ralf

Firewall configuration¶

We need to make sure the Clearwater nodes can all talk to each other. To do this, you will need to open up some ports in the firewalls in your network. The ports used by Clearwater are listed in Clearwater IP Port Usage. Configure all of these ports to be open to the appropriate hosts before continuing to the next step. If you are running on a platform that has multiple physical or virtual interfaces and the option to apply different firewall rules on each, make sure that you open these ports on the correct interfaces.

Create the per-node configuration.¶

On each machine create the file /etc/clearwater/local_config with the following contents.

local_ip=<privateIP>
public_ip=<publicIP>
public_hostname=<hostname>
etcd_cluster="<comma separated list of private IPs>"

Note that the etcd_cluster variable should be set to a comma separated list that contains the private IP address of the nodes you created above. For example if the nodes had addresses 10.0.0.1 to 10.0.0.6, etcd_cluster should be set to "10.0.0.1,10.0.0.2,10.0.0.3,10.0.0.4,10.0.0.5,10.0.0.6"

If you are creating a geographically redundant deployment, then:

• etcd_cluster should contain the IP addresses of nodes in both sites
• you should set local_site_name and remote_site_name in /etc/clearwater/local_config.

These names are arbitrary, but should reflect the node’s location (e.g. a node in site A should have local_site_name=siteA and remote_site_name=siteB, whereas a node in site B should have local_site_name=siteB and remote_site_name=siteA):

If this machine will be a Sprout or Ralf node create the file /etc/chronos/chronos.conf with the following contents:

[http]
bind-address = <privateIP>
bind-port = 7253
threads = 50

[logging]
folder = /var/log/chronos
level = 2

[alarms]
enabled = true

[exceptions]
max_ttl = 600

Install Node-Specific Software¶

ssh onto each box in turn and follow the appropriate instructions below according to the role the node will take in the deployment:

sudo DEBIAN_FRONTEND=noninteractive apt-get install ellis --yes
sudo DEBIAN_FRONTEND=noninteractive apt-get install clearwater-management --yes

sudo DEBIAN_FRONTEND=noninteractive apt-get install bono restund --yes
sudo DEBIAN_FRONTEND=noninteractive apt-get install clearwater-management --yes

sudo DEBIAN_FRONTEND=noninteractive apt-get install sprout --yes
sudo DEBIAN_FRONTEND=noninteractive apt-get install clearwater-management --yes

sudo DEBIAN_FRONTEND=noninteractive apt-get install memento-as memento-nginx --yes

sudo DEBIAN_FRONTEND=noninteractive apt-get install homer --yes
sudo DEBIAN_FRONTEND=noninteractive apt-get install clearwater-management --yes

sudo DEBIAN_FRONTEND=noninteractive apt-get install homestead homestead-prov clearwater-prov-tools --yes
sudo DEBIAN_FRONTEND=noninteractive apt-get install clearwater-management --yes

sudo DEBIAN_FRONTEND=noninteractive apt-get install ralf --yes
sudo DEBIAN_FRONTEND=noninteractive apt-get install clearwater-management --yes

SNMP statistics¶

Sprout, Bono and Homestead nodes expose statistics over SNMP. This function is not installed by default. If you want to enable it follow the instruction in our SNMP documentation.

Provide Shared Configuration¶

Log onto any node in the deployment and create the file /etc/clearwater/shared_config with the following contents:

# Deployment definitions
home_domain=<zone>
sprout_hostname=sprout.<zone>
hs_hostname=hs.<zone>:8888
hs_provisioning_hostname=hs.<zone>:8889
ralf_hostname=ralf.<zone>:10888
xdms_hostname=homer.<zone>:7888

# Email server configuration
smtp_smarthost=<smtp server>
smtp_username=<username>
smtp_password=<password>
email_recovery_sender=clearwater@example.org

# Keys
signup_key=<secret>
turn_workaround=<secret>
ellis_api_key=<secret>
ellis_cookie_key=<secret>

If you wish to enable the optional I-CSCF function, also add the following:

# I-CSCF/S-CSCF configuration
icscf=5052
upstream_hostname=<sprout_hostname>
upstream_port=5052

If you wish to enable the optional external HSS lookups, add the following:

# HSS configuration
hss_hostname=<address of your HSS>
hss_port=3868

If you want to host multiple domains from the same Clearwater deployment, add the following (and configure DNS to route all domains to the same servers):

# Additional domains
additional_home_domains=<domain 1>,<domain 2>,<domain 3>...

If you want your Sprout nodes to include Gemini/Memento Application Servers add the following:

# Application Servers
gemini_enabled='Y'
memento_enabled='Y'

See the Chef instructions for more information on how to fill these in. The values marked <secret> must be set to secure values to protect your deployment from unauthorized access. To modify these settings after the deployment is created, follow these instructions.

Now run the following to upload the configuration to a shared database and propagate it around the cluster.

/usr/share/clearwater/clearwater-config-manager/scripts/upload_shared_config

{
   "s-cscfs" : [
       {   "server" : "sip:<sprout_domain>:5054;transport=TCP",
           "priority" : 0,
           "weight" : 100,
           "capabilities" : [<comma separated capabilities>]
       }
   ]
}

Provision Telephone Numbers in Ellis¶

Log onto you Ellis node and provision a pool of numbers in Ellis. The command given here will generate 1000 numbers starting at sip:6505550000@<zone>, meaning none of the generated numbers will be routable outside of the Clearwater deployment. For more details on creating numbers, see the create_numbers.py documentation.

sudo bash -c "export PATH=/usr/share/clearwater/ellis/env/bin:$PATH ;
              cd /usr/share/clearwater/ellis/src/metaswitch/ellis/tools/ ;
              python create_numbers.py --start 6505550000 --count 1000"

On success, you should see some output from python about importing eggs and then the following.

Created 1000 numbers, 0 already present in database

This command is idempotent, so it’s safe to run it multiple times. If you’ve run it once before, you’ll see the following instead.

Created 0 numbers, 1000 already present in database

DNS Records¶

Clearwater uses DNS records to allow each node to find the others it needs to talk to to carry calls. At this point, you should create the DNS entries for your deployment before continuing to the next step. Clearwater DNS Usage describes the entries that are required before Clearwater will be able to carry service.

Although not required, we also suggest that you configure individual DNS records for each of the machines in your deployment to allow easy access to them if needed.

Be aware that DNS record creation can take time to propagate, you can check whether your newly configured records have propagated successfully by running ``dig <record>`` on each Clearwater machine and checking that the correct IP address is returned.

Where next?¶

Once you’ve reached this point, your Clearwater deployment is ready to handle calls. See the following pages for instructions on making your first call and running the supplied regression test suite.

• Making your first call
• Running the live test suite

Larger-Scale Deployments¶

If you’re intending to spin up a larger-scale deployment containing more than one node of each types, it’s recommended that you use the automated install process, as this makes scaling up and down very straight-forward. If for some reason you can’t, you can add nodes to the deployment using the Elastic Scaling Instructions

Launch Process¶

Project Clearwater’s all-in-one node is already available as a pre-built AMI, which can be found in the Community AMIs list on the US East region of EC2. Launching this follows exactly the same process as for other EC2 AMIs.

Before you launch the node, you will need an EC2 keypair, and a security group configured to provide access to the required ports.

To launch the node

• From the EC2 console, make sure you’re in the US East region, then select “Instances”, “Launch instance” and then “Classic Wizard”
• Select the “Community AMIs” tab, and search for “Clearwater”
• Press “Select” for the Clearwater all-in-one AMI. Take the most recent version unless you have a good reason not to.
• Choose the Instance Type you require (the node runs fine for basic functional testing on an t2.small)
• From the remaining pages of parameters, the only ones that need to be set are Name (give the node whatever convenient name you wish), keypair and security group.

On the last page, press “Launch”, and wait for the node to be started by EC2.

Running and Using the Image¶

Once the node has launched, you can SSH to it using the keypair you supplied at launch time, and username ubuntu.

You can then try making your first call and running the live tests - for these you will need the signup key, which is secret. You will probably want to change this to a more secure value - see “Modifying Clearwater settings” for how to do this.

Supported Platforms¶

This process should work on any virtualization platform that supports OVFs using x86-64 CPUs, but has only been tested on

• VMware Player
• VirtualBox
• VMware ESXi.

The image uses DHCP to gets its IP configuration, so the virtualization platform must either serve DHCP natively or be connected to a network with a DHCP server.

If you install/run the OVF on another platform, please let us know how you get on on the mailing list so we can update this page.

Installation Process¶

To install the OVF, you must first download it from http://vm-images.cw-ngv.com/cw-aio.ova and save it to disk.

Then, you must import it into your virtualization platform. The process for this varies.

• On VMware Player, choose File->Open a Virtual Machine from the menu and select the cw-aio.ova file you downloaded. On the Import Virtual Machine dialog that appears, the defaults are normally fine, so you can just click Import.
• On VirtualBox, choose File->Import Appliance... from the menu. In the Appliance Import Wizard, click Choose..., select the cw-aio.ova file you downloaded and click Next. On the next tab, you can view the settings and then click Import.
• On VMware ESXi, using the VMware vSphere Client, choose File->Deploy OVF Template... from the menu. Select the cw-aio.ova file you downloaded and click through assigning it a suitable name, location and attached network (which must support DHCP) before finally clicking Finish to create the virtual machine.

Running and Using the Image¶

Once you’ve installed the virtual machine, you should start it in the usual way for your virtualization platform.

If you attach to the console, you should see an Ubuntu loading screen and then be dropped at a cw-aio login prompt. The username is ubuntu and the password is cw-aio. Note that the console is hard-coded to use a US keyboard, so if you have a different keyboard you might find that keys are remapped - in particular the - key.

The OVF provides 3 network services.

• SSH - username is ubuntu and password is cw-aio
• HTTP to ellis for subscriber management - sign-up code is secret. You will probably want to change this to a more secure value - see “Modifying Clearwater settings” for how to do this.
• SIP to bono for call signaling - credentials are provisioned through ellis.

How these network services are exposed can vary depending on the capabilities of the platform.

• VMware Player sets up a private network between your PC and the virtual machine and normally assigns the virtual machine IP address 192.168.28.128. To access ellis, you’ll need to point your browser at http://192.168.28.128. To register over SIP, you’ll need to configure an outbound proxy of 192.168.28.128 port 5060.
• VirtualBox uses NAT on the local IP address, exposing SSH on port 8022, HTTP on port 8080 and SIP on port 8060. To access ellis, you’ll need to point your browser at http://localhost:8080. To register over SIP, you’ll need to configure an outbound proxy of localhost port 8060.
• VMware ESXi runs the host as normal on the network, so you can connect to it directly. To find out its IP address, log in over the console and type hostname -I. To access ellis, just point your browser at this IP address. To register over SIP, you’ll need to configure an outbound proxy for this IP address.

Once you’ve successfully connected to ellis, try making your first call - just remember to configure the SIP outbound proxy as discussed above.

Prerequisites¶

• An Amazon EC2 account.
• A DNS root domain configured as a hosted zone with Route53 (Amazon’s built-in DNS service, accessible from the EC2 console). This domain will be referred to as <zone> in this document.

Create the instance¶

Create a t2.small AWS EC2 instance running Ubuntu Server 14.04.2 LTS using the AWS web interface. The SSH keypair you provide here is referred to below as <amazon_ssh_key>. It is easiest if you use the same SSH keypair for all of your instances.

Configure its security group to allow access SSH, HTTP, and HTTPS access.

Configure a DNS entry for this machine, chef-server.<zone>. (The precise name isn’t important, but we use this consistently in the documentation that follows.) It should have a non-aliased A record pointing at the public IP address of the instance as displayed in the EC2 console.

Once the instance is up and running and you can connect to it over SSH, you may continue to the next steps.

If you make a mistake, simply delete the instance permanently by selecting “Terminate” in the EC2 console, and start again. The terminated instance may take a few minutes to disappear from the console.

Install and configure the Chef server¶

The chef documentation explains how to install and configure the chef server. These instructions involve setting up a user and an organization.

• The user represents you as a user of chef. Pick whatever user name and password you like. We refer to these as <chef-user-name> and <chef-user-password>
• Organizations allow different groups to use the same chef server, but be isolated from one another. You can choose any organization name you like (e.g. “clearwater”). We refer to this as <org-name>

Follow steps 1-6 in the chef docs.

Once you have completed these steps, copy the <chef-user-name>.pem file off of the chef server - you will need it when installing a chef workstation.

Next steps¶

Once your server is installed, you can continue on to install a chef workstation.

Prerequisites¶

• An Amazon EC2 account.
• A DNS root domain configured with Route53 (Amazon’s built-in DNS service, accessible from the EC2 console. This domain will be referred to as <zone> in this document.
• You must have installed a Chef server and thus know the <chef-user-name> and <chef-user-password> for your server.
• A web-browser with which you can visit the Chef server Web UI.

Create the instance¶

Create a t2.micro AWS EC2 instance running Ubuntu Server 14.04.2 LTS using the AWS web interface. Configure its security group to allow access on port 22 (for SSH). The SSH keypair you provide here is referred to below as <amazon_ssh_key>. It is easiest if you use the same SSH keypair for all of your instances.

Configure a DNS entry for this machine, chef workstation.<zone>. (The precise name isn’t important, but we use this consistently in the documentation that follows.) It should have a non-aliased A record pointing at the public IP address of the instance as displayed in the EC2 console.

Once the instance is up and running and you can connect to it over SSH, you may continue to the next steps.

If you make a mistake, simply delete the instance permanently by selecting “Terminate” in the EC2 console, and start again. The terminated instance may take a few minutes to disappear from the console.

Install Ruby 1.9.3¶

The Clearwater chef plugins use features from Ruby 1.9.3. To start run the following command.

curl -L https://get.rvm.io | bash -s stable

This may fail due to missing GPG signatures. If this happens it will suggest a command to run to resolve the problem (e.g. gpg –keyserver hkp://keys.gnupg.net –recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3`). Run the command suggested, then run the above command again, which should now succeed).

Next install the required ruby version.

source ~/.rvm/scripts/rvm
rvm autolibs enable
rvm install 1.9.3
rvm use 1.9.3

At this point, ruby --version should indicate that 1.9.3 is in use.

Installing the Clearwater Chef extensions¶

On the chef workstation machine, install git and dependent libraries.

sudo apt-get install git libxml2-dev libxslt1-dev

Clone the Clearwater Chef repository.

git clone -b stable --recursive git://github.com/Metaswitch/chef.git ~/chef

This will have created a chef folder in your home directory, navigate there now.

cd ~/chef

Finally install the Ruby libraries that are needed by our scripts.

bundle install

Creating a Chef user¶

You will need to configure yourself as a user on the chef server in order to use chef.

• If you are the person who created the chef server you wil already have added yourself as a user, and will know your username, organization name, and you will have a private key (<chef-user-name>, <org-name>, <chef-user-name>.pem respectively). These will be needed later.

• If you did not create the chef server, you will need to add an account for yourself. Log SSH on to the chef server and run the following commands, substituting in appropriate values for USER_NAME, FIRST_NAME, LAST_NAME, PASSWORD and ORG_NAME. We’ll refer to the username as <chef-user-name> and the organization as <org-name>. This will create a <chef-user-name>.pem file in the current directory - save it for later.

  sudo chef-server-ctl user-create USER_NAME FIRST_NAME LAST_NAME EMAIL PASSWORD --filename USER_NAME.pem
  sudo chef-server-ctl org-user-add ORG_NAME USER_NAME --admin
  

Configure the chef workstation machine¶

Back on the chef workstation machine, create a .chef folder in your home directory.

mkdir ~/.chef

Copy the <chef-user-name>.pem file you saved off earlier to ~/.chef/<chef-user-name.pem>

Copy the validator key from the chef server to your client. You will need to either copy the Amazon SSH key to the client and use it, or copy the validator

scp -i <amazon_ssh_key>.pem ubuntu@chef-server.<zone>:<org-name>-validator.pem ~/.chef/

or (on an intermediate box with the SSH key available)

scp -i <amazon_ssh_key>.pem ubuntu@chef-server.<zone>:<org-name>-validator.pem .
scp -i <amazon_ssh_key>.pem <org-name>-validator.pem ubuntu@chef workstation.<zone>:~/.chef/

Configure knife using the built in auto-configuration tool.

knife configure

• Use the default value for the config location.
• The Chef server URL should be https://chef-server.<zone>/organizations/<org-name>
• The Chef client name should be <chef-user-name>
• The validation client name should be <org-name>-validator
• The validation key location should be ~/.chef/<org-name>-validator.pem.
• The chef repository path should be ~/chef/

Obtain AWS access keys¶

To allow the Clearwater extensions to create AWS instances or configure Route53 DNS entries, you will need to supply your AWS access key and secret access key. To find your AWS keys, you must be logged in as the main AWS user, not an IAM user. Go to http://aws.amazon.com and click on My Account/Console then Security Credentials. From there, under the Access Credentials section of the page, click on the Access Keys tab to view your access key. The access key is referred to as <accessKey> below. To see your secret access key, just click on the Show link under Secret Access Key. The secret access key will be referred to as <secretKey> below.

Add deployment-specific configuration¶

Now add the following lines to the bottom of your ~/.chef/knife.rb file.

# AWS deployment keys.
knife[:aws_access_key_id]     = "<accessKey>"
knife[:aws_secret_access_key] = "<secretKey>"

# Signup key. Anyone with this key can create accounts
# on the deployment. Set to a secure value.
knife[:signup_key]        = "secret"

# TURN workaround password, used by faulty WebRTC clients.
# Anyone with this password can use the deployment to send
# arbitrary amounts of data. Set to a secure value.
knife[:turn_workaround]   = "password"

# Ellis API key. Used by internal scripts to
# provision, update and delete user accounts without a password.
# Set to a secure value.
knife[:ellis_api_key]     = "secret"

# Ellis cookie key. Used to prevent spoofing of Ellis cookies. Set
# to a secure value.
knife[:ellis_cookie_key]  = "secret"

# SMTP credentials as supplied by your email provider.
knife[:smtp_server]       = "localhost"
knife[:smtp_username]     = ""
knife[:smtp_password]     = ""

# Sender to use for password recovery emails. For some
# SMTP servers (e.g., Amazon SES) this email address
# must be validated or email sending will fail.
knife[:email_sender]      = "clearwater@example.com"

Fill in the values appropriate to your deployment using a text editor as directed.

• The AWS deployment keys are the ones you obtained above.

• The keys and passwords marked “Set to a secure value” above should be set to secure random values, to protect your deployment from unauthorised access. An easy way to generate a secure random key on a Linux system is as follows:

  head -c6 /dev/random | base64
  

The signup_key must be supplied by new users when they create an account on the system.

The turn_workaround must be supplied by certain WebRTC clients when using TURN. It controls access to media relay function.

The ellis_api_key and ellis_cookie_key are used internally.

• The SMTP credentials are required only for password recovery. If you leave them unchanged, this function will not work.

Test your settings¶

Test that knife is configured correctly

knife client list

This should return a list of clients and not raise any errors.

Upload Clearwater definitions to Chef server¶

The Chef server needs to be told the definitions for the various Clearwater node types. To do this, run

cd ~/chef
knife cookbook upload apt
knife cookbook upload chef-solo-search
knife cookbook upload clearwater
find roles/*.rb -exec knife role from file {} \;

You will need to re-do this step if make any changes to your knife.rb settings.

Next steps¶

At this point, the Chef server is up and running and ready to manage installs and the chef client is ready to create deployments. The next step is to create a deployment environment.

Prerequisites¶

• You must have installed a Chef workstation.
• You must have SSH access to the ubuntu user on the chef workstation machine.

Creating a keypair¶

You need to configure the AWS keypair used to access your Clearwater deployment. You may reuse the one you created when creating your Chef client and server; or you may create a new one (EC2 Dashboard > Network & Security > Key Pairs > Create Key Pair) for finer-grained access rights. The name you give your keypair will be referred to as <keypair_name> from this point.

Amazon will give you a <keypair_name>.pem file; copy that file to the /home/ubuntu/.chef/ directory on your Chef client, and make it readable only to your user with chmod 0700 /home/ubuntu/.chef ; chmod 0400 /home/ubuntu/.chef/<keypair_name>.pem.

Creating the environment¶

Before creating an environment, choose a name (e.g. “clearwater”) which will be referred to as <name> in the following steps. Now, on the chef workstation machine, create a file in ~/chef/environments/<name>.rb with the following contents:

name "<name>"
description "Clearwater deployment - <name>"
cookbook_versions "clearwater" => "= 0.1.0"
override_attributes "clearwater" => {
  "root_domain" => "<zone>",
  "availability_zones" => ["us-east-1a", "us-east-1b"],
  "repo_server" => "http://repo.cw-ngv.com/stable",
  "number_start" => "6505550000",
  "number_count" => 1000,
  "keypair" => "<keypair_name>",
  "keypair_dir" => "~/.chef/",
  "pstn_number_count" => 0,
  "gr" => false
}

Note that the value of keypair should not include the trailing .pem. Note also that for an all-in-one node the root-domain parameter is superfluous and will be ignored.

By default, your deployment will be created in the US East (North Virginia) region. However, if you want to deploy in another region, you must

• set the region property in the override_attributes "clearwater" block (e.g. to us-west-2)
• set the availability_zones property correspondingly (e.g. ["us-west-2a", "us-west-2b"])
• open ~/chef/plugins/knife/boxes.rb, search for @@default_image, and comment in the EC2 AMI entry for the region you desire.

These fields override attributes defined and documented in the clearwater-infrastructure role.

If you want to use a different SIP registration period from the default (which is 5 minutes) add a line like "reg_max_expires" => <timeout_in_secs>, to the override_attributes "clearwater" block.

If you want to test geographic redundancy function, use "gr" => true instead of "gr" => false. This will cause odd-numbered and even-numbered nodes to be treated as separate logical “sites” - they are not actually in different EC2 regions (cross-region security group rules make that difficult), but does allow you to see and test GR function.

To modify these settings after the deployment is created, follow these instructions.

Uploading the environment¶

The newly created environment needs to be uploaded to the Chef server before it can be used.

cd ~/chef
knife environment from file environments/<name>.rb

Next steps¶

At this point, your deployment environment is created and can be used to create a new deployment.

Prerequisites¶

• You must have created the chef workstation machine and have SSH access to the ubuntu user on it.
• You must have created a deployment environment and know its name, <name>.

Creating a Deployment¶

You now have two options - you can create an All-in-One node, where all the Clearwater components are run on a single machine instance, or a larger deployment which can potentially have numerous instances of each component.

cd ~/chef
knife box create cw_aio -E <name>

cd ~/chef
knife deployment resize -E <name> --ralf-count 1 -V

Next steps¶

Now your deployment is installed and ready to use, you’ll want to test it.

• Making your first call
• Running the live tests

Prerequisites¶

• You’ve installed Clearwater
• You have access to two SIP clients.
• If you have installed Clearwater on VirtualBox using the All-In-One image you must use Zoiper as one of your clients. For the other client (or for other install modes) you may use any standard SIP client, we’ve tested with the following:
  • X-Lite
  • Bria
  • Jitsi
  • Blink
  • Stock Android SIP client
  • Zoiper Android/iOS SIP client
  • Media5-fone iOS SIP client (on iPhone 4 and 4S)
• You have access to a web-browser. We’ve tested with:
• Google Chrome

Work out your base domain¶

If you installed Clearwater manually, your base DNS name will simply be <zone>. If you installed using the automated install process, your base DNS name will be <name>.<zone>. If you installed an All-in-One node, your base name will be example.com.

For the rest of these instructions, the base DNS name will be referred to as <domain>.

Work out your All-in-One node’s identity¶

This step is only required if you installed an All-in-One node, either from an AMI or an OVF. If you installed manually or using the automated install process, just move on to the next step.

If you installed an All-in-One node from an Amazon AMI, you need the public DNS name that EC2 has assigned to your node. This will look something like ec2-12-34-56-78.compute-1.amazonaws.com and can be found on the EC2 Dashboard on the “instances” panel.

If you installed an All-in-One node from an OVF image in VMPlayer or VMWare, you need the IP address that was assigned to the node via DHCP. You can find this out by logging into the node’s console and typing hostname -I.

If you installed an All-in-One node from an OVF in VirtualBox, you simply need localhost.

For the rest of these instructions, the All-in-One node’s identity will be referred to as <aio-identity>.

Work out your Ellis URL¶

If you installed Clearwater manually or using the automated install process, your Ellis URL will simply be http://ellis.<domain>. If you installed an All-in-One node, your Ellis URL will be http://<aio-identity>.

Create a number for your client¶

In your browser, navigate to the Ellis URL you worked out above.

Sign up as a new user, using the signup code you set as signup_key when configuring your deployment.

Ellis will automatically allocate you a new number and display its password to you. Remember this password as it will only be displayed once. From now on, we will use <username> to refer to the SIP username (e.g. 6505551234) and <password> to refer to the password.

Configure your client¶

Client configuration methods vary by client, but the following information should be sufficient to allow your client to register with Clearwater.

• SIP Username: <username>
• SIP Password: <password>
• SIP Domain: <domain>
• Authorization Name: <username>@<domain>
• Transport: TCP
• STUN/TURN/ICE:
• Enabled: true
• Server: <domain> (or <aio-identity> on an All-in-One node)
• Username: <username>@<domain>
• Password: <password>

Extra configuration to use an All-in-One node

If you are using an All-in-One node, you will also need to configure an outbound proxy at your client.

• Outbound Proxy address: <aio-identity>
• Port: 5060 (or 8060 if installed in VirtualBox)

Once these settings have been applied, your client will register with Clearwater. Note that X-Lite may need to be restarted before it will set up a STUN connection.

Configure a second number and client¶

Create a new number in Ellis, either by creating a new Ellis user, or by clicking the Add Number button in the Ellis UI to add one for the existing user.

Configure a second SIP client with the new number’s credentials as above.

Make the call¶

From one client (Zoiper if running an All-in-One node in VirtualBox), dial the <username> of the other client to make the call. Answer the call and check you have two-way media.

Next steps¶

Now that you’ve got a basic call working, check that all the features of your deployment are working by running the live tests or explore Clearwater to see what else Clearwater offers.

Instructions¶

1. Ensure that you have a data connection (either through 3G or WiFi).
2. Launch your standard dialer (often called Phone or Dialer).
3. Bring up the menu. Depending on your phone, this may be via a physical Menu button, or via an icon (consisting of three dots in a vertical line in either the top right or bottom right).
4. From the menu, navigate to the internet calling accounts menu.
   • On many devices this is located in Settings (or Call Settings) > Internet Call Settings > Accounts.
   • On the Nexus 6 running 5.1.1 this is located in Settings > Calls > Calling accounts > Internet Call Settings > Internet calling (SIP) accounts.
5. Press Add account and enter its configuration.
   • Set Username, Password and Server to their usual values.
   • Open the Optional settings panel to see and set the following options.
   • Set Transport type to TCP.
   • Set Authentication username to “sip:<username>@<server>”, substituting (note: once we switch to storing user digest by private id, this will change)
   • Some SIP clients do not have an Authentication username field. To make calls through Clearwater using these clients you will have to configure an external HSS to allow one subscriber to have two private identities. This will allow the subscriber to have one private identity of the form ``1234@example.com`` (the IMS standard form), and one private identity ``1234`` (which the phone will default to in the absence of a Authentication username).
6. Once done, choose Save.
7. Enable making internet calls.
   • On many devices this is located in the Call menu (which can be accessed from the dialler by selecting Settings as above). Once here under Internet Call Settings select Use Internet Calling and set this to Ask for each call.
   • On a Nexus 6 running 5.1.1 this is located in the Calling accounts menu. From here select Use internet calling and set this to Only for Internet Calls.
8. Enable receiving internet calls (note this significantly reduces your battery life).
   • On many devices this is located in Settings > Internet Call Settings > Accounts once here select Receive Incoming Calls.
   • On the Nexus 6 running 5.1.1 this is located in Settings > Calls > Calling accounts once here select Receive Incoming Calls.

Known supported devices¶

• Samsung Galaxy Nexus - 4.2.2
• Orange San Francisco II - 2.3.5
• HTC One M9 - 5.1
• Samsung Galaxy S4 - 4.3
• Nexus 6 - 5.1.1

Instructions¶

1. Download the application from the Play Store or iTunes.
2. Once installed, go to Config -> Accounts -> Add account -> SIP
3. Fill in the following details:

• Account name: Clearwater
• Host: the root domain of your Clearwater deployment
• Username: your username, e.g. 6505551234
• Password: password for this account
• Authentication user: <username>@<server> e.g. 6505551234@my-clearwater.com

4. Hit Save
5. If your account was successfully enabled you should see a green tick notification
6. Go back to the main Config menu and select Codecs
7. Unselect everything except uLaw and aLaw.
8. Hit Save
9. You are now ready to make calls.

Known supported devices¶

• Samsung Galaxy S3
• Samsung Galaxy S2
• Samsung Galaxy S4
• Nexus One
• Huawei G510
• Huawei G610
• Qmobile A8
• HTC 1X
• iPhone 4
• iPhone 4S

Known unsupported devices¶

• Galaxy Nexus (no media)

Quick Upgrade¶

The quickest way to upgrade is to simply upgrade all your nodes in any order. There is no need to wait for one node to finish upgrading before moving onto the next.

This is recommended if you:

• Do not have a fault-tolerant deployment (e.g. the All-in-One image, or a deployment with one node of each type) OR
• Do not need to provide uninterrupted service to users of your system.

Note that during a quick upgrade your users may be unable to register, or make and receive calls. If you do have a fault-tolerant deployment and need uninterrupted service, see “Seamless Upgrade” below.

Seamless Upgrade¶

If your deployment contains at least two nodes of each type (excluding Ellis) it is possible to perform a seamless upgrade that does not result in any loss of service.

To achieve this the nodes must be upgraded one at a time and in a specific order: first all Ralf nodes (if present), then Homestead, Homer, Sprout, Memento (if present), Gemini (if present), Bono, and finally Ellis.

For example if your deployment has two Homesteads, two Homers, two Sprouts, two Bonos, and one Ellis, you should upgrade them in the following order:

• Homestead-1
• Homestead-2
• Homer-1
• Homer-2
• Sprout-1
• Sprout-2
• Bono-1
• Bono-2
• Ellis

Modifying Settings in /etc/clearwater/shared_config¶

This will have a service impact of up to five minutes.

Settings in /etc/clearwater/shared_config can be safely changed without entirely recreating the system. The one exception to this is the home_domain; if you want to change this go to the “Starting from scratch” section instead.

To change one of these settings, if you are using Clearwater’s automatic configuration sharing functionality:

• Edit /etc/clearwater/shared_config on one node and change to the new value.
• Run /usr/share/clearwater/clearwater-config-manager/scripts/upload_shared_config to upload the new config to etcd.

If you are not using automatic clustering, do the following on each node:

• Edit /etc/clearwater/shared_config and change the setting to the new value.
• Run sudo service clearwater-infrastructure restart to ensure that the configuration changes are applied consistently across the node.
• Restart the individual component by running the appropriate command below. This will stop the service and allow monit to restart it.
  • Sprout - sudo service sprout quiesce
  • Bono - sudo service bono quiesce
  • Homestead - sudo service homestead stop && sudo service homestead-prov stop
  • Homer - sudo service homer stop
  • Ralf -sudo service ralf stop
  • Ellis - sudo service ellis stop
  • Memento - sudo service memento stop

Modifying Sprout JSON Configuration¶

This configuration can be freely modified without impacting service.

Some of the more complex sprout-specific configuration is stored in JSON files

• /etc/clearwater/s-cscf.json - contains information to allow the Sprout I-CSCF to select an appropriate S-CSCF to handle some requests.
• /etc/clearwater/bgcf.json - contains routing rules for the Sprout BGCF.
• /etc/clearwater/enum.json - contains ENUM rules when using file-based ENUM instead of an external ENUM server.

To change one of these files, if you are using Clearwater’s automatic configuration sharing functionality:

• Edit the file on one of your sprout nodes.
• Run one of sudo /usr/share/clearwater/clearwater-config-manager/scripts/upload_{scscf|bgcf|enum}_json depending on which file you modified.
• The change will be automatically propagated around the deployment and will start being used.

If you are not using automatic clustering do the following on each node:

• Make the necessary changes to the file.
• Run sudo service sprout reload to make sprout pick up the changes.

Starting from scratch¶

This will have a service impact of up to half an hour.

If other settings (such as the Clearwater home domain) are being changed, we recommend that users delete their old deployment and create a new one from scratch, either with Chef or manually. This ensures that the new settings are consistently applied.

Local Settings¶

This section describes settings that are specific to a single node and are not applicable to any other nodes in the deployment. They are entered early on in the node’s life and are not normally changed. These options should be set in /etc/clearwater/local_config. Once this file has been created it is highly recommended that you do not change it unless instructed to do so. If you find yourself needing to change these settings, you should destroy and recreate then node instead.

• local_ip - this should be set to an IP address which is configured on an interface on this system, and can communicate on an internal network with other Clearwater nodes and IMS core components like the HSS.
• public_ip - this should be set to an IP address accessible to external clients (SIP UEs for Bono, web browsers for Ellis). It does not need to be configured on a local interface on the system - for example, in a cloud environment which puts instances behind a NAT.
• public_hostname - this should be set to a hostname which resolves to public_ip, and will communicate with only this node (i.e. not be round-robined to other nodes). It can be set to public_ip if necessary.
• node_idx - an index number used to distinguish this node from others of the same type in the cluster (for example, sprout-1 and sprout-2). Optional.
• etcd_cluster - this is a comma separated list of IP addresses, for example etcd_cluster=10.0.0.1,10.0.0.2. It should be set on one of two ways:
• If the node is forming a new deployment, it should contain the IP addresses of all the nodes that are forming the new deployment (including this node).
• If the node is joining an existing deployment, it should contain the IP addresses of all the nodes that are currently in the deployment.
• etcd_cluster_key - this is the name of the etcd datastore clusters that this node should join. It defaults to the function of the node (e.g. a Homestead node defaults to using ‘homestead’ as its etcd datastore cluster name when it joins the Cassandra cluster). This must be set explicitly on nodes that colocate function.

Core options¶

This section describes options for the basic configuration of a Clearwater deployment - such as the hostnames of the six node types and external services such as email servers or the Home Subscriber Server. These options should be set in the /etc/clearwater/shared_config file (in the format name=value, e.g. home_domain=example.com).

• home_domain - this is the main SIP domain of the deployment, and determines which SIP URIs Clearwater will treat as local. It will usually be a hostname resolving to all the P-CSCFs (e.g. the Bono nodes). Other domains can be specified through additional_home_domains, but Clearwater will treat this one as the default (for example, when handling tel: URIs).
• sprout_hostname - a hostname that resolves by DNS round-robin to all Sprout nodes in the cluster.
• bono_hostname - a hostname that resolves by DNS round-robin to all Bono nodes in the cluster.
• hs_hostname - a hostname that resolves by DNS round-robin to all Homesteads in the cluster. Should include the HTTP port (usually 8888). This is also used (without the port) as the Origin-Realm of the Diameter messages Homestead sends.
• hs_provisioning_hostname - a hostname that resolves by DNS round-robin to all Homesteads in the cluster. Should include the HTTP provisioning port (usually 8889). Not needed when using an external HSS.
• ralf_hostname - a hostname that resolves by DNS round-robin to all Ralf nodes in the cluster. Should include the port (usually 9888). This is also used (without the port) as the Origin-Realm of the Diameter messages Ralf sends. Optional if no Ralf nodes exist.
• cdf_identity - a Diameter identity that represents the address of an online Charging Function. Subscribers provisioned through Ellis will have this set as their Primary Charging Collection Function on P-Charging-Function-Addresses headers on responses to their successful REGISTERs, and Bono will add similarly in originating requests.
• xdms_hostname - a hostname that resolves by DNS round-robin to all Homer nodes in the cluster. Should include the port (usually 7888).
• hss_realm - this sets the Destination-Realm of your external HSS. When this field is set, Homestead will then attempt to set up multiple Diameter connections using an SRV lookup on this realm.
• hss_hostname - this sets the Destination-Host of your external HSS, if you have one. Homestead will also try and establish a Diameter connection to this host (on port 3868) if no SRV-discovered peers exist.
• signup_key - this sets the password which Ellis will require before allowing self-sign-up.
• turn_workaround - if your STUN/TURN clients are not able to authenticate properly (for example, because they can’t send the @ sign), this specifies an additional password which will authenticate clients even without a correct username.
• smtp_smarthost - Ellis allows password recovery by email. This sets the SMTP server used to send those emails.
• smtp_username - Ellis allows password recovery by email. This sets the username used to log in to the SMTP server.
• smtp_password - Ellis allows password recovery by email. This sets the password used to log in to the SMTP server.
• email_recovery_sender - Ellis allows password recovery by email. This sets the email address those emails are sent from.
• ellis_api_key - sets a key which can be used to authenticate automated requests to Ellis, by setting it as the value of the X-NGV-API header. This is used to expire demo users regularly.
• ellis_hostname - a hostname that resolves to Ellis, if you don’t want to use ellis.home_domain. This should match Ellis’s SSL certificate, if you are using one.
• memento_hostname - a hostname that resolves by DNS round-robin to all Mementos in the cluster (the default is memento.<home_domain>). This should match Memento’s SSL certificate, if you are using one.

Advanced options¶

This section describes optional configuration options, particularly for ensuring conformance with other IMS devices such as HSSes, ENUM servers, application servers with strict requirements on Record-Route headers, and non-Clearwater I-CSCFs. These options should be set in the /etc/clearwater/shared_config file (in the format name=value, e.g. icscf=5052).

• icscf - the port which Sprout nodes are providing I-CSCF service on. If not set, Sprout will only provide S-CSCF function.

• scscf - the port which Sprout nodes are providing S-CSCF service on. If this not set but icscf is, Sprout will only provide I-CSCF function. If neither is set, this will default to 5054 and Sprout will only provide S-CSCF function.

• homestead_provisioning_port - the HTTP port the Homestead provisioning interface listens on. Defaults to 8889. Not needed when using an external HSS.

• sas_server - the IP address or hostname of your Metaswitch Service Assurance Server for call logging and troubleshooting. Optional.

• reg_max_expires - determines the maximum expires= parameter Sprout will set on Contact headers at registrations, and therefore the amount of time before a UE has to re-register - must be less than 2^31 ms (approximately 25 days). Default is 300 (seconds).

• sub_max_expires - determines the maximum Expires header Sprout will set in subscription responses, and therefore the amount of time before a UE has to re-subscribe - must be less than 2^31 ms (approximately 25 days).

• upstream_hostname - the I-CSCF which Bono should pass requests to. Defaults to the sprout_hostname.

• upstream_port - the port on the I-CSCF which Bono should pass requests to. Defaults to 5052. If set to 0, Bono will use SRV resolution of the upstream_hostname hostname to determine a target for traffic.

• sprout_rr_level - this determines how the Sprout S-CSCF adds Record-Route headers. Possible values are:

  • pcscf - a Record-Route header is only added just after requests come from or go to a P-CSCF - that is, at the start of originating handling and the end of terminating handling
  • pcscf,icscf - a Record-Route header is added just after requests come from or go to a P-CSCF or I-CSCF - that is, at the start and end of originating handling and the start and end of terminating handling
  • pcscf,icscf,as - a Record-Route header is added after requests come from or go to a P-CSCF, I-CSCF or application server - that is, at the start and end of originating handling, the start and end of terminating handling, and between each application server invoked

• force_hss_peer - when set to an IP address or hostname, Homestead will create a connection to the HSS using this value, but will still use the hss_realm and hss_hostname settings for the Destination-Host and Destination-Realm Diameter AVPs. This is useful when your HSS’s Diameter configuration does not match the DNS records.

• hss_mar_lowercase_unknown - some Home Subscriber Servers (particularly old releases of OpenIMSCore HSS) expect the string ‘unknown’ rather than ‘Unknown’ in Multimedia-Auth-Requests when Clearwater cannot tell what authentication type is expected. Setting this option to ‘Y’ will make Homestead send requests in this format.

• hss_mar_force_digest - if Clearwater cannot tell what authentication type a subscriber is trying to use, this forces it to assume ‘SIP Digest’ and report that in the Multimedia-Auth-Request, rather than ‘Unknown’.

• hss_mar_force_aka - if Clearwater cannot tell what authentication type a subscriber is trying to use, this forces it to assume ‘Digest-AKA-v1’ and report that in the Multimedia-Auth-Request, rather than ‘Unknown’.

• force_third_party_reg_body - if the HSS does not allow the IncludeRegisterRequest/IncludeRegisterResponse fields (which were added in 3GPP Rel 9) to be configured, setting force_third_party_reg_body=Y makes Clearwater behave as though they had been sent, allowing interop with application servers that need them.

• enforce_user_phone - by default, Clearwater will do an ENUM lookup on any SIP URI that looks like a phone number, due to client support for user-phone not being widespread. When this option is set to ‘Y’, Clearwater will only do ENUM lookups for URIs which have the user=phone parameter.

• enforce_global_only_lookups - by default, Clearwater will do ENUM lookups for SIP and Tel URIs containing global and local numbers (as defined in RFC 3966). When this option is set to ‘Y’, Clearwater will only do ENUM lookups for SIP and Tel URIs that contain global numbers.

• hs_listen_port - the Diameter port which Homestead listens on. Defaults to 3868.

• ralf_listen_port - the Diameter port which Ralf listens on. Defaults to 3869 to avoid clashes when colocated with Homestead.

• alias_list - this defines additional hostnames and IP addresses which Sprout or Bono will treat as local for the purposes of SIP routing (e.g. when removing Route headers).

• default_session_expires - determines the Session-Expires value which Sprout will add to INVITEs, to force UEs to send keepalive messages during calls so they can be tracked for billing purposes. This cannot be set to a value less than 90 seconds, as specified in RFC 4028, section 4.

• max_session_expires - determines the maximum Session-Expires/Min-SE value which Sprout will accept in requests. This cannot be set to a value less than 90 seconds, as specified in RFC 4028, sections 4 and 5.

• enum_server - a comma-separated list of DNS servers which can handle ENUM queries.

• enum_suffix - determines the DNS suffix used for ENUM requests (after the digits of the number). Defaults to “e164.arpa”

• enum_file - if set (to a file path), and if enum_server is not set, Sprout will use this local JSON file for ENUM lookups rather than a DNS server. An example file is at http://clearwater.readthedocs.org/en/stable/ENUM/index.html#deciding-on-enum-rules.

• icscf_uri - the SIP address of the external I-CSCF integrated with your Sprout node (if you have one).

• scscf_uri - the SIP address of the Sprout S-CSCF. This defaults to sip:$sprout_hostname:$scscf;transport=TCP - this includes a specific port, so if you need NAPTR/SRV resolution, it must be changed to not include the port.

• additional_home_domains - this option defines a set of home domains which Sprout and Bono will regard as locally hosted (i.e. allowing users to register, not routing calls via an external trunk). It is a comma-separated list.

• billing_realm - this sets the Destination-Realm on Diameter messages to your external CDR. CDR connections are not based on this but on configuration at the P-CSCF (which sets the P-Charging-Function-Addresses header).

• diameter_timeout_ms - determines the number of milliseconds Homestead will wait for a response from the HSS before failing a request. Defaults to 200.

• max_peers - determines the maximum number of Diameter peers which Ralf or Homestead can have open connections to at the same time.

• num_http_threads (Ralf/Memento) - determines the number of threads that will be used to process HTTP requests. For Memento this defaults to the number of CPU cores on the system. For Ralf it defaults to 50 times the number of CPU cores (Memento and Ralf use different threading models, hence the different defaults). Note that for Homestead, this can only be set in /etc/clearwater/user_settings.

• num_http_worker_threads - determines the number of threads that will be used to process HTTP requests once they have been parsed. Only used by Memento.

• ralf_diameteridentity - determines the Origin-Host that will be set on the Diameter messages Ralf sends. Defaults to public_hostname (with some formatting changes if public_hostname is an IPv6 address).

• hs_diameteridentity - determines the Origin-Host that will be set on the Diameter messages Homestead sends. Defaults to public_hostname (with some formatting changes if public_hostname is an IPv6 address).

• gemini_enabled - When this field is set to ‘Y’, then the node (either a Sprout or a standalone application server) will include a Gemini AS.

• memento_enabled - When this field is set to ‘Y’, then the node (either a Sprout or a standalone application server) will include a Memento AS.

• max_call_list_length - determines the maximum number of complete calls a subscriber can have in the call list store. This defaults to no limit. This is only relevant if the node includes a Memento AS.

• call_list_store_ttl - determines how long each call list fragment should be kept in the call list store. This defaults to 604800 seconds (1 week). This is only relevant if the node includes a Memento AS.

• memento_disk_limit - determines the maximum size that the call lists database may occupy. This defaults to 20% of disk space. This is only relevant if the node includes a Memento AS. Can be specified in Bytes, Kilobytes, Megabytes, Gigabytes, or a percentage of the available disk. For example:

  memento_disk_limit=10240 # Bytes
  memento_disk_limit=100k  # Kilobytes
  memento_disk_limit=100M  # Megabytes
  memento_disk_limit=100G  # Gigabytes
  memento_disk_limit=45%   # Percentage of available disk
  

• memento_threads - determines the number of threads dedicated to adding call list fragments to the call list store. This defaults to 25 threads. This is only relevant if the node includes a Memento AS.

• memento_notify_url - If set to an HTTP URL, memento will make a POST request to this URL whenever a subscriber’s call list changes. The body of the POST request will be a JSON document with the subscriber’s IMPU in a field named impu. This is only relevant if the node includes a Memento AS. If empty, no notifications will be sent. Defaults to empty.

• signaling_dns_server - a comma-separated list of DNS servers for non-ENUM queries. Defaults to 127.0.0.1 (i.e. uses dnsmasq)

• target_latency_us - Target latency (in microsecs) for requests above which throttling applies. This defaults to 100000 microsecs

• max_tokens - Maximum number of tokens allowed in the token bucket (used by the throttling code). This defaults to 20 tokens

• init_token_rate - Initial token refill rate of tokens in the token bucket (used by the throttling code). This defaults to 250 tokens per second per core

• min_token_rate - Minimum token refill rate of tokens in the token bucket (used by the throttling code). This defaults to 10.0

• override_npdi - Whether the I-CSCF, S-CSCF and BGCF should check for number portability data on requests that already have the ‘npdi’ indicator. This defaults to false

• exception_max_ttl - determines the maximum time before a process exits if it crashes. This defaults to 600 seconds

• check_destination_host - determines whether the node checks the Destination-Host on a Diameter request when deciding whether it should process the request. This defaults to true.

• astaire_cpu_limit_percentage - the maximum percentage of total CPU that Astaire is allowed to consume when resyncing memcached data (as part of a scale-up, scale-down, or following a memcached failure). Note that this only limits the CPU usage of the Astaire process, and does not affect memcached’s CPU usage. Must be an integer. Defaults to 5.

• sip_blacklist_duration - the time in seconds for which SIP peers are blacklisted when they are unresponsive (defaults to 30 seconds).

• http_blacklist_duration - the time in seconds for which HTTP peers are blacklisted when they are unresponsive (defaults to 30 seconds).

• diameter_blacklist_duration - the time in seconds for which Diameter peers are blacklisted when they are unresponsive (defaults to 30 seconds).

• snmp_ip - the IP address to send alarms to (defaults to being unset). If this is set then Sprout, Ralf, Homestead and Chronos will send alarms - more details on the alarms are here. This can be a single IP address, or a comma-separated list of IP addresses.

• impu_cache_ttl - the number of seconds for which Homestead will cache the SIP Digest from a Multimedia-Auth-Request. Defaults to 0, as Sprout does enough caching to ensure that it can handle an authenticated REGISTER after a challenge, and subsequent challenges should be rare.

• sip_tcp_connect_timeout - the time in milliseconds to wait for a SIP TCP connection to be established (defaults to 2000 milliseconds).

• sip_tcp_send_timeout - the time in milliseconds to wait for sent data to be acknowledgered at the TCP level on a SIP TCP connection (defaults to 2000 milliseconds).

• session_continued_timeout_ms - if an Application Server with default handling of ‘continue session’ is unresponsive, this is the time that Sprout will wait (in milliseconds) before bypassing the AS and moving onto the next AS in the chain (defaults to 2000 milliseconds).

• session_terminated_timeout_ms - if an Application Server with default handling of ‘terminate session’ is unresponsive, this is the time that Sprout will wait (in milliseconds) before terminating the session (defaults to 4000 milliseconds).

• pbxes - a comma separated list of IP address that Bono considers to be PBXes that are incapable of registering. Non-REGISTER requests from these addresses are passed upstream to Sprout with a Proxy-Authorization header. It is strongly recommended that Sprout’s non_register_authentication option is set to if_proxy_authorization_present so that the request will be challenged. Bono also permits requests to these addresses from the core to pass through it.

• pbx_service_route - the SIP URI to which Bono routes originating calls from non-registering PBXes (which are identified by the pbxes option). This is used to route requests directly to the S-CSCF rather than going via an I-CSCF (which could change the route header and prevent the S-CSCF from processing the request properly). This URI is used verbatim and should almost always include the lr, orig, and auto-reg parameters. If this option is not specified, the requests are routed to the address specified by the upstream_hostname and upstream_port options.

  • e.g. sip:sprout.example.com:5054;transport=tcp;lr;orig;auto-reg

• non_register_authentication - controls when Sprout will challenge a non-REGISTER request using SIP Proxy-Authentication. Possible values are never (meaning Sprout will never challenge) or if_proxy_authorization_present (meaning Sprout will only challenge requests that have a Proxy-Authorization header).

• ralf_threads - used on Sprout nodes, this determines how many worker threads should be started to do Ralf request processing (defaults to 25).

Experimental options¶

This section describes optional configuration options which may be useful, but are not heavily-used or well-tested by the main Clearwater development team. These options should be set in the /etc/clearwater/shared_config file (in the format name=value, e.g. cassandra_hostname=db.example.com).

• cassandra_hostname - if using an external Cassandra cluster (which is a fairly uncommon configuration), a hostname that resolves to one or more Cassandra nodes.
• ralf_secure_listen_port - this determines the port Ralf listens on for TLS-secured Diameter connections.
• hs_secure_listen_port - this determines the port Homestead listens on for TLS-secured Diameter connections.
• ellis_cookie_key - an arbitrary string that enables Ellis nodes to determine whether they should be in the same cluster. This function is not presently used.
• stateless_proxies - a comma separated list of domain names that are treated as SIP stateless proxies. Stateless proxies are not blacklisted if a SIP transaction sent to them times out. This field should reflect how the servers are identified in SIP. For example if a cluster of nodes is identified by the name ‘cluster.example.com’, the option should be set to ‘cluster.example.com’ instead of the hostnames or IP addresses of individual servers.
• hss_reregistration_time - determines how many seconds should pass before Homestead sends a Server-Assignment-Request with type RE_REGISTRATION to the HSS. (On first registration, it will always send a SAR with type REGISTRATION). This determines a minimum value - after this many seconds have passed, Homestead will send the Server-Assignment-Request when the next REGISTER is received. Note that Homestead invalidates its cache of the registration and iFCs after twice this many seconds have passed, so it is not safe to set this to less than half of reg_max_expires. The default value of this option is whichever is the greater of the following.
  • Half of the value of reg_max_expires.

User settings¶

This section describes settings that may vary between systems in the same deployment, such as log level (which may be increased on certain machines to track down specific issues) and performance settings (which may vary if some servers in your deployment are more powerful than others). These settings are set in /etc/clearwater/user_settings, not /etc/clearwater/shared_config (in the format name=value, e.g. log_level=5).

• log_level - determines how verbose Clearwater’s logging is, from 1 (error logs only) to 5 (debug-level logs). Defaults to 2.
• log_directory - determines which folder the logs are created in. This folder must exist, and be owned by the service. Defaults to /var/log/ (this folder is created and has the correct permissions set for it by the install scripts of the service).
• max_log_directory_size - determines the maximum size of each Clearwater process’s log_directory in bytes. Defaults to 1GB. If you are co-locating multiple Clearwater processes, you’ll need to reduce this value proportionally.
• num_pjsip_threads - determines how many PJSIP transport-layer threads should run at once. Defaults to 1, and it may be dangerous to change this as it is not necessarily thread-safe.
• num_worker_threads - for Sprout and Bono nodes, determines how many worker threads should be started to do SIP/IMS processing. Defaults to 50 times the number of CPU cores on the system.
• upstream_connections - determines the maximum number of TCP connections which Bono will open to the I-CSCF(s). Defaults to 50.
• upstream_recycle_connections - the average number of seconds before Bono will destroy and re-create a connection to Sprout. A higher value means slightly less work, but means that DNS changes will not take effect as quickly (as new Sprout nodes added to DNS will only start to receive messages when Bono creates a new connection and does a fresh DNS lookup).
• authentication - by default, Clearwater performs authentication challenges (SIP Digest or IMS AKA depending on HSS configuration). When this is set to ‘Y’, it simply accepts all REGISTERs - obviously this is very insecure and should not be used in production.
• num_http_threads (Homestead) - determines the number of HTTP worker threads that will be used to process requests. Defaults to 50 times the number of CPU cores on the system.

Other configuration options¶

There is further documentation for Chronos configuration here and Homer/Homestead-prov configuration here.

Upgrade the Deployment¶

Upgrade to the latest stable Clearwater release. You will also need to update your firewall settings to support the new clearwater management packages; open port 2380 and 4000 between every node (see here for the complete list).

Verify Configuration Files¶

Do the following on each node in turn:

1. Run /usr/share/clearwater/infrastructure/migration-utils/configlint.py. This examines the existing /etc/clearwater/config file and checks that the migration scripts can handle all the settings defined in it.
2. If configlint.py produces a warning about a config option, this can mean one of two things:
   • The config option is invalid (for example, because there is a typo, or this option has been retired). Check the configuration options reference for a list of valid options.
   • The config option is valid, but the migration script doesn’t recognise the option and won’t automatically migrate it. In this case, you will need to make a note of this config option now, and add it back in after the rest of the migration has run. (A later step in this process covers that.)

Once you have checked your configuration file and taken a note of any unrecognised settings, continue with the next step.

Prepare Local Configuration Files¶

Do the following on each node in turn:

1. Run sudo /usr/share/clearwater/infrastructure/migration-utils/migrate_local_config /etc/clearwater/config. This examines the existing /etc/clearwater/config file and produces a new /etc/clearwater/local_config which contains the settings only relevant to this node. Check that this file looks sensible.
2. Edit /etc/clearwater/local_config to add a line etcd_cluster="<NodeIPs>" where NodeIPs is a comma separated list of the private IP addresses of nodes in the deployment. For example if your deployment contained nodes with IP addresses of 10.0.0.1 to 10.0.0.6, NodeIPs would be 10.0.0.1,10.0.0.2,10.0.0.3,10.0.0.4,10.0.0.5,10.0.0.6. If your deployment was GR, this should include the IP addresses of nodes in both sites.
3. If your deployment was geographically redundant, you should choose arbitrary names for each site (e.g. ‘site1’ and ‘site2’), and set the local_site_name and remote_site_name settings in /etc/clearwater/local_config accordingly. For example, if the node is in ‘site1’, you should have local_site_name=site1 and remote_site_name=site2.
4. If the node is a Sprout or Ralf node, run sudo /usr/share/clearwater/bin/chronos_configuration_split.py. This examines the existing /etc/chronos/chronos.conf file and extracts the clustering settings into a new file called /etc/chronos/chronos_cluster.conf. Check each of these files by hand to make sure they look sensible. If the chronos_cluster.conf file already exists, then the script will exit with a warning. In this case, please check the configuration files by hand, and either delete the chronos_cluster.conf file and re-run the script, or manually split the configuration yourself. Details of the expected configuration are here.
5. Run sudo touch /etc/clearwater/no_cluster_manager on all nodes. This temporarily disables the cluster manager (which is installed in the next step) so that you can program it with the current deployment topology.

Prepare Shared Configuration Files¶

This step merges the config from all the nodes in your deployment into a single config file that will be managed by etcd.

1. Log onto to one of the nodes in the deployment and create a temporary directory (e.g. ~/config-migration). Copy the /etc/clearwater/config file from each node in the deployment to this directory. Give each of these a name of the form <node>_orig_config, e.g. sprout-1_orig_config.
2. Run sudo /usr/share/clearwater/infrastructure/migration-utils/migrate_shared_config, passing it all the config files in the temporary directory
   • sudo /usr/share/clearwater/infrastructure/migration-utils/migrate_shared_config *_orig_config
3. This will produce the file /etc/clearwater/shared_config. Check the contents of this file looks sensible. If running configlint.py indicated that any configuration would not be automatically migrated, add that configuration to /etc/clearwater/shared_config now.
4. Copy this file to /etc/clearwater/shared_config on each node in the deployment.
5. On each node in turn:
   • Run sudo /usr/share/clearwater/infrastructure/migration-utils/switch_to_migrated_config
   • Run sudo service clearwater-infrastructure restart to regenerate any dependant configuration files
   • Restart the Clearwater services on the node.

Install Clustering and Configuration Management Services¶

On each node run sudo apt-get install clearwater-management.

Upload the Current Cluster Settings¶

Now you need to tell the cluster manager about the current topology of the various database clusters that exist in a Clearwater deployment. For each of the nodes types listed below, log onto one of the nodes of that type and run the specified commands.

/usr/share/clearwater/clearwater-cluster-manager/scripts/load_from_memcached_cluster sprout
/usr/share/clearwater/clearwater-cluster-manager/scripts/load_from_chronos_cluster sprout

/usr/share/clearwater/clearwater-cluster-manager/scripts/load_from_memcached_cluster ralf
/usr/share/clearwater/clearwater-cluster-manager/scripts/load_from_chronos_cluster ralf

/usr/share/clearwater/clearwater-cluster-manager/scripts/load_from_cassandra_cluster homestead

/usr/share/clearwater/clearwater-cluster-manager/scripts/load_from_cassandra_cluster homer

/usr/share/clearwater/clearwater-cluster-manager/scripts/load_from_memcached_cluster memento
/usr/share/clearwater/clearwater-cluster-manager/scripts/load_from_cassandra_cluster memento

Upload the Shared Configuration¶

Run the following commands on one of your Sprout nodes. This will upload the configuration that is shared across the deployment to etcd. If you add any new nodes to the deployment they will automatically learn this configuration from etcd.

• sudo /usr/share/clearwater/clearwater-config-manager/scripts/upload_shared_config
• sudo /usr/share/clearwater/clearwater-config-manager/scripts/upload_scscf_json
• sudo /usr/share/clearwater/clearwater-config-manager/scripts/upload_bgcf_json
• sudo /usr/share/clearwater/clearwater-config-manager/scripts/upload_enum_json

Tidy Up¶

The final step is to re-enable the cluster manager by running the following commands:

sudo rm /etc/clearwater/no_cluster_manager

Network configuration¶

Each application server must be able to communicate with Sprout and Bono in both directions.

• The application server must be able to contact all sprout and bono nodes on the trusted SIP ports, 5054 and 5058 (respectively, for both TCP and UDP).
• All sprout and bono nodes must be able to contact the application server on the port and protocol configured in your subscribers’ iFCs (typically TCP and UDP port 5060).
• The application server must also be able to exchange SIP messages (in both directions) with any other application server that may appear in the signaling path of any call going through it.
• Since the application server is in the trusted zone, it should not be accessible to untrusted SIP entities.

If Clearwater and your application server are both in the Amazon EC2 cloud, you can achieve all of these by placing the application server in the <deployment>-sprout security group.

Application server configuration¶

No special application server configuration is required.

• Your application server should be prepared to accept SIP messages from any of your deployment’s bono or sprout nodes, and from any SIP entity that may appear in the signaling path of any call going through it.
• If your application server needs to spontaneously originate calls, it should do this via Sprout’s trusted interface: sprout.<deployment-domain>:5054 over either TCP or UDP.

The headers set by Clearwater are described in the Application Server Guide.

iFC configuration¶

To configure a subscriber to invoke your application server, you must configure the appropriate iFCs for that subscriber. You can do this via the Ellis UI, the Homestead API, or if you are using an HSS, directly in the HSS.

{
"MMTEL" : "<InitialFilterCriteria><Priority>0</Priority><TriggerPoint><ConditionTypeCNF></ConditionTypeCNF><SPT><ConditionNegated>0</ConditionNegated><Group>0</Group><Method>INVITE</Method><Extension></Extension></SPT></TriggerPoint><ApplicationServer><ServerName>sip:mmtel.example.com</ServerName><DefaultHandling>0</DefaultHandling></ApplicationServer></InitialFilterCriteria>",
"Voicemail" : "<InitialFilterCriteria><Priority>1</Priority><TriggerPoint><ConditionTypeCNF></ConditionTypeCNF><SPT><ConditionNegated>0</ConditionNegated><Group>0</Group><Method>INVITE</Method><Extension></Extension></SPT></TriggerPoint><ApplicationServer><ServerName>sip:vm.example.com</ServerName><DefaultHandling>0</DefaultHandling></ApplicationServer></InitialFilterCriteria>"
}

user=sip:6505550269@example.com
as_hostnames="as1.example.com mmtel.example.com" # the list of zero or more application servers to invoke - don't forget mmtel
hs_hostname=hs.example.com:8888

curl http://$hs_hostname:8889/public/$user/service_profile

Location: /irs/<irs-uuid>/service_profiles/<service-profile-uuid>

curl http://$hs_hostname:8889/irs/<irs-uuid>/service_profiles/<service-profile-uuid>/filter_criteria

{ cat <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<ServiceProfile>
EOF
for as_hostname in $as_hostnames ; do
  cat <<EOF
  <InitialFilterCriteria>
    <Priority>1</Priority>
    <TriggerPoint>
      <ConditionTypeCNF>0</ConditionTypeCNF>
      <SPT>
        <ConditionNegated>0</ConditionNegated>
        <Group>0</Group>
        <Method>INVITE</Method>
        <Extension></Extension>
      </SPT>
    </TriggerPoint>
    <ApplicationServer>
      <ServerName>sip:$as_hostname</ServerName>
      <DefaultHandling>0</DefaultHandling>
    </ApplicationServer>
  </InitialFilterCriteria>
EOF
done
cat <<EOF
</ServiceProfile>
EOF
} | curl -X PUT http://$hs_hostname:8889/irs/<irs-uuid>/service_profiles/<service-profile-uuid>/filter_criteria --data-binary @-

How It Works¶

When Clearwater is deployed without an external HSS, all HSS data is mastered in Homestead’s own Cassandra database.

When Clearwater is deployed with an external HSS, HSS data is queried from the external HSS via its Cx/Diameter interface and is then cached in the Cassandra database.

Clearwater uses the following Cx message types.

• Multimedia-Auth - to retrieve authentication details
• Server-Assignment - to retrieve Initial Filter Criteria documents and register for change notifications
• Push-Profile - to be notified of changes to Initial Filter Criteria documents
• User-Authorization - to retrieve S-CSCF details on initial registrations
• Location-Information - to retrieve S-CSCF details on calls
• Registration-Termination - to be notified of de-registrations

How to Enable It¶

This section discusses how to enable support for an external HSS.

# HSS configuration
hss_hostname=0.0.0.0
hss_port=3868

# HSS configuration
hss_hostname=<address of your HSS>
hss_realm=<realm your HSS is located in>
hss_port=<port of your HSS's Cx interface>

Restrictions¶

• Since Homestead uses the Cx/Diameter interface to the HSS, and this interface is read-only, the Homestead API is read-only when external HSS integration is enabled.
• Since Homestead’s API is read-only, this means that Ellis can’t be used alongside a deployment using an external HSS. Provisioning and subscriber management must be performed via the HSS’s own management interface.
• Clearwater currently only supports SIP digest or AKA authentication; details for other authentication methods are silently ignored from Multimedia-Auth responses. Other authentication methods may be added in future.
• Homestead currently assumes that private user IDs are formed by removing the sip: prefix from the public user ID. This restriction may be relaxed in future.
• While Homestead caches positive results from the external HSS, it does not currently cache negative results (e.g. for non-existent users). Repeated requests for a non-existent user will increase the load on the external HSS. This restriction may be relaxed in future.

Installation with Chef¶

If you have a deployment environment created by following the automated install instructions, then you can create a HSS by running knife box create -E <env> openimscorehss. You should then follow the configuration instructions below.

Installing OpenIMSCore HSS manually¶

To install OpenIMSCore HSS,

1. Follow the “Adding this PPA to your system” steps at https://launchpad.net/~rkd-u/+archive/ubuntu/fhoss
2. Run sudo apt-get update; sudo apt-get install openimscore-fhoss
3. Answer the installation questions appropriately, including providing the IMS home domain, your local IP, and the MySQL root password (which you will have to provide both when installing MySQL and when intalling the HSS).

Configuration¶

How it works¶

When Sprout or Bono have handled an item of work for a given subscriber, they generate a record of that work and transmit it to the Ralf cluster for billing to the CDF. This record will be used by Ralf to generate an Rf billing ACR (Accounting-Request) message.

To determine which CDF to send the ACR to, the node acting as the P-CSCF/IBCF is responsible for adding a P-Charging-Function-Addresses header to all SIP messages it proxies into the core. This header contains a prioritised list of CDFs to send ACRs to.

Bono supports adding this header when acting as either a P-CSCF or IBCF and configuring the contents of this header is the only requirement for enabling Rf billing in your deployment.

How to enable it¶

This section discusses how to enable Rf billing to a given CDF.

billing_realm=<DIAMETER billing realm>

cdf_identity=<CDF DIAMETER Identity>

Restrictions¶

The very first release of Ralf, from the Counter-Strike release of Project Clearwater, does not generate Rf billing messages since the related changes to Sprout and Bono (to report billable events) were not enabled. This version was released to allow systems integrators to get a head start on spinning up and configuring Ralf nodes rather than having to wait for the next release.

All nodes¶

All nodes need to allow the following ICMP messages:

Echo Request
Echo Reply
Destination Unreachable

They also need the following ports open to the world (0.0.0.0/0):

TCP/22 for SSH access

If your deployment uses SNMP monitoring software (cacti for example), each node will also have to open appropriate ports for this service.

• SNMP

  UDP/161-162
  

If your deployment uses our automatic clustering and configuration sharing feature, open the following ports between every node

• etcd

  TCP/2380
  TCP/4000
  

All-in-one¶

All-in-one nodes need the following ports opened to the world

• Web UI

  TCP/80
  TCP/443
  

• STUN signaling:

  TCP/3478
  UDP/3478
  

• SIP signaling:

  TCP/5060
  UDP/5060
  TCP/5062
  

• RTP forwarding:

  UDP/32768-65535
  

Ellis¶

The Ellis node needs the following ports opened to the world:

• Web UI

  TCP/80
  TCP/443
  

Bono¶

The Bono nodes need the following ports opened to the world:

• STUN signaling:

  TCP/3478
  UDP/3478
  

• SIP signaling:

  TCP/5060
  UDP/5060
  TCP/5062
  

• RTP forwarding:

  UDP/32768-65535
  

They also need the following ports open to all other Bono nodes and to all the Sprout nodes:

• Internal SIP signaling:

  TCP/5058
  

Sprout¶

The Sprout nodes need the following ports open to all Bono nodes:

• Internal SIP signaling:

  TCP/5054
  TCP/5052 (if I-CSCF function is enabled)
  

They also need the following ports opened to all other Sprout nodes:

• Shared registration store:

  For releases using a memcached store - that is all releases up to release 28 (Lock, Stock and Two Smoking Barrels) and from release 32 (Pulp Fiction) onwards

  TCP/11211
  

• Chronos:

  TCP/7253
  

• Cassandra (if including a Memento AS):

  TCP/7000
  TCP/9160
  

They also need the following ports opened to all homestead nodes:

• Registration Termination Requests (if using an HSS):

  TCP/9888
  

They also need the following ports opened to the world:

• HTTP interface (if including a Memento AS):

  TCP/443
  

Homestead¶

The Homestead nodes need the following ports open to all the Sprout nodes and the Ellis node:

• RESTful interface:

  TCP/8888
  

They also need the following ports open to just the Ellis node:

• RESTful interface:

  TCP/8889
  

They also need the following ports opened to all other Homestead nodes:

• Cassandra:

  TCP/7000
  

They also need the following ports opened to the world:

Homer¶

The Homer nodes need the following ports open to all the Sprout nodes and the Ellis node:

• RESTful interface:

  TCP/7888
  

They also need the following ports opened to all other Homer nodes:

• Cassandra:

  TCP/7000
  

They also need the following ports opened to the world:

Ralf¶

The Ralf nodes need the following ports open to all the Sprout and Bono nodes:

• RESTful interface:

  TCP/10888
  

They also need to following ports open to all other Ralf nodes:

• Chronos:

  TCP/7253
  

• Memcached:

  TCP/11211
  

Standalone Application Servers¶

Standalone Project Clearwater application servers (e.g. Memento and Gemini) need the following ports open to all Sprout nodes:

• SIP signaling:

  TCP/5054
  

They also need the following ports open to all other standalone application servers (if they include a Memento AS):

• Cassandra:

  TCP/7000
  TCP/9160
  

They also need the following ports open to all Homestead nodes (if they include a Memento AS):

• RESTful interface:

  TCP/11888
  

They also need the following ports opened to the world (if they include a Memento AS):

• HTTP interface:

  TCP/443
  

Strategy¶

Clearwater makes heavy use of DNS to refer to its nodes. It uses it for

• identifying individual nodes, e.g. sprout-1.example.com might resolve the IP address of the first sprout node
• identifying the nodes within a cluster, e.g. sprout.example.com resolves to all the IP addresses in the cluster
• fault-tolerance
• selecting the nearest site in a multi-site deployments, using latency-based routing.

Clearwater also supports using DNS for identifying non-Clearwater nodes. In particular, it supports DNS for identifying SIP peers using NAPTR and SRV records, as described in RFC 3263.

Resiliency¶

By default, Clearwater routes all DNS requests through an instance of dnsmasq running on localhost. This round-robins requests between the servers in /etc/resolv.conf, as described in its FAQ:

By default, dnsmasq treats all the nameservers it knows about as equal: it picks the one to use using an algorithm designed to avoid nameservers which aren’t responding.

If the signaling_dns_server option is set in /etc/clearwater/shared_config (which is mandatory when using traffic separation), Clearwater will not use dnsmasq. Instead, resiliency is achieved by being able to specify up to three servers in a comma-separated list (e.g. signaling_dns_server=1.2.3.4,10.0.0.1,192.168.1.1), and Clearwater will fail over between them as follows:

• It will always query the first server in the list first
• If this returns SERVFAIL or times out (which happens after a randomised 500ms-1000ms period), it will resend the query to the second server
• If this returns SERVFAIL or times out, it will resend the query to the third server
• If all servers return SERVFAIL or time out, the DNS query will fail

Clearwater caches DNS responses for several minutes (to reduce the load on DNS servers, and the latency introduced by querying them). If a cache entry is stale, but the DNS servers return SERVFAIL or time out when Clearwater attempts to refresh it, Clearwater will continue to use the cached value until the DNS servers become responsive again. This minimises the impact of a DNS server failure on calls.

Requirements¶

Configuration¶

Clearwater can work with any DNS server that meets the requirements above. However, most of our testing has been performed with

• AWS Route 53 - see configuration instructions
• BIND - see configuration instructions.

The Clearwater nodes also need to know the identity of their DNS server. Ideally, this is done via DHCP within your virtualization infrastructure. Alternatively, you can configure it manually.

The UEs need to know the identity of the DNS server too. In a testing environment, you may be able to use DHCP or manual configuration. In a public network, you will need to register the <zone> domain name you are using and arranging for an NS record for <zone> to point to your DNS server.

zone "<zone>" IN { type master; file "/etc/bind/db.<zone>"; };

$TTL 5m ; Default TTL

; SOA, NS and A record for DNS server itself
@                 3600 IN SOA  ns admin ( 2014010800 ; Serial
                                          3600       ; Refresh
                                          3600       ; Retry
                                          3600       ; Expire
                                          300 )      ; Minimum TTL
@                 3600 IN NS   ns
ns                3600 IN A    1.0.0.1 ; IPv4 address of BIND server
ns                3600 IN AAAA 1::1    ; IPv6 address of BIND server

; bono
; ====
;
; Per-node records - not required to have both IPv4 and IPv6 records
bono-1                 IN A     2.0.0.1
bono-2                 IN A     2.0.0.2
bono-1                 IN AAAA  2::1
bono-2                 IN AAAA  2::2
;
; Cluster A and AAAA records - UEs that don't support RFC 3263 will simply
; resolve the A or AAAA records and pick randomly from this set of addresses.
@                      IN A     2.0.0.1
@                      IN A     2.0.0.2
@                      IN AAAA  2::1
@                      IN AAAA  2::2
;
; NAPTR and SRV records - these indicate a preference for TCP and then resolve
; to port 5060 on the per-node records defined above.
@                      IN NAPTR 1 1 "S" "SIP+D2T" "" _sip._tcp
@                      IN NAPTR 2 1 "S" "SIP+D2U" "" _sip._udp
_sip._tcp              IN SRV   0 0 5060 bono-1
_sip._tcp              IN SRV   0 0 5060 bono-2
_sip._udp              IN SRV   0 0 5060 bono-1
_sip._udp              IN SRV   0 0 5060 bono-2

; sprout
; ======
;
; Per-node records - not required to have both IPv4 and IPv6 records
sprout-1               IN A     3.0.0.1
sprout-2               IN A     3.0.0.2
sprout-1               IN AAAA  3::1
sprout-2               IN AAAA  3::2
;
; Cluster A and AAAA records - P-CSCFs that don't support RFC 3263 will simply
; resolve the A or AAAA records and pick randomly from this set of addresses.
sprout                 IN A     3.0.0.1
sprout                 IN A     3.0.0.2
sprout                 IN AAAA  3::1
sprout                 IN AAAA  3::2
;
; NAPTR and SRV records - these indicate TCP support only and then resolve
; to port 5054 on the per-node records defined above.
sprout                 IN NAPTR 1 1 "S" "SIP+D2T" "" _sip._tcp.sprout
_sip._tcp.sprout       IN SRV   0 0 5054 sprout-1
_sip._tcp.sprout       IN SRV   0 0 5054 sprout-2
;
; Per-node records for I-CSCF (if enabled) - not required to have both
; IPv4 and IPv6 records
sprout-3               IN A     3.0.0.3
sprout-3               IN AAAA  3::3
;
; Cluster A and AAAA records - P-CSCFs that don't support RFC 3263 will simply
; resolve the A or AAAA records and pick randomly from this set of addresses.
icscf.sprout           IN A     3.0.0.3
icscf.sprout           IN AAAA  3::3
;
; NAPTR and SRV records for I-CSCF (if enabled) - these indicate TCP
; support only and then resolve to port 5052 on the per-node records
; defined above.
icscf.sprout           IN NAPTR 1 1 "S" "SIP+D2T" "" _sip._tcp.icscf.sprout
_sip._tcp.icscf.sprout IN SRV   0 0 5052 sprout-3

; homestead
; =========
;
; Per-node records - not required to have both IPv4 and IPv6 records
homestead-1            IN A     4.0.0.1
homestead-2            IN A     4.0.0.2
homestead-1            IN AAAA  4::1
homestead-2            IN AAAA  4::2
;
; Cluster A and AAAA records - sprout picks randomly from these.
hs                     IN A     4.0.0.1
hs                     IN A     4.0.0.2
hs                     IN AAAA  4::1
hs                     IN AAAA  4::2
;
; (No need for NAPTR or SRV records as homestead doesn't handle SIP traffic.)

; homer
; =====
;
; Per-node records - not required to have both IPv4 and IPv6 records
homer-1                IN A     5.0.0.1
homer-2                IN A     5.0.0.2
homer-1                IN AAAA  5::1
homer-2                IN AAAA  5::2
;
; Cluster A and AAAA records - sprout picks randomly from these.
homer                  IN A     5.0.0.1
homer                  IN A     5.0.0.2
homer                  IN AAAA  5::1
homer                  IN AAAA  5::2
;
; (No need for NAPTR or SRV records as homer doesn't handle SIP traffic.)

; ralf
; =====
;
; Per-node records - not required to have both IPv4 and IPv6 records
ralf-1                IN A     6.0.0.1
ralf-2                IN A     6.0.0.2
ralf-1                IN AAAA  6::1
ralf-2                IN AAAA  6::2
;
; Cluster A and AAAA records - sprout and bono pick randomly from these.
ralf                  IN A     6.0.0.1
ralf                  IN A     6.0.0.2
ralf                  IN AAAA  6::1
ralf                  IN AAAA  6::2
;
; (No need for NAPTR or SRV records as ralf doesn't handle SIP traffic.)

; ellis
; =====
;
; ellis is not clustered, so there's only ever one node.
;
; Per-node record - not required to have both IPv4 and IPv6 records
ellis-1                IN A     7.0.0.1
ellis-1                IN AAAA  7::1
;
; "Cluster"/access A and AAAA record
ellis                  IN A     7.0.0.1
ellis                  IN AAAA  7::1

ENUM overview¶

The NAPTR article on Wikipedia gives a pretty good overview of how ENUM works. Essentially, you

• convert your PSTN number into a domain name by stripping all punctuation, reversing the order of the digits, separating them with dots and suffixing with “e164.arpa” (e.g. 12012031234 becomes 4.3.2.1.3.0.2.1.0.2.1.e164.arpa)
• issue a DNS NAPTR query for this domain and receive the response (which is for the best prefix match to your number)
• parse the response, which contains a series of regular expressions and replacements
• find the best match for your number (after stripping all punctuation except any leading +) and apply the replacement
• if the NAPTR response indicated this match was “terminal”, then use the resulting SIP URI - if not, use the output of this pass as input for another ENUM query.

The RFCs to refer to for a definitive position are RFC 3401 (which covers Dynamic Delegation Discovery System (DDDS), on which ENUM is built) and RFC 3761 (which covers ENUM itself). Also relevant is RFC 4769 (which specifies ENUM service types).

Clearwater ENUM Support¶

Clearwater supports ENUM as set out in RFC 3761 and RFC 4769, with the following points/omissions.

• ENUM processing should only be applied to TEL URIs and SIP URIs that specify that this was dialed by the user. Since no SIP UAs that we’re aware of indicate that the SIP URI was dialed by a user, we assume that any all-numeric (including +) URIs should have ENUM processing applied.
• RFC 3761 section 2.4 says that the ENUM replacement string is UTF-8-encoded. We do not support this - all characters must be ASCII (i.e. a subset of UTF-8). It is not particularly meaningful to use UTF-8 replacements for SIP URIs as SIP URIs themselves must be ASCII.
• RFC 3761 section 6.1 says that a deployed ENUM service SHOULD include mechanisms to ameliorate security threats and mentions using DNSSEC. We don’t support DNSSEC, so other security approaches (such as private ENUM servers) must be used.
• RFC 4769 describes ENUM rules that output TEL URIs. Since we have no way (apart from ENUM itself) of routing to TEL URIs, we ignore such rules.
• RFC 3761 section 2.1 describes the construction of the Application Unique String from the E.164 number. We have no way of converting a user-dialed number into an E.164 number, so we assume that the number provided by the user is E.164.

Deciding on ENUM rules¶

ENUM rules will vary with your deployment, but the (JSON-ized) rules from an example deployment might form a reasonable basis:

{
    "number_blocks" : [
        {   "name" : "Demo system number 2125550270",
            "prefix" : "2125550270",
            "regex" : "!(^.*$)!sip:\\1@10.147.226.2!"
        },
        {
            "name" : "Clearwater external numbers 2125550271-9",
            "prefix" : "212555027",
            "regex" : "!(^.*$)!sip:+1\\1@ngv.example.com!"
        },
        {
            "name" : "Clearwater external numbers +12125550271-9",
            "prefix" : "+1212555027",
            "regex" : "!(^.*$)!sip:\\1@ngv.example.com!"
        },
        {
            "name" : "Clearwater external number 2125550280",
            "prefix" : "2125550280",
            "regex" : "!(^.*$)!sip:+1\\1@ngv.example.com!"
        },
        {
            "name" : "Clearwater external number +12125550280",
            "prefix" : "+12125550280",
            "regex" : "!(^.*$)!sip:\\1@ngv.example.com!"
        },
        {   "name" : "Clearwater internal numbers",
            "prefix" : "650555",
            "regex" : "!(^.*$)!sip:\\1@ngv.example.com!"
        },
        {   "name" : "Clearwater internal numbers dialled with +1 prefix",
            "prefix" : "+1650555",
            "regex" : "!+1(^.*$)!sip:\\1@ngv.example.com!"
        },
        {   "name" : "NANP => SIP trunk",
            "prefix" : "",
            "regex" : "!(^.*$)!sip:\\1@10.147.226.2!"
        }
    ]
}

Configuring ENUM rules¶

Normally, the ENUM server would be an existing part of the customer’s network (not part of Clearwater itself) but, for testing and demonstration, it is useful to be able to configure our own.

Unfortunately, AWS Route 53 does not support the required NAPTR records, so we can’t use this. Fortunately, BIND (easy to install) does, and dnsmasq (our standard DNS forwarder on Clearwater nodes) does to a limited extent. This section describes how to configure BIND or dnsmasq to respond to ENUM queries.

You can set up ENUM rules on either BIND or dnsmasq. BIND is the better choice, but requires installing an additional package and a few additional file tweaks. dnsmasq is the weaker choice - it does not support wildcard domains properly, so rules for each directory number must be added separately.

; Demo system number 2125550270
0.7.2.0.5.5.5.2.1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .

; Demo system number +12125550270
0.7.2.0.5.5.5.2.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .

; Clearwater external numbers 2125550271-9
*.7.2.0.5.5.5.2.1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:+1\\1@ngv.example.com!" .

; Clearwater external numbers +12125550271-9
; Note that we can't define a domain name containing + so we must define a
; domain without it and then match a telephone number starting with a + (if
; present) or (if not) use the default route via the SIP trunk.
*.7.2.0.5.5.5.2.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(\+^.*$)!sip:\\1@ngv.example.com!" .
*.7.2.0.5.5.5.2.1.2.1 IN NAPTR 2 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .

; Clearwater external number 2125550280
0.8.2.0.5.5.5.2.1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:+1\\1@ngv.example.com!" .

; Clearwater external number +12125550280
; Note that we can't define a domain name containing + so we must define a
; domain without it and then match a telephone number starting with a + (if
; present) or (if not) use the default route via the SIP trunk.
0.8.2.0.5.5.5.2.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(\+^.*$)!sip:\\1@ngv.example.com!" .
0.8.2.0.5.5.5.2.1.2.1 IN NAPTR 2 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .

; Clearwater internal numbers
*.5.5.5.0.5.6 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@ngv.example.com!" .

; Clearwater internal numbers dialled with +1 prefix
; Note that we can't define a domain name containing + so we must define a
; domain without it and then match a telephone number starting with a + (if
; present) or (if not) use the default route via the SIP trunk.
*.5.5.5.0.5.6.1 IN NAPTR 1 1 "u" "E2U+sip" "!\+1(^.*$)!sip:\\1@ngv.example.com!" .
*.5.5.5.0.5.6.1 IN NAPTR 2 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .

; NANP => SIP trunk.
* IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
2.1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.2.1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
5.2.1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.5.2.1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
5.5.2.1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.5.5.2.1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
5.5.5.2.1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.5.5.5.2.1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
0.5.5.5.2.1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.0.5.5.5.2.1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
2.0.5.5.5.2.1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.2.0.5.5.5.2.1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
7.2.0.5.5.5.2.1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
2.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.2.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
5.2.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.5.2.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
5.5.2.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.5.5.2.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
5.5.5.2.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.5.5.5.2.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
0.5.5.5.2.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.0.5.5.5.2.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
2.0.5.5.5.2.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.2.0.5.5.5.2.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
7.2.0.5.5.5.2.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
8.2.0.5.5.5.2.1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.8.2.0.5.5.5.2.1.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
8.2.0.5.5.5.2.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.8.2.0.5.5.5.2.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.0.8.2.0.5.5.5.2.1.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
6 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.6 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
5.6 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.5.6 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
5.0.5.6 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.5.0.5.6 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
5.5.0.5.6 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
*.5.5.0.5.6 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .
5.5.5.0.5.6 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@10.147.226.2!" .

# Demo system number 2125550270
naptr-record=0.7.2.0.5.5.5.2.1.2.e164.arpa,1,1,U,E2U+SIP,!(^.*$)!sip:\\1@10.147.226.2!

# Clearwater external numbers 2125550271-9
naptr-record=7.2.0.5.5.5.2.1.2.e164.arpa,1,1,U,E2U+SIP,!(^.*$)!sip:+1\\1@ngv.example.com!

# Clearwater external numbers +12125550271-9
# Note that we can't define a domain name containing + so we must define a # domain without it and then match a telephone number starting with a + (if
# present) or (if not) use the default route via the SIP trunk.
naptr-record=7.2.0.5.5.5.2.1.2.1.e164.arpa,1,1,U,E2U+SIP,!(\+^.*$)!sip:\\1@ngv.example.com!
naptr-record=7.2.0.5.5.5.2.1.2.1.e164.arpa,2,1,U,E2U+SIP,!(^.*$)!sip:\\1@10.147.226.2!

# Clearwater external number 2125550280
naptr-record=0.8.2.0.5.5.5.2.1.2.e164.arpa,1,1,U,E2U+SIP,!(^.*$)!sip:+1\\1@ngv.example.com!

# Clearwater external number +12125550280
# Note that we can't define a domain name containing + so we must define a
# domain without it and then match a telephone number starting with a + (if
# present) or (if not) use the default route via the SIP trunk.
naptr-record=0.8.2.0.5.5.5.2.1.2.1.e164.arpa,1,1,U,E2U+SIP,!(\+^.*$)!sip:\\1@ngv.example.com!
naptr-record=0.8.2.0.5.5.5.2.1.2.1.e164.arpa,2,1,U,E2U+SIP,!(^.*$)!sip:\\1@10.147.226.2!

# Clearwater internal numbers
naptr-record=5.5.5.0.5.6.e164.arpa,1,1,U,E2U+SIP,!(^.*$)!sip:\\1@ngv.example.com!

# Clearwater internal numbers dialled with +1 prefix
# Note that we can't define a domain name containing + so we must define a
# domain without it and then match a telephone number starting with a + (if
# present) or (if not) use the default route via the SIP trunk.
naptr-record=5.5.5.0.5.6.1.e164.arpa,1,1,U,E2U+SIP,!\+1(^.*$)!sip:\\1@ngv.example.com!
naptr-record=5.5.5.0.5.6.1.e164.arpa,2,1,U,E2U+SIP,!(^.*$)!sip:\\1@10.147.226.2!

# NANP => SIP trunk
naptr-record=e164.arpa,1,1,U,E2U+SIP,!(^.*$)!sip:\\1@10.147.226.2!

ENUM Domain Suffix¶

RFC 3761 mandates that domain names used during ENUM processing are suffixed with .e164.arpa. Obviously, this means that there can only be one such public domain. If you need your domain to be public (rather than private as set up above), you can instead change the suffix, e.g. to .e164.arpa.ngv.example.com, by

• configuring sprout to use this suffix via the –enum-suffix parameter
• configuring your DNS server to respond to this domain rather than .e164.arpa (by search-replacing in the config files described above)
• configuring Route 53 to forward DNS requests for e164.arpa.ngv.example.com to your DNS server by creating an NS (Name Server) record with name “e164.arpa.ngv.example.com” and value set to the name/IP address of your DNS server.

ENUM and Sprout¶

To enable ENUM lookups on Sprout, edit /etc/clearwater/shared_config and add the following configuration to use either an ENUM server (recommended) or an ENUM file:

enum_server=<IP addresses of enum servers>
enum_file=<location of enum file>

If you use the ENUM file, enter the ENUM rules in the JSON format (shown above). If you are using the enhanced node management framework provided by clearwater-etcd, and you use /etc/clearwater/enum.json as your ENUM filename, you can automatically synchronize changes across your deployment by running sudo /usr/share/clearwater/clearwater-config-manager/scripts/upload_enum_json after creating or updating the file. In this case, other Sprout nodes will automatically download and use the uploaded ENUM rules.

It’s possible to configure Sprout with secondary and tertiary ENUM servers, by providing a comma-separated list (e.g. enum_server=1.2.3.4,10.0.0.1,192.168.1.1). If this is done:

• Sprout will always query the first server in the list first
• If this returns SERVFAIL or times out (which happens after a randomised 500ms-1000ms period), Sprout will resend the query to the second server
• If this returns SERVFAIL or times out, Sprout will resend the query to the third server
• If all servers return SERVFAIL or time out, the ENUM query will fail

Configuration¶

There are 3 configuration steps.

1. Configure Clearwater’s IBCF to trust Voxbone’s servers.
2. Create a subscriber on Clearwater.
3. Associate the subscriber with a number on Voxbone.

Testing¶

The Voxbone web UI allows you to make a test call. Alternatively, you can make a normal call through the PSTN to your Voxbone number.

The only way to test SMS function is to actually send an SMS to your Voxbone number.

Troubleshooting¶

If your call or SMS doesn’t get through, there are three things to check.

• Does the SIP message actually reach the IBCF? You can check this using network capture (e.g. tcpdump) on the IBCF? If not, the problem is likely to either be that the SIP URI’s domain doesn’t resolve to your IBCF, or that security groups or firewall rules are blocking traffic to it.
• Is the SIP message trusted? You can again check this with network capture - if you see a 403 Forbidden response, this indicates that the IBCF does not trust the sender of the message. Check the trusted_peers entry in your /etc/clearwater/user_settings file and that you have restarted bono since updating it.
• Would the call go through if it were sent on-net? Try making a call to the subscriber from another subscriber on Clearwater and check that it goes through (or follow the standard troubleshooting process).

Configuration¶

These SNMP statistics require:

• the clearwater-snmpd package to be installed for all node types
• the clearwater-snmp-handler-astaire package to be installed for Sprout and Ralf nodes

These packages will be automatically installed when installing through the Chef automation system; for a manual install, you will need to install the packages with sudo apt-get install.

Usage¶

Clearwater nodes provide SNMP statistics over port 161 using SNMP v2c and community clearwater. The MIB definition file can be downloaded from here.

Our SNMP statistics are provided through plugins or subagents to the standard SNMPd packaged with Ubuntu, so querying port 161 (the standard SNMP port) on a Clearwater node will provide system-level stats like CPU% as well as any available Clearwater stats.

To load the MIB file, allowing you to refer to MIB objects by name, first place it in the ~/.snmp/mibs directory. To load the MIB file for just the current session, run export MIBS=+PROJECT-CLEARWATER-MIB. To load the MIB file every time, add the line mibs +PROJECT-CLEARWATER-MIB to a snmp.conf file in the ~/.snmp directory.

If a statistic is indexed by time period, then it displays the relevant statistics over:

• the previous five-second period
• the current five-minute period
• the previous five-minute period

For example, a stat queried at 12:01:33 would display the stats covering:

• 12:01:25 - 12:01:30 (the previous five-second period)
• 12:00:00 - 12:01:33 (the current five-minute period)
• 11:55:00 - 12:00:00 (the previous five-minute period)

All latency values are in microseconds.

(although the MIB file should be examined to determine exactly which ones). The full table can be retrieved by using the snmptable command. For example, the Initial Registrations table for Sprout can be retrieved by running: | snmptable -v2c -c clearwater <ip> PROJECT-CLEARWATER-MIB::sproutInitialRegistrationTable

The individual table elements can be accessed using: snmpget -v2c -c clearwater <ip> <table OID>.1.<column>.<row>

.1.2.826.0.1.1578918.9.3.9, so the number of initial registration attempts in the current five-minute period can be retrieved by: | snmpget -v2c -c clearwater <ip> .1.2.826.0.1.1578918.9.3.9.1.2.2 or by: | snmpget -v2c -c clearwater <ip> PROJECT-CLEARWATER-MIB::sproutInitialRegistrationAttempts.scopeCurrent5MinutePeriod

OIDs beneath a certain point in the MIB tree. For example, you can retrieve all of the entries in the Sprout Initial Registrations table using: | snmpwalk -v2c -c clearwater <ip> PROJECT-CLEARWATER-MIB::sproutInitialRegistrationTable

Running snmpwalk -v2c -c clearwater <ip> PROJECT-CLEARWATER-MIB::projectClearwater will output a very long list of all available Clearwater stats.

Note that running an ‘snmpget’ on a table OID will result in a “No Such Object available on this agent at this OID” message.

EMS¶

To integrate with an EMS, the EMS must support the following capabilities of RFC 3877 to obtain the brief description, detailed description, and severity for the alarm:

• The EMS must be configured to catch the SNMP INFORM messages used to report alarms from Clearwater. It is also recommended that the EMS must display the alarm information provided by the following MIBs.
• Upon receiving a SNMP INFORM message from Clearwater the EMS can obtain the alarm data by the following:
  • The EMS must retrieve the AlarmModelEntry MIB table data associated with the current SNMP INFORM message from Clearwater. This MIB provides the active/clear state, alarm description, and a table index for the ituAlarmEntry MIB.
  • The EMS must the retrieve the ituAlarmEntry MIB table data associated with the current alarm from Clearwater. This MIB provides the alarm severity and the additional detailed alarm description.

Alarm Models¶

The alarm models used by Clearwater are defined in alarmdefinition.h.

The authentication process¶

On attempting to access a node via SSH or SFTP, a user is either expected to use a key, or to provide a password to verify their identity, and thus pass through authentication successfully. This process requires a locally provisioned set of authentication details for each user that the sshd process can compare to the provided credentials for verification. In the case of password authentication, by enabling RADIUS all user accounts can be configured centrally on a RADIUS server, or in a database said server can access, and each node can pass user credentials provided at log in across to this server to complete the authentication process.

As the user attempting access may not exist locally on the node, which sshd requires, any unknown user is mapped to the default Ubuntu user to allow authentication to proceed correctly. As such, once authenticated, they will be acting as if they were the default user, but for auditing purposes it is the username provided at login that is recorded.

Prerequisites¶

The following conditions are assumed by this process:

• Your nodes are running on Ubuntu 14.04.
• Your nodes have access to both Clearwater and Ubuntu repositories.
• Your SSH configuration allows password authentication and PAM (the correct configuration will be detailed below).
• You have access to a third-party RADIUS server (such as freeRADIUS).
• Your firewall allows UDP traffic to the above server on port 1812.

Installation¶

sudo apt-get install clearwater-radius-auth

Usage¶

Once the above is installed and configured, any user provisioned in the RADIUS server can attempt SSH or SFTP access to the configured node, and on providing their password they will be authenticated against the details held on the RADIUS server, and logged in, acting as the default Ubuntu user. Commands such as who or last will output the username supplied at login, and this will also be recorded in the auth log /var/log/auth.log.

Any users provisioned locally on the node will see no change to their authentication experience. By default, RADIUS authentication is set to be a sufficient, but not required condition. As such, failing to authenticate against the server credentials will cause the authentication attempt to fall back to checking locally provisioned details. See below for further details on configuration options.

Troubleshooting¶

• If you are not seeing any traffic reaching your RADIUS server, and entries in /var/log/auth.log state that no RADIUS server was reachable, re-check the RADIUS server entry in /etc/pam_radius_auth.conf, and ensure that your firewall is configured to allow UDP traffic to the RADIUS server on port 1812.
• If your RADIUS server is rejecting authentication requests, ensure that the server is configured correctly.

Removal¶

To properly remove clearwater-radius-auth, and the components it brings with it, run the following:

sudo apt-get purge clearwater-radius-auth
sudo apt-get purge libpam-radius-auth
sudo apt-get purge libnss-ato

This will remove all configuration put in place by the installation. Should your configuration become corrupt, purging and re-installing the associated module will re-instate the correct configuration.

Further configuration¶

This section details the configuration put in place by the installation. It is highly recommended that these be left as their defaults. The following is for information purposes only.

ubuntu:x:1000:1000:Ubuntu:/home/ubuntu:/bin/bash

# PAM configuration for the Secure Shell service
# +clearwater-radius-auth
auth sufficient pam_radius_auth.so
# -clearwater-radius-auth
# Standard Un*x authentication.

passwd:   compat ato
group:    compat
shadow:   compat ato

What is an application server?¶

An application server (AS) is a server which is brought into or notified of a call by the network in order to provide call features or other behaviour.

In IMS, an application server is a SIP entity which is invoked by the S-CSCF for each dialog, both originating and terminating, as configured by the initial filter criteria (iFCs) of the relevant subscribers. The application server may reject or accept the call itself, redirect it, pass it back to the S-CSCF as a proxy, originate new calls, or perform more complex actions (such as forking) as a B2BUA. It may choose to remain in the signaling path, or drop out.

The application server communicates with the IMS core via the ISC interface. This is a SIP interface. The details are given in 3GPP TS 24.229, especially s5.4 (the S-CSCF side) and s5.7 (the AS side). Further useful information can be found in 3GPP TS 23.228. Section references below are to 3GPP TS 24.229 unless otherwise specified.

For a more detailed look at the ISC interface, and how it is implemented in Clearwater, see the ISC interface guide in the Sprout developer documentation.

Clearwater interfaces¶

In Clearwater most S-CSCF function, including the ISC interface, is implemented in Sprout. Sprout invokes application servers over the ISC interface, as specified in the iFCs.

• Per the IMS specs, this invocation occurs on dialog-initiating requests such as INVITE, SUBSCRIBE, MESSAGE, etc, and according to the triggers within the iFCs (s5.4.3.2, s5.4.3.3):
  • When specified, Sprout will route the message to the AS; the AS can either route it onward, act as a B2BUA for e.g. call diversion, or give a final response.
  • Clearwater has full support for chained iFCs. The original dialog is tracked by Sprout using an ODI token in the Route header. This support includes support for forking ASs at any point in the chain.
  • Service trigger points (i.e., conditions) are implemented, including SIP method, SIP headers, SDP lines, registration parameters, and request URIs.
  • Service trigger points can have session case configuration, allowing the AS to only be invoked on originating calls or on terminating calls.
• No per-AS configuration is required; ASs are invoked simply by their URI appearing in the iFCs.
• AS invocation also occurs on REGISTER - this is called third-party registration (3GPP TS 24.229 s5.4.1.7 and 7A):
  • When a UE registers with Sprout, if the iFCs require it, it passes a third-party registration onto an AS.
  • Message body handling for third-party registration, per 3GPP TS 24.229 s5.4.1.7A: including optionally service info, a copy of the registration, and a copy of the response.
  • Network-initiated deregister. If the third-party registration fails and the iFC requests it, we must deregister the UE.

The built-in MMTEL application server¶

Clearwater has a built-in application server, mmtel.<deployment-domain>, which implements a subset of the MMTEL services defined in GSMA PRD IR.92, ETSI TS 129.364 and 3GPP TS 24.623:

• Originating Identification Presentation (OIP)
• Originating Identification Restriction (OIR)
• Communication Diversion (CDIV)
• Communication Barring (CB)

Note that Clearwater is also able to support a number of other IR.92 services, either through inherent support (by transparently passing messages between UEs) or through integration with external application servers. See the IR.92 Supplementary Services document for more information.

The built-in MMTEL application server is invoked only for calls configured to use it. To use it, simply configure a subscriber’s iFCs to indicate the use of mmtel.<deployment-domain> as an application server. The MMTEL application server can be used on its own, or as one of a chain of application servers handling a call. The default iFCs configured by Ellis specify that the built-in MMTEL application server should be used for all originating and terminating calls.

Instructions¶

In the instructions below, <domain> is your Clearwater domain.

• Configure a line in ellis.
• Check the camera is not in use: check the camera light is off. If it is on, close the app that is using it (e.g., Skype, or a phone client).
• Visit http://sipml5.org/ and click on the button to start the live demo. If the website is down, you can also use http://doubango.org/sipml5/.
• By default sipML5 uses a SIP<->WebRTC gateway run by sipml5.org. Clearwater supports WebRTC directly. To tell sipML5 to speak WebRTC directly to Clearwater:
  • Click on the “expert mode” button to open the “expert mode” tab, and fill in the following field:
    • WebSocket Server URL: ws://<domain>:5062
  • Click Save.
  • Go back to the calls tab.
• Fill in the fields as follows:
  • Display name: <whatever you like, e.g., your name>
  • Private identity: <phone number>@<domain>
  • Public identity: sip:<phone number>@<domain>
  • Password: <password as displayed in ellis>
  • Realm: <domain>
• Now click Log In. This sometimes takes a few seconds. If it fails (e.g., due to firewall behaviour), click Log Out then Log In again.

Some firewalls or proxies may not support WebSockets (the technology underlying WebRTC). If this is the case, logging in will not work (probably it will time out). Clearwater uses port 5062 for WebSockets.

Once you have this set up on two tabs, or two browsers, you can make calls between them as follows:

• Entering the phone number
• Press the Call button.
• Accept the local request to use the camera/mic.
• Remote browser rings.
• Accept the remote request to use the camera/mic.
• Click Answer.
• Talk.
• Click Hang up on either end.

Video can be disabled on the ‘expert mode’ tab.

Note: Clearwater does not transcode media between WebRTC and non-WebRTC clients, so such calls are not possible. Transcoding will be supported in future.

Known sipML5 bugs¶

At the time of writing, the following sipML5 bugs are known:

• Calls where one or both ends do not have a webcam do not always complete correctly. It is best to ensure both ends have a webcam, even if audio-only calls are being made.
• Hang up doesn’t work - you have to hang up on both ends.

Using STUN and TURN¶

Note: The above uses a default (non-Clearwater) STUN server, and no TURN server. Clearwater has its own STUN and TURN servers which can be used to support clients behind NATs and firewalls, but at the time of writing (April 2013) there are some difficulties with the WebRTC spec that make the Clearwater TURN server inaccessible to compliant browsers. These are being tracked.

If one or both clients are behind a symmetric firewall, you must use TURN. Configure this as follows:

• On the ‘expert mode’ tab, fill in the following field with your SIP username and the TURN workaround password set by your administrator (turn_workaround in Installing a Chef workstation). This is a workaround for Chrome WebRTC issue 1508 which will be fixed in Chrome 28 (not yet released).
  • ICE Servers: [{url:”turn:<phone number>%40<domain>@<domain>”,credential:”<password>”}]

Testing TURN¶

To test TURN when one or both of the clients are running under Windows 7, find the IP of the remote client, and block direct communication between them as follows:

• Control Panel > Windows Firewall > Advanced Settings > Inbound Rules > New Rule > Custom > All Programs > Any Protocol > specify remote IP address > Block the connection > (give it a name).
• Do the same for outbound.

Supported by Clearwater’s Built-In MMTel Application Server¶

The following Supplementary Services are supported by Clearwater’s built-in MMTel Application Server.

• Originating Identification Presentation - See the Privacy Feature document for more information.
• Originating Identification Restriction - See the Privacy Feature document for more information.
• Call Forwarding Unconditional - See the Call Diversion Support document for more information.
• Call Forwarding on not Logged in - See the Call Diversion Support document for more information.
• Call Forwarding on Busy - See the Call Diversion Support document for more information.
• Call Forwarding on not Reachable - See the Call Diversion Support document for more information.
• Call Forwarding on No Reply - See the Call Diversion Support document for more information.
• Barring of All Incoming Calls - See the Call Barring Support document for more information.
• Barring of All Outgoing Calls - See the Call Barring Support document for more information.
• Barring of Outgoing International Calls - See the Call Barring Support document for more information.

Supported by Clearwater Inherently¶

The following Supplementary Services are primarily implemented on the UE, and just require Clearwater to proxy messages and headers, which it does.

• Communication Hold
• Communication Waiting

Requiring an External Application Server¶

The following Supplementary Services require an external application server.

• Terminating Identification Presentation
• Terminating Identification Restriction
• Barring of Outgoing International Calls - ex Home Country - Clearwater does not currently have sufficient configuration to know the subscriber’s home country.
• Message Waiting Indication - Clearwater proxies SUBSCRIBE and NOTIFY messages for the MWI event package (RFC 3842), but requires an external application server to handle/generate them.
• Ad-Hoc Multi Party Conference - Clearwater proxies messages between the UE and an external application server that conferences sessions together.

Not Supported¶

The following Supplementary Service is not currently supported by Clearwater.

• Barring of Incoming Calls - When Roaming - Clearwater does not currently support roaming, and so can’t support barring on this basis.

Listing Backups¶

The process for listing backups varies between ellis and homestead, homer and memento.

Backups for ellis:
1372294741  /usr/share/clearwater/ellis/backup/backups/1372294741
1372294681  /usr/share/clearwater/ellis/backup/backups/1372294681
1372294621  /usr/share/clearwater/ellis/backup/backups/1372294621
1372294561  /usr/share/clearwater/ellis/backup/backups/1372294561

No backup directory specified, defaulting to /usr/share/clearwater/homestead/backup/backups
provisioning1372812963174
provisioning1372813022822
provisioning1372813082506
provisioning1372813143119

Taking a Manual Backup¶

The process for taking a manual backup varies between ellis and homestead, homer and memento. Note that in all cases,

• the backup is stored locally and should be copied to a secure backup server to ensure resilience
• this process only backs up a single local node, so the same process must be run on all nodes in a cluster to ensure a complete set of backups
• these processes cause a small amount of extra load on the disk, so it is recommended not to perform this during periods of high load
• only 4 backups are stored locally - when a fifth backup is taken, the oldest is deleted.

Creating backup in /usr/share/clearwater/ellis/backup/backups/1372336317/db_backup.sql

snapshot=<snapshot>
sudo bash -c 'cp /usr/share/clearwater/ellis/backup/backups/'$snapshot'/db_backup.sql ~'$USER' &&
              chown '$USER.$USER' db_backup.sql'

...
Deleting old backup: /usr/share/clearwater/homestead/backup/backups/1372812963174
Creating backup for keyspace homestead_provisoning...
Requested snapshot for: homestead_provisioning
Snapshot directory: 1372850637124
Backups can be found at: /usr/share/clearwater/homestead/backup/backups/provisioning/

Periodic Automated Local Backups¶

Ellis, homestead, homer and memento are all automatically configured to take daily backups if you’ve installed them through chef, at midnight local time every night.

If you want to turn this on, edit your crontab by running sudo crontab -e and add the following lines if not already present:

• 0 0 * * * /usr/share/clearwater/ellis/backup/do_backup.sh on ellis
• 0 0 * * * /usr/share/clearwater/bin/run-in-signaling-namespace /usr/share/clearwater/bin/do_backup.sh homestead_provisioning and 5 0 * * * /usr/share/clearwater/bin/run-in-signaling-namespace /usr/share/clearwater/bin/do_backup.sh homestead_cache on homestead
• 0 0 * * * /usr/share/clearwater/bin/run-in-signaling-namespace /usr/share/clearwater/bin/do_backup.sh homer on homer
• 0 0 * * * /usr/share/clearwater/bin/run-in-signaling-namespace /usr/share/clearwater/bin/do_backup.sh memento on memento.

These backups are stored locally, in the same locations as they would be generated for a manual backup.

Restoring from a Backup¶

There are three stages to restoring from a backup.

1. Copying the backup files to the correct location.
2. Running the restore backup script.
3. Synchronizing ellis, homestead, homer and memento’s views of the system state.

This process will impact service and overwrite data in your database.

snapshot=<snapshot>
sudo chown root.root db_backup.sql
sudo mkdir -p /usr/share/clearwater/ellis/backup/backups/$snapshot
sudo mv ~/backup/$snapshot/db_backup.sql /usr/share/clearwater/ellis/backup/backups/$snapshot

Will attempt to backup from backup 1372336317
Found backup directory 1372336317
Restoring backup for ellis...
--------------
/*!40101 SET @OLD_CHARACTER_SET_CLIENT=@@CHARACTER_SET_CLIENT */
--------------

...

--------------
/*!40111 SET SQL_NOTES=@OLD_SQL_NOTES */
--------------

Will attempt to backup from backup 1372336442947
Will attempt to backup from directory /home/ubuntu/bkp_test/
Found backup directory /home/ubuntu/bkp_test//1372336442947
Restoring backup for keyspace homestead_provisioning...
xss =  -ea -javaagent:/usr/share/cassandra/lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xm
s826M -Xmx826M -Xmn100M -XX:+HeapDumpOnOutOfMemoryError -Xss180k
Clearing commitlog...
filter_criteria: Deleting old .db files...
filter_criteria: Restoring from backup: 1372336442947
private_ids: Deleting old .db files...
private_ids: Restoring from backup: 1372336442947
public_ids: Deleting old .db files...
public_ids: Restoring from backup: 1372336442947
sip_digests: Deleting old .db files...
sip_digests: Restoring from backup: 1372336442947

cd /usr/share/clearwater/ellis
sudo env/bin/python src/metaswitch/ellis/tools/sync_databases.py

Shared Configuration¶

In addition to the data stored in ellis, homer, homestead and memento, a Clearwater deployment also has shared configuration that is automatically shared between nodes. This is stored in a distributed database, and mirrored to files on the disk of each node.

/usr/share/clearwater/clearwater-config-manager/scripts/upload_shared_config
/usr/share/clearwater/clearwater-config-manager/scripts/upload_bgcf_json
/usr/share/clearwater/clearwater-config-manager/scripts/upload_enum_json
/usr/share/clearwater/clearwater-config-manager/scripts/upload_scscf_json

International Call detection¶

The 3GPP specification states that to detect an international number, a SIP router should discount SIP/SIPS URIs that do not have a user=phone parameter set on them (this parameter is intended to distinguish between dialed digits and call-by-name with numeric names) but X-Lite and other SIP clients never set this parameter. Because of this behavior, the Clearwater implementation considers all SIP URIs for international call categorization.

National dialing prefix¶

Another limitation is that the international call detection algorithm only checks against a global ‘national’ dialing prefix (which is fine unless subscribers are expected to be multi-national within one Clearwater deployment). There does not appear to be a mechanism in IMS networks to store a subscriber-specific locale (e.g. in the HSS) so this would require some smarts to solve. These issues may be addressed in subsequent development.

Before scaling your deployment¶

Before scaling up or down, you should decide how many each of Bono, Sprout, Homestead, Homer and Ralf nodes you need (i.e. your target size). This should be based on your call load profile and measurements of current systems, though based on experience we recommend scaling up a tier of a given type (sprout, bono, etc.) when the average CPU utilization within that tier reaches ~60%. The Deployment Sizing Spreadsheet may also provide useful input.

Performing the resize¶

knife deployment resize -E <env> --sprout-count <n> --bono-count <n> --homer-count <n> --homestead-count <n> --ralf-count <n>

Configuration¶

User-specified configuration is held and validated by the XDM. When a subscriber is provisioned, Ellis injects a default document into the XDM for that user, this document turns off OIR by default. After this, if the user wishes to change their call services configuration, they use the XCAP interface (Ut) to change their document.

If the user has turned on Privacy by default, then all calls originating from the user will have identifying headers (e.g. User-Agent, Organization, Server...) stripped out from the originating request and will have the From: header re-written as "Anonymous" <sip:anonymous@anonymous.invalid\>;tag=xxxxxxx, preventing the callee from working out who the caller is. With the current implementation, there are some headers that are left in the message that could be used to determine some information about the caller, see below.

Application¶

Dialog-Wide Issues¶

There are a few issues with our stateless model and privacy that are worth drawing out here, pending a fuller discussion.

• We only apply OIR processing and privacy obfuscation to originating requests (i.e. not on in-dialog requests or responses)
• We cannot apply Contact/Via/RR-stripping since to do so would require us to restore the headers on not just the response but also on all callee in-dialog requests.
• Similarly for Call-Id obfuscation.
• Stripping Contact/Via/RR would prevent features such as callee call push from working (since they require the callee to send a REFER that identifies the caller).

Install and Configure an IBCF¶

Install and configure an IBCF node with the following steps.

• Install the node as if installing a Bono node, either manually or using Chef. If using Chef, use the ibcf role, for example

  knife box create -E <name> ibcf
  

• Edit the /etc/clearwater/user_settings file (creating it if it does not already exist) and add or update the line defining the IP addresses of SIP trunk peer nodes.

  trusted_peers="<trunk 1 IP address>,<trunk 2 IP address>,<trunk 3 IP address>, ..."
  

• Restart the Bono daemon.

  service stop bono (allow monit to restart Bono)
  

ENUM Configuration¶

The number ranges that should be routed over the SIP trunk must be set up in the ENUM configuration to map to a SIP URI owned by the SIP trunk provider, and routable to that provider. For example, you would normally want to map a dialed number to a URI of the form <number>@<domain>.

This can either be achieved by defining rules for each number range you want to route over the trunk, for example

*.0.1.5 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@<trunk IP address>!" .
*.0.1.5.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@<trunk IP address>!" .

or by defining a default mapping to the trunk and exceptions for number ranges you want to keep on-net, for example

* IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@<trunk IP address>!" .
*.3.2.1.2.4.2 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@<local domain>!" .
*.3.2.1.2.4.2.1 IN NAPTR 1 1 "u" "E2U+sip" "!(^.*$)!sip:\\1@<local domain>!" .

You can also use ENUM to provide number portability information, for example

*.3.2.1.2.4.2 IN NAPTR 1 1 "u" "E2U+pstn:tel" "!(^.*$)!sip:\\1;npdi;rn=+242123@<local domain>!" .
*.3.2.1.2.4.2.1 IN NAPTR 1 1 "u" "E2U+pstn:tel" "!(^.*$)!sip:\\1;npdi@<local domain>!" .

Refer to the ENUM guide for more about how to configure ENUM.

BGCF Configuration¶

BGCF (Border Gateway Control Function) configuration is stored in the bgcf.json file in /etc/clearwater on each sprout node (both I- and S-CSCF). The file stores two types of mappings. This first maps from SIP trunk IP addresses and/or host names to IBCF host names, and the second maps from a routing number (using prefix matching) to IBCF host names. These mappings control which IBCF nodes are used to route to a particular destination. Each entry can only apply to one type of mapping.

Multiple nodes to route to can be specified. In this case, Route headers are added to the message such that it is sent to the first node and the first node sends it to the second node and so on; the message is not tried at the second node if it fails at the first node.

The file is in JSON format, for example.

{
    "routes" : [
        {   "name" : "<route 1 descriptive name>",
            "domain" : "<SIP trunk IP address or host name>",
            "route" : ["<IBCF SIP URI>"]
        },
        {   "name" : "<route 2 descriptive name>",
            "domain" : "<SIP trunk IP address or host name>",
            "route" : ["<IBCF SIP URI>", "<IBCF SIP URI>"]
        },
        {   "name" : "<route 3 descriptive name>",
            "number" : "<Routing number>",
            "route" : ["<IBCF SIP URI>", "<IBCF SIP URI>"]
        }
    ]
}

There can be only one route set for any given SIP trunk IP address or host name. If there are multiple routes with the same destination IP address or host name, only the first will be used. Likewise, there can only be one route set for any given routing number; if there are multiple routes with the same routing number only the first will be used.

A default route set can be configured by having an entry where the domain is set to "*". This will be used by the BGCF if it is trying to route based on the the domain and there’s no explicit match for the domain in the configuration.

There is no default route set if the BGCF is routing based on the routing number provided.

After making a change to this file you should run sudo /usr/share/clearwater/clearwater-config-manager/scripts/upload_bgcf_json to ensure the change is synchronized to other Sprout nodes on your system (including nodes added in the future).

Background¶

Clearwater acts as an S-CSCF for one or more home domains. It can only provide service to users within a home domain for which it is the S-CSCF. Traffic to or from users in any other home domains must go via the S-CSCF for those other domains.

Home domains are a routing concept. There is a similar (but distinct) concept of authentication realms. When a user authenticates, as well as providing their username and digest, they must also provide a realm, which describes which user database to authenticate against. Many clients default their home domain to match their authentication realm.

A single Clearwater deployment can support multiple home domains and multiple realms.

Configuration¶

There are three steps to configuring multiple home domains and/or multiple realms.

• Configure the Clearwater deployment to know about all the home domains it is responsible for.
• Add DNS records to point all home domains at the deployment.
• Create subscribers within these multiple home domains, and optionally using multiple different realms.

Restriction¶

Clearwater does not support connections to multiple Diameter realms for the Cx interface to the HSS or the Rf interface to the CDF - only a single Diameter realm is supported.

Introduction¶

As of the Ultima release, Project Clearwater can be configured to provide signaling service (SIP/HTTP/Diameter) on a separate physical (or virtual) network to management services (SNMP/SSH/Provisioning). This may be used to protect management devices (on a firewalled network) from attack from the outside world (on the signaling network).

Technical Details¶

Configuration¶

ip netns add signaling
ip link set eth1 netns signaling
ip netns exec signaling ifconfig lo up
ip netns exec signaling ifconfig eth1 1.2.3.4/16 up
ip netns exec signaling route add default gateway 1.2.0.1 dev eth1

nameserver <DNS IP address>

signaling_namespace=<namespace name>
signaling_dns_server=<DNS IP address>
management_local_ip=<Node's local IP address in the management

Diagnostic Changes¶

All the built-in Clearwater diagnostics will automatically take note of network namespaces, but if you are running diagnostics yourself (e.g. following instructions from the troubleshooting page) you may need to prefix your commands with ip netns exec <namespace> to run them in the signaling namespace. The following tools will need this prefix:

• cqlsh - For viewing Cassandra databases
• nodetool - For viewing Cassandra status
• telnet - For inspecting Memcached stores

Architecture¶

The architecture of a geographically-redundant system is as follows.

Diagram

Sprout has one memcached cluster per geographic region. Although memcached itself does not support the concept of local and remote peers, Sprout builds this on top - writing to both local and remote clusters and reading from the local but falling back to the remote. Communication between the nodes should be secure - for example, if it is going over the public internet rather than a private connection between datacenters, it should be encrypted and authenticated with IPsec. Each Sprout uses Homers and Homesteads in the same region only. Sprout also has one Chronos cluster per geographic region; these clusters do not communicate.

Separate instances of Bono in each geographic region front the Sprouts in that region. Clearwater uses a geo-routing DNS service such as Amazon’s Route 53 to achieve this. A geo-routing DNS service responds to DNS queries based on latency, so if you’re nearer to geographic region B’s instances, you’ll be served by them.

Homestead and Homer are each a single cluster split over the geographic regions. Since they are backed by Cassandra (which is aware of local and remote peers), they can be smarter about spatial locality. As with Sprout nodes, communication between the nodes should be secure.

Ellis is not redundant, whether deployed in a single geographic region or more. It is deployed in one of the geographic regions and a failure of that region would deny all provisioning function.

Ralf does not support geographic redundancy. Each geographic region has its own Ralf cluster; Sprout and Bono should only communicate with their local Ralfs.

While it appears as a single node in our system, Route 53 DNS is actually a geographically-redundant service provided by Amazon. Route 53’s DNS interface has had 100% uptime since it was first turned up in 2010. (Its configuration interface has not, but that is less important.)

The architecture above is for 2 geographic regions - we do not currently support more regions.

Note that there are other servers involved in a deployment that are not described above. Specifically,

• communication back to the repository server is via HTTP. There is a single repository server. The repository server is not required in normal operation, only for upgrades.

Limitations¶

• The local IP addresses of all nodes in a deployment most be reachable from all other nodes - there must not be a NAT between the two GR sites. (This currently precludes having the GR sites in different EC2 regions.)

Impact¶

This section considers the probable impact on a subscriber of a total outage of a region in a 2-region geographically-redundant deployment. It assumes that the deployments in both regions have sufficient capacity to cope with all subscribers (or the deployments scale elastically) - if this is not true, we will hit overload scenarios, which are much more complicated.

The subscriber interacts with Clearwater through 3 interfaces, and these each have a different user experience.

• SIP to Bono for calls
• HTTP to Homer for direct call service configuration (not currently exposed)
• HTTP to Ellis for web-UI-based provisioning

For the purposes of the following descriptions, we label the two regions A and B, and the deployment in region A has failed.

  50% chance of being on a Bono node in region A *
  30s/300s chance of having a stale DNS entry *
  300s re-registration time
= 5% chance of a 300s outage
= 15s average outage

Setup¶

The process for setting up a geographically-redundant deployment is as follows.

1. Create independent deployments for each of the regions, with separate DNS entries. See the manual install instructions for the required GR settings.
2. Set up DNS (probably using SRV records) so that:
   • Bono nodes prefer the Sprout node local to them, but will fail over to the one in the other site.
   • Sprout nodes only use the Homer, Homestead and Ralf nodes in their local site.
   • The Ellis node ony uses the Homer and Homesteads in its local site.
3. Configure Route 53 to forward requests for Bono according to latency. To do this, for each region, create one record set, as follows.
   • Name: <shared (non-geographically-redundant) DNS name>
   • Type: “A - IPv4 address” or “AAAA - IPv6 address”
   • Alias: No
   • TTL (Seconds): 30 (or as low as you can go - in a failure scenario, we need to go back to DNS ASAP)
   • Value: <list of IPv4 or IPv6 addresses for this region>
   • Routing Policy: Latency
   • Region: <AWS region matching geographic region>
   • Set ID: <make up a unique name, e.g. gr-bono-us-east-1>

You can also use Chef to try GR function, by setting "gr" => true in the environment, as described in the automated install docs.

Configuration¶

As discussed in the Clearwater installation instructions, there are several ways to install Clearwater, and which way you choose affects how you must configure IPv6.

Missing Function¶

The following function is not yet supported on IPv6.

• Automated install
• SIP/WebSockets (WebRTC)

Supported¶

The following RFCs are already supported by Clearwater. Note that a number of these are supported simply because they require no active function from SIP proxies other than forwarding headers unchanged.

Relevant to Clearwater and partially supported¶

Relevant to Clearwater but not currently supported¶

These are the RFCs which are relevant to Clearwater and not yet supported.

SDP/Media RFCs¶

TS 24.229 references a number of specifications which relate to media function - either SDP negotiation or media transport or both. Clearwater currently passes SDP transparently and does not get involved in media flows. Unless we change this position, Clearwater can either be considered to support the RFC (because it supports passing SDP with the relevant contents) or that the RFC is not applicable to Clearwater.

The following is a brief explanation of each RFC, and its relevance to IMS.

• SDP (RFC 4566) - defines basic SDP protocol.
• Offer/Answer model for SDP media negotiation (RFC 3264) - defines how SDP is used to negotiate media.
• Default RTP/AV profile (RFC 3551) - defines default mappings from audio and video encodings to RTP payload types.
• Media Resource Reservation (RFC 3312 and RFC 4032) - defines additional SDP parameters to signal media resource reservation between two SIP UAs. TS 24.229 requires UEs, MGCFs and ASs to use these mechanisms, and that P-CSCF should monitor the flows if it is performing IMS-ALG or media gating functions.
• Mapping of media streams to resource reservation (RFC 3524 and RFC 3388) - define how multiple media streams can be grouped in SDP and mapped to a single resource reservation. IMS requires UEs to use these encodings when doing resource reservations.
• Signaling bandwidth required for RTCP (RFC 3556) - by default RTCP is assumed to consume 5% of session bandwidth, but this is not accurate for some encodings, so this RFC defines explicit signaling of RTCP bandwidth in SDP. This function is optional according to TS 24.229.
• TCP media transport (RFC 3556 and RFC 6544) - defines how support for TCP media transport is encoded in SDP for basic SDP exchanges and for ICE exchanges. According to TS 24.229, media over TCP is optional in most of an IMS network, but mandatory in an IBCF implementing IMS-ALG function.
• ICE (RFC 5245) - defines ICE media negotiation used to establish efficient media paths through NATs. According to TS 24.229 support for ICE is optional in most of the IMS network, but mandatory on an IBCF implementing IMS-ALG function.
• STUN (RFC 5389) - defines a protocol used to allow UAs to obtain their server reflexive address for use in ICE.
• TURN (RFC 5766) - defines extensions to STUN used to relay media sessions via a TURN server to traverse NATs when STUN-only techniques don’t work.
• Indicating support for ICE in SIP (RFC 5768) - defines a media feature tag used to signal support for ICE.
• SDP for binary floor control protocol (RFC 4583) - defines SDP extensions for establishing conferencing binary floor control protocol streams. Optional according to TS 24.229.
• Real-time RTCP feedback (RFC 4585 and RFC 5104) - defines extensions to RTCP to provide extended real-time feedback on network conditions. Mandatory for most IMS components handling media (MRFs, IBCFs, MGCFs), but optional for UEs.
• SDP capability negotiation (RFC 5939) - defines extensions to SDP to allow for signaling and negotiation of media capabilities (such as alternate transports - SRTP). Mandatory for most IMS components handling media (MRFs, IBCFs, MGCFs), but optional for UEs.
• SDP transport independent bandwidth signaling (RFC 3890) - defines extensions to SDP to signal codec-only (ie. independent of transport) bandwidth requirements for a stream. Optional for IMS components handling media.
• Secure RTP (RFC 3711, RFC 4567, RFC 4568, RFC 6043) - defines transport of media over a secure variant of RTP, supporting encryption (to prevent eavesdropping) and integrity protecting (to protect against tampering). Keys are either exchanged in SDP negotiation (specified in RFC 4567 and RFC 4568) or distributed via a separate key management service - termed MIKEY-TICKEY (specified in RFC 6043).
• Transcoder invocation using 3PCC (RFC 4117) - defines how a transcoding service can be inserted into the media path when required using third-party call control flows. According to TS 24.229 this is only applicable to app servers and MRFC, and even then is optional.
• MSRP (RFC 4975) - defines a protocol for session-based transmission of a sequence of instant messages. Only applicable to UEs, ASs and MRFCs and optional.
• SDP for file transfer (RFC 5547) - defines SDP extensions to support simple file transfers between two IMS nodes. Optional.
• Explicit congestion notifications for RTP over UDP (RFC 6679) - defines RTCP extensions for reporting urgent congestion conditions and reporting congestion summaries. Optional.
• Setting up audio streams over circuit switched bearers (referenced as draft-ietf-mmusic-sdp-cs-00) - defines SDP extensions for negotiating audio streams over a circuit switched network. Mandatory for ICS capable UEs and SCC-AS, otherwise optional.
• Media capabilities negotiation in SDP (referenced as draft-ietf-mmusic-sdp-media-capabilities-08) - defines SDP extensions for signaling media capabilities such as encodings and formats. Mandatory for ICS capable UEs and SCC-AS, otherwise optional.
• Miscellaneous capabilities negotiation in SDP (referenced as draft-ietf-mmusic-sdp-miscellaneous-caps-00) - defines SDP extensions for signaling some miscellaneous capabilities. Mandatory for ICS capable UEs and SCC-AS, otherwise optional.

Is my Deployment Using Automatic Clustering and Configuration Sharing?¶

To tell if your deployment is using this feature, log onto one of the nodes in your deployment and run dpkg --list | grep clearwater-etcd. If this does not give any output the feature is not in use.

Migrating to Automatic Clustering and Configuration Sharing¶

Deployments that are not using the feature may be migrated so they start using it. To perform this migration, follow these instructions.

Creating users¶

New users can be created with the create_user tool. As well as creating single users, it’s also possible to create multiple users with a single command. Note that this is not recommended for provisioning large numbers of users - for that, bulk provisioning is much quicker.

usage: create_user.py [-h] [-k] [-q] [--hsprov IP:PORT] [--plaintext]
                      [--ifc iFC-FILE] [--prefix TWIN_PREFIX]
                      <directory-number>[..<directory-number>] <domain>
                      <password>

Create user

positional arguments:
  <directory-number>[..<directory-number>]
  <domain>
  <password>

optional arguments:
  -h, --help            show this help message and exit
  -k, --keep-going      keep going on errors
  -q, --quiet           don't display the user
  --hsprov IP:PORT      IP address and port of homestead-prov
  --plaintext           store password in plaintext
  --ifc iFC-FILE        XML file containing the iFC
  --prefix TWIN_PREFIX  twin-prefix (default: 123)

Update users¶

Existing users’ passwords can be updated with the update_user tool.

usage: update_user.py [-h] [-k] [-q] [--hsprov IP:PORT] [--plaintext]
                      <directory-number>[..<directory-number>] <domain>
                      <password>

Update user

positional arguments:
  <directory-number>[..<directory-number>]
  <domain>
  <password>

optional arguments:
  -h, --help            show this help message and exit
  -k, --keep-going      keep going on errors
  -q, --quiet           don't display the user
  --hsprov IP:PORT      IP address and port of homestead-prov
  --plaintext           store password in plaintext

Delete users¶

Users can be deleted with the delete_user tool.

usage: delete_user.py [-h] [-f] [-q] [--hsprov IP:PORT]
                      <directory-number>[..<directory-number>] <domain>

Delete user

positional arguments:
  <directory-number>[..<directory-number>]
  <domain>

optional arguments:
  -h, --help            show this help message and exit
  -f, --force           proceed with delete in the face of errors
  -q, --quiet           silence 'forced' error messages
  --hsprov IP:PORT      IP address and port of homestead-prov

Display users¶

Users’ details can be displayed with the display_user tool.

usage: display_user.py [-h] [-k] [-q] [-s] [--hsprov IP:PORT]
                       <directory-number>[..<directory-number>] <domain>

Display user

positional arguments:
  <directory-number>[..<directory-number>]
  <domain>

optional arguments:
  -h, --help            show this help message and exit
  -k, --keep-going      keep going on errors
  -q, --quiet           suppress errors when ignoring them
  -s, --short           less verbose display
  --hsprov IP:PORT      IP address and port of homestead-prov

List users¶

All the users provisioned on the system can be listed with the list_users tool.

Note that the --full parameter defaults to off because it greatly decreases the performance of the tool (by more than an order of magnitude).

The --pace parameter’s default values should ensure that this does not use more than 10% of the homestead cluster’s CPU - that is 5 users per second if --force is set and 500 if not. If you set the “–pace” parameter to more than the default, you’ll be prompted to confirm (or specify the --force parameter).

usage: list_users.py [-h] [-k] [--hsprov IP:PORT] [--full] [--pace PACE] [-f]

List users

optional arguments:
  -h, --help        show this help message and exit
  -k, --keep-going  keep going on errors
  --hsprov IP:PORT  IP address and port of homestead-prov
  --full            displays full information for each user
  --pace PACE       sets the target number of users to list per second
  -f, --force       forces specified pace

Removing a Failed Node¶

If a node permanently fails scaling the deployment up and down may stop working, or if a scaling operation is in progress it may get stuck (because other nodes in the tier will wait forever for the failed node to react). To recover from this situation the failed node should be removed from the deployment using the following steps:

• Remove the node from the underlying etcd cluster. To do this:
  • Run clearwater-etcdctl cluster-health and make a note of the ID of the failed node.
  • Run clearwater-etcdctl member list to check that the failed node reported is the one you were expecting (by looking at its IP address).
  • Run clearwater-etcdctl member remove <ID>, replacing <ID> with the ID learned above.
• Remove the failed node from any back-end data store clusters it was a part of (see Removing a Node From a Data Store).

Multiple Failed Nodes¶

If your deployment loses half or more of its nodes permanently, it loses “quorum” which means that the underlying etcd cluster becomes read-only. This means that scaling up and down is not possible and changes to shared config cannot be made. It also means that the steps for removing a single failed node won’t work. This section describes how to recover from this state.

In this example, your initial cluster consists of servers A, B, C, D, E and F. D, E and F die permanently, and A, B and C enter a read-only state (because they lack quorum). Recent changes may have been permanently lost at this point (if they were not replicated to the surviving nodes).

You should follow this process completely - the behaviour is unspecified if this process is started but not completed. It is always safe to restart this process from the beginning (for example, if you encounter an error partway through).

To recover from this state:

• stop etcd on A, B and C by running sudo monit stop -g etcd
• create a new cluster, only on A, by:
  • editing etcd_cluster in /etc/clearwater/local_config to just contain A’s IP (e.g. etcd_cluster=10.0.0.1)
  • running sudo service clearwater-etcd force-new-cluster. This will warn that this is dangerous and should only be run during this process; choose to proceed.
  • running clearwater-etcdctl member list to check that the cluster only has A in
  • running clearwater-etcdctl cluster-health to check that the cluster is healthy
  • running clearwater-etcdctl get configuration/shared_config to check that the data is safe.
  • running sudo monit monitor -g etcd to put etcd back under monit control
• get B to join that cluster by:
  • editing etcd_cluster in /etc/clearwater/local_config to just contain A’s IP (e.g. etcd_cluster=10.0.0.1)
  • running sudo service clearwater-etcd force-decommission. This will warn that this is dangerous and offer the chance to cancel; do not cancel.
  • running sudo monit monitor -g etcd.
• get C to join that cluster by following the same steps as for B:
  • editing etcd_cluster in /etc/clearwater/local_config to just contain A’s IP (e.g. etcd_cluster=10.0.0.1)
  • running sudo service clearwater-etcd force-decommission. This will warn that this is dangerous and should only be run during this process; choose to proceed.
  • running sudo monit monitor -g etcd.
• check that the cluster is now OK by doing the following on A:
  • running clearwater-etcdctl member list to check that the cluster now has A, B and C in
  • running clearwater-etcdctl cluster-health to check that the cluster is healthy
  • running clearwater-etcdctl get clearwater/<site_name>/configuration/shared_config to check that the data is safe. The site_name is set in `local_config <http://clearwater.readthedocs.org/en/stable/Manual_Install/index.html#create-the-per-node-configuration>`__ if the deployment is geographically redundant, and defaults to site1 if unset.
• log on to A. For each of D, E and F follow the instructions in Removing a Node From a Data Store.

Removing a Node From a Data Store¶

The mark_node_failed script can be used to remove a failed node from a back-end data store. You will need to know the type of the failed node (e.g. “sprout”) and its IP address. To remove the failed node log onto a working node in the same site and run the following commands (depending on the failed node’s type). If you are removing multiple nodes simultaneously, ensure that you run the mark_node_failed scripts for each store type simultaneously (e.g. for multiple sprout removal, mark all failed nodes for memcached simultaneously first, and then mark all failed nodes for chronos).

If you cannot log into a working node in the same site (e.g. because an entire geographically redundant site has been lost), you can use a working node in the other site, but in this case you must run /usr/share/clearwater/clearwater-cluster-manager/scripts/mark_remote_node_failed instead of /usr/share/clearwater/clearwater-cluster-manager/scripts/mark_node_failed.

If you are using separate signaling and management networks, you must use the signaling IP address of the failed node as the failed node IP in the commands below.

sudo /usr/share/clearwater/clearwater-cluster-manager/scripts/mark_node_failed "sprout" "memcached" <failed node IP>
sudo /usr/share/clearwater/clearwater-cluster-manager/scripts/mark_node_failed "sprout" "chronos" <failed node IP>

sudo /usr/share/clearwater/clearwater-cluster-manager/scripts/mark_node_failed "homestead" "cassandra" <failed node IP>

sudo /usr/share/clearwater/clearwater-cluster-manager/scripts/mark_node_failed "homer" "cassandra" <failed node IP>

sudo /usr/share/clearwater/clearwater-cluster-manager/scripts/mark_node_failed "ralf" "chronos" <failed node IP>
sudo /usr/share/clearwater/clearwater-cluster-manager/scripts/mark_node_failed "ralf" "memcached" <failed node IP>

sudo /usr/share/clearwater/clearwater-cluster-manager/scripts/mark_node_failed "memento" "cassandra" <failed node IP>
sudo /usr/share/clearwater/clearwater-cluster-manager/scripts/mark_node_failed "memento" "memcached" <failed node IP>

Complete Site Failure¶

In a geographically redundant deployment, you may encounter the situation where an entire site has permanently failed (e.g. because the location of that geographic site has been physically destroyed). To recover from this situation:

• If the failed site contained half or more of your nodes, you have lost quorum in your etcd cluster. You should follow the “Multiple Failed Nodes” instructions above to rebuild the etcd cluster, containing only nodes from the surviving site.
• If the failed site contained fewer than half of your nodes, you have not lost quorum in your etcd cluster. You should follow the “Removing a Failed Node” instructions above to remove each failed node from the cluster.

After following the above instructions, you will have removed the nodes in the failed site from etcd, but not from the Cassandra/Chronos/Memcached datastore clusters. To do this, follow the “Removing a Node From a Data Store” instructions above for each failed node, using the mark_remote_node_failed script instead of the mark_node_failed script.

You should now have a working single-site cluster, which can continue to run as a single site, or be safely paired with a new remote site.

Naming¶

• Class names are CamelCase, beginning with a capital letter.
• Constant names are ALL_CAPITALS with underscores separating words.
• Method, method parameter and local variable names are lowercase with underscores separating words, e.g. method_name.
• Member variable names are lowercase with underscores separating words and also begin with an underscore, e.g. _member_variable.

C++ Feature Use¶

• We assume a C++11-compliant compiler, so C++11 features are allowed with some exceptions.
• No use of auto type declarations.
• No use of using namespace ..., if the namespace is particularly lengthy, consider using namespace aliasing (e.g. namespace po = boost::program_options).
• Avoid using Boost (or similar) libraries that return special library-specific pointers, to minimize “infection” of the code-base. Consider using the C++11 equivalents instead.

Formatting¶

C++ code contributed to Clearwater should be formatted according to the following conventions:

• Braces on a separate line from function definitions, if statements, etc.
• Two-space indentation
• Pointer operators attached to the variable type (i.e. int* foo rather than int *foo)
• if blocks must be surrounded by braces

For example:

if (x)
       int *foo = do_something();

will be replaced with

if (x)
{
 int* foo = do_something();
}

It’s possible to fix up some code automatically using astyle, with the options astyle --style=ansi -s2 -M80 -O -G -k1 -j -o. This fixes up a lot of the most common errors (brace style, indentation, overly long lines), but isn’t perfect - there are some cases where breaking the rules makes the code clearer, and some edge cases (e.g. around switch statements and casts on multiple lines) where our style doesn’t always match astyle’s.

Language Features¶

• Use of the auto keyword is forbidden.

Commenting¶

Where it is necessary to document the interface of classes, this should be done with Doxygen-style comments - three slashes and appropriate @param and @returns tags.

/// Apply first AS (if any) to initial request.
//
// See 3GPP TS 23.218, especially s5.2 and s6, for an overview of how
// this works, and 3GPP TS 24.229 s5.4.3.2 and s5.4.3.3 for
// step-by-step details.
//
// @Returns whether processing should stop, continue, or skip to the end.
AsChainLink::Disposition
AsChainLink::on_initial_request(CallServices* call_services,

Formatting:¶

• Use UTF-8 encoding in your source files.
• Use 2 space indent, no tabs.
• Use Unix-style line endings, including on the last line of the file.
• Use spaces around operators, after commas, colons and semicolons, around { and before }.
• No spaces after (, [ and before ], ).
• Prefer postfix modifiers (if, unless, rescue) when possible.
• Indent when as deep as case then indent the contents one step more.
• Use an empty line before the return value of a method (unless it only has one line), and an empty line between defs.
• Use Yard and its conventions for API documentation. Don’t put an empty line between the comment block and the definition.
• Use empty lines to break up a long method into logical paragraphs.
• Keep lines shorter than 80 characters.
• Avoid trailing whitespace.

Syntax:¶

• Use def with parentheses when there are arguments.
• Conversely, avoid parentheses when there are none.
• Never use for, unless you exactly know why. Prefer each or loop.
• Never use then, a newline is sufficient.
• Prefer words to symbols.
• and and or in place of && and ||
• not in place of !
• Avoid ?:, use if (remember: if returns a value, use it).
• Avoid if not, use unless.
• Suppress superfluous parentheses when calling methods, unless the method has side-effects.
• Prefer do...end over {...} for multi-line blocks.
• Prefer {...} over do...end for single-line blocks.
• Avoid chaining function calls over multiple lines (implying, use {...} for chained functions.
• Avoid return where not required.
• Avoid line continuation (\) where not required.
• Using the return value of = is okay.
• if v = array.grep(/foo/)
• Use ||= freely for memoization.
• When using regexps, freely use =~, -9, :math:`~, ` and $` when needed.
• Prefer symbols (:name) to strings where applicable.