Welcome to computer’s documentation!

CLI-Computer is a library created by the Climate Impact Lab team.

This code is open source and available on github.

We can add any additional information about the library here. Please make suggestions.

Intro and Setup

This is where the introduction to our libraries will go. Ideally, this will be a short overview of the library, with one or two very simple use cases. It will, hopefully, answer the question: ‘Why should I use this tool?’

Description

Paths

Servers have collections of roots, used to abstract away the location of files. Most methods take a path argument, which can take three forms:

  • Starting with a /: An absolute path, not relative to root.
  • Containing a /: The root is up to the first slash, and the remaining is a subpath.
  • Not containing a /: Just a root directory.

Available Servers

Abstract Servers

  • DiskServer: A server with a unique ID and a collection of named file root directories.
  • SizelessServer (DiskServer): The main top-level server class, able to transfer data and run commands. Does not monitor or limit data usage (hence, “sizeless”).
  • SizelessConnectableServer (SizelessServer): A server that has a protocol for connecting to it.
  • SizelessLinuxServer (SizelessConnectableServer): A standard Linux server, with everything except run_command implemented.
  • ParamikoServer (SizelessLinuxServer): A server connected through ssh, using the Paramiko interface. The self.session variable needs to be set in the connect method before these functions will work.

Concrete Servers

  • LocalServer (SizelessServer): A server representing the local machine, using python commands.
  • LoginServer (ParamikoServer): A server using a Paramiko ssh connection, with a normal password login.

Installation Guide

git clone https://github.com/ClimateImpactLab/computer

The API Documentation / Guide

If you are looking for information on a specific function, class, or method, this part of the documentation is for you.

Computer library

computer.linux_server module

class computer.linux_server.RemoteLinuxProcess(server, pid, logfile)[source]

Bases: computer.process.RemoteProcess

clean()[source]

Remove the log file

halt()[source]

Temporarily stop a process.

is_running()[source]

Returns bool.

kill()[source]

Kill process if running

resume()[source]

Resume a halted process.

class computer.linux_server.SizelessLinuxServer(utup, cpus, roots, credentials)[source]

Bases: computer.server.SizelessConnectableServer

active_processes(procname)[source]
check_connection()[source]
cwd(path)[source]
list_disk(path)[source]

path: subdirectory

read_file(filepath=None)[source]

Returns string.

start_process(command, path=None)[source]

computer.local_server module

class computer.local_server.LocalProcess(server, proc, stdout, logfile)[source]

Bases: computer.process.SingleCPUProcess

clean()[source]
halt()[source]

Temporarily stop a process.

is_running()[source]
kill()[source]
resume()[source]

Resume a halted process.

class computer.local_server.LocalServer(utup, cpus, roots)[source]

Bases: computer.server.SizelessServer

has_file(path)[source]
list_disk(path=None)[source]
read_file(path)[source]
run_command(command, path=None)[source]
start_process(command, logfile, path=None)[source]

computer.login_server module

class computer.login_server.LoginServer(utup, cpus, roots, credentials)[source]

Bases: computer.paramiko_server.ParamikoServer

connect()[source]
get_cpu_util()[source]

implement the idea of getting cpu_percents in psutil: 1. Get “cpu times” from /proc/stat once and wait a second before getting another reading of those. 2. Compare the two readings and define the utilization rate as:

(Non-idle and Non-iowait CPU time) / (total CPU time)

http://stackoverflow.com/questions/23367857/accurate-calculation-of-cpu-usage-given-in-percentage-in-linux

computer.manager module

class computer.manager.DiskDependency(root, path, size)[source]

Bases: object

satisfy(server)[source]
satisfy_cost(server)[source]
class computer.manager.Manager[source]

Bases: object

add_server(server, priority)[source]
prepare_server(dependencies, required=False)[source]

Finds a server with resources and minimal transfer requirements; transfers additional requirements as needed. Returns server.

process_check()[source]
run_command(root, path, command, dependencies)[source]
select_server(dependencies, required)[source]
server_check()[source]
submit(root, path, command, dependencies, callback)[source]

Does not return process (so we can manage). callback only called on check().

computer.paramiko_server module

class computer.paramiko_server.ParamikoServer(utup, cpus, roots, credentials)[source]

Bases: computer.linux_server.SizelessLinuxServer

disconnect()[source]
receive()[source]
receive_all()[source]
receive_each()[source]
run_command(command, root=None, path=None)[source]

Returns (output, error) as strings.

run_command_each(command, root=None, path=None)[source]

computer.process module

class computer.process.RemoteProcess(server, pid, logfile)[source]

Bases: computer.process.SingleCPUProcess

class computer.process.SingleCPUProcess(server, logfile=None)[source]

Bases: object

clean()[source]

Remove the log file

close()[source]
is_running()[source]

Returns bool.

kill()[source]

Kill process if running

computer.server module

class computer.server.DiskServer(utup, roots)[source]

Bases: object

fullpath(path)[source]
splitpath(path)[source]
class computer.server.SizelessConnectableServer(utup, cpus, roots, credentials)[source]

Bases: computer.server.SizelessServer

check_connection()[source]
connect()[source]
disconnect()[source]
class computer.server.SizelessServer(utup, cpus, roots)[source]

Bases: computer.server.DiskServer

has_file(filepath)[source]

path: subdirectory

list_disk(path)[source]

path: subdirectory

pull_disk(path, localds)[source]

path: subdirectory local: local DiskServer

push_disk(path, localds)[source]

path: subdirectory local: local DiskServer

read_file(filepath)[source]

Returns string.

run_command(command, path=None)[source]

Returns (output, error) as strings.

start_process(command, path=None)[source]

Returns process.

computer.task module

class computer.task.LambdaDependency(desc, check, install_func=None, atrun_func=None)[source]

Bases: computer.task.OnceDependency

atrun(server)[source]
install(server)[source]
raiseunimplemented()[source]
satisfied(server)[source]
class computer.task.LambdaTask(dependencies, perform)[source]

Bases: computer.task.SingleCPUTask

selfrun(server)[source]
class computer.task.OnceDependency(desc)[source]

Bases: object

satisfied(server)[source]
class computer.task.SingleCPUTask(dependencies)[source]

Bases: object

fullrun(server, doublecheck=True)[source]
prepare(server)[source]
selfrun(server)[source]

The Developers Guide

If you want to contribute to the project, this part of the documentation is for you.

Contributor’s Guide

This document lays out guidelines and advice for contributing to this project. If you’re thinking of contributing, please start by reading this document and getting a feel for how contributing to this project works. If you have any questions, feel free to reach out to either James Rising, Mike Delgado, or Justin Simcock.

When contributing code, you’ll want to follow this checklist:

  1. Fork the repository on GitHub.
  2. Run the tests to confirm they all pass on your system. If they don’t, you’ll need to investigate why they fail. If you’re unable to diagnose this yourself, raise it as a bug report by following the guidelines in this document: Bug Reports.
  3. Write tests that demonstrate your bug or feature. Ensure that they fail.
  4. Make your change.
  5. Run the entire test suite again, confirming that all tests pass including the ones you just added.
  6. Send a GitHub Pull Request to the main repository’s master branch. GitHub Pull Requests are the expected method of code collaboration on this project.

Bug Reports

Bug reports are hugely important! Before you raise one, though, please check through the GitHub issues, both open and closed, to confirm that the bug hasn’t been reported before. Duplicate bug reports are a huge drain on the time of other contributors, and should be avoided as much as possible.

Indices and tables