Welcome to Junxian’s Documentation!¶
[Contents]¶
Python Phthonic¶
Overview¶
Pythonic means code that doesn’t just get the syntax right but that follows the conventions of the python community and uses the lanuage in the way it’s intended to be used.
>>> import this
The Zen of Python, by Tim Peters
Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated. Flat is better than nested. Sparse is better than dense. Readability counts. Special cases aren’t special enough to break the rules. Although practicality beats purity. Errors should never pass silently. Unless explicitly silenced. In the face of ambiguity, refuse the temptation to guess. There should be one– and preferably only one –obvious way to do it. Although that way may not be obvious at first unless you’re Dutch. Now is better than never. Although never is often better than right now. If the implementation is hard to explain, it’s a bad idea. If the implementation is easy to explain, it may be a good idea. Namespaces are one honking great idea – let’s do more of those!
PEP8 first¶
Python Built-in Functions¶
https://docs.python.org/2.7/library/functions.html
>>>print dir(__builtins__) # Or: print dir(sys.modules['__builtin__'])
['ArithmeticError', 'AssertionError', 'AttributeError', 'BaseException', 'BufferError', 'BytesWarning', 'DeprecationWarning', 'EOFError', 'Ellipsis', 'EnvironmentError', 'Exception', 'False', 'FloatingPointError', 'FutureWarning', 'GeneratorExit', 'IOError', 'ImportError', 'ImportWarning', 'IndentationError', 'IndexError', 'KeyError', 'KeyboardInterrupt', 'LookupError', 'MemoryError', 'NameError', 'None', 'NotImplemented', 'NotImplementedError', 'OSError', 'OverflowError', 'PendingDeprecationWarning', 'ReferenceError', 'RuntimeError', 'RuntimeWarning', 'StandardError', 'StopIteration', 'SyntaxError', 'SyntaxWarning', 'SystemError', 'SystemExit', 'TabError', 'True', 'TypeError', 'UnboundLocalError', 'UnicodeDecodeError', 'UnicodeEncodeError', 'UnicodeError', 'UnicodeTranslateError', 'UnicodeWarning', 'UserWarning', 'ValueError', 'Warning', 'ZeroDivisionError', '_', '__debug__', '__doc__', '__import__', '__name__', '__package__', 'abs', 'all', 'any', 'apply', 'basestring', 'bin', 'bool', 'buffer', 'bytearray', 'bytes', 'callable', 'chr', 'classmethod', 'cmp', 'coerce', 'compile', 'complex', 'copyright', 'credits', 'delattr', 'dict', 'dir', 'divmod', 'enumerate', 'eval', 'execfile', 'exit', 'file', 'filter', 'float', 'format', 'frozenset', 'getattr', 'globals', 'hasattr', 'hash', 'help', 'hex', 'id', 'input', 'int', 'intern', 'isinstance', 'issubclass', 'iter', 'len', 'license', 'list', 'locals', 'long', 'map', 'max', 'memoryview', 'min', 'next', 'object', 'oct', 'open', 'ord', 'pow', 'print', 'property', 'quit', 'range', 'raw_input', 'reduce', 'reload', 'repr', 'reversed', 'round', 'set', 'setattr', 'slice', 'sorted', 'staticmethod', 'str', 'sum', 'super', 'tuple', 'type', 'unichr', 'unicode', 'vars', 'xrange', 'zip']
math¶
abs(x)
divmod(a, b)
max()
min()
pow(x, y)
range([start,] stop [,step]) # return list
xrange([start,] stop [,step]) # iterable
round(x, [,n])
sum([item])
logical¶
all(s) # True if all is True
any(s) # True if any is True
cmp(x, y)
isinstance(obj, class) # Trure if obj is a isinstance of class
sorted()
reversed(sequence)
conversion¶
repr(o)
ascii(x) #python 3
str()
bin()
float()
hex()
int() # equal to math.floor()
long()
ord(c)
chr(i)
set([item])
I/O¶
input()
raw_input()
print()
iteration¶
enumerate(iterable)
>>> for i, c in enumerate("Like"):
... print "S[%d]=%c" %(i, c)
...
S[0]=L
S[1]=i
S[2]=k
S[3]=e
s = 'I like Python'
>>> reversed(s)
<reversed object at 0x7f2383001850>
>>> list(reversed(s))
['n', 'o', 'h', 't', 'y', 'P', ' ', 'e', 'k', 'i', 'l', ' ', 'I']
>>> list(sorted(s))
[' ', ' ', 'I', 'P', 'e', 'h', 'i', 'k', 'l', 'n', 'o', 't', 'y']
others¶
map(function, sequence, ...) # list that one to one return after map
filter(function, sequence) # list that all ture values
reduce(function, sequence[, initial]) # final one result after function(lastreturn/initial, next)
zip(a, b)
>>> names = ["Alex", "John", "Mike"]
>>> ages = [15, 36, 42]
>>> zip(names, ages)
[('Alex', 15), ('John', 36), ('Mike', 42)]
unzip()
>>> zip(*c) #c=[(1, 4), (2, 5), (3, 6)]
[(1, 2, 3), (4, 5, 6)]
Python Data Modules¶
Introduction¶
Objects are Python’s abstraction for data. All data in a Python program is represented by objects or by relations between objects.
- Data Modules:
- number, string, sequence, mapping(dictionary), set/frozenset
- Build-in Sequences:
- list, tuple, string, unicode, buffer, xrange
The start index for all Sequences is 0, and -1 is means the last one.
Immutable sequences: string, unicode, tuple Mutable sequences: list, byte array
(1, ) # If there is only one item in the tuple
The key of dictionary must can not be changed, for example:
int, float, string, tuple, frozenset.
But the value of dictionary can be any data type.
Note
- The differences between reference and copy
- See also pickle module
Other operation supported by sequences¶
https://docs.python.org/3/library/stdtypes.html#sequence-types-list-tuple-range
Slicing:
>>> i=3
>>> numbers=[0,1,2,3,4,5,6,7,8,9]
>>> numbers[i] # numbers[3] = 3
3
>>> numbers[-3] # numbers[10-3] = numbers[7]
7
>>> numbers[3:6] # from 3 to 6, but not include 6
[3, 4, 5]
>>> numbers[-3:-1] # from -3 to -1, the same as numbers[7:9]
[7, 8]
>>> numbers[3:] # from 3 to the end
[3, 4, 5, 6, 7, 8, 9]
>>> numbers[:3] # from the beginning to 3, doesn't include 3
[0, 1, 2]
>>> numbers[-3:] # the last 3 items
[7, 8, 9]
>>> numbers[:-3] # from the beginning to the last third one
[0, 1, 2, 3, 4, 5, 6]
>>> numbers[::2] # step is 2
[0, 2, 4, 6, 8]
>>> numbers[::-1] # step is -1, which means reverse the sequence
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
>>> numbers[-1:-10:-1]
[9, 8, 7, 6, 5, 4, 3, 2, 1]
>>> numbers[9:0:-1]
[9, 8, 7, 6, 5, 4, 3, 2, 1]
>>> numbers=[0,1,2,3,4,5,6,7,8,9]
>>> numbers[-2:] = [-2, -1] # change value
>>> numbers
[0, 1, 2, 3, 4, 5, 6, 7, -2, -1]
>>> numbers[2:2] = [20, 21] # insert
>>> numbers
[0, 1, 20, 21, 2, 3, 4, 5, 6, 7, -2, -1]
>>> numbers[-2:] = [-3, -2, -1] # change value and append
>>> numbers
[0, 1, 20, 21, 2, 3, 4, 5, 6, 7, -3, -2, -1]
>>> numbers=[0,1,2,3,4,5,6,7,8,9]
>>> numbers[0:9:2] = [10, 12, 14, 16, 18] # change value in specified step
>>> numbers
[10, 1, 12, 3, 14, 5, 16, 7, 18, 9]
Append:
>>> numbers=[0,1,2,3,4,5,6,7,8,9]
>>> numbers + numbers
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Times:
>>> numbers=[0,1,2,3,4,5,6,7,8,9]
>>> numbers * 2
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Change value:
>>> numbers[0] = 10
>>> numbers
[10, 1, 2, 3, 4, 5, 6, 7, 8, 9]
Check item existence:
>>> 2 in numbers
True
>>> 'P' in "Python"
True
Build in functions:
>>> max("Python")
'y'
>>> max(numbers)
9
>>> min(numbers)
0
>>> len(numbers)
10
Delete:
>>> del numbers[0]
>>> numbers
[1, 2, 3, 4, 5, 6, 7, 8, 9]
Operations supported by string¶
String is a special kind of sequence, so most of the above sequence operations are valid to string, such as slicing, ‘+’, ‘*’, and build-in functions ‘len’, ‘cmp’, for example:
s[::-1] # reverse the string
str="This is a string"" and some others together"
str1 = 'strcat'
str2 = 'append'
str1 += str2
# basestring is the base calse of all string and unicode type:
# The following way can check whether the it's a string.
isinstance(myobj, basesring)
Format a string¶
Usage: format_string % tuple/dictionary https://docs.python.org/2/library/stdtypes.html#string-formatting
Format string: %[[+/-]width][.precision]type
Examples:
%s, %8d, %-16s, %10.2f, %010.2f, %+5d
print "My name is %s, age is %d\n" % (name, age)
d = {'name': 'Alex', 'age': 42}
print "Name is %(name)s, age is %d" % d
Usage: Template() https://docs.python.org/2/library/string.html#template-strings
Examples:
>>> s = Template("My name is $name, age is $age")
>>> s.substitute(name="John", age=18) #使用变量赋值
'My name is John, age is 18'
>>> d = {'name':'John', 'age':18}
>>> s.substitute(d) #使用字典赋值
'My name is John, age is 18'
Usage: format() https://docs.python.org/2/library/string.html#custom-string-format_string
- Examples::
>>> '{}, {}, {}'.format('arg0', 'arg1', 'arg2') 'arg0, arg1, arg2'
>>> '{0}, {1}, {2}'.format('arg0', 'arg1', 'arg2') 'arg0, arg1, arg2'
>>> '{2}, {1}, {1}'.format('arg0', 'arg1', 'arg2') 'arg2, arg1, arg1'
>>> '{0}, {arg1}, {arg2}'.format('arg0', arg1='arg1', arg2='arg2') 'arg0, arg1, arg2'
>>> '{arg0}, {0}, {arg2}'.format('arg1', arg0='arg0', arg2='arg2') # named arguments must be at last. 'arg0, arg1, arg2'
>>> config['conf0':'arg0', 'conf1':'arg1'] >>> '{0[conf0]}, {0[conf1]}, {1}'.format(config, 'arg2') 'arg0, arg1, arg2'
>>> '|{0[conf0]:>10}|{0[conf1]:<10}|{1:^10}|'.format(config, 'arg2') # alignment '| arg0|arg1 | arg2 |'
>>> '{0:.{1}f}'.format(1/3.0, 4) '0.3333'
Build-in functions for string¶
https://docs.python.org/2/library/string.html#string-functions
Note
As string is not changable, so all the return value is the new copy.
>>> dir(string)
>>> dir(s)
Build-in Functions:
S.find(substr, [start, [end]])
S.rfind(substr, [start, [end]])
S.index(substr, [start, [end]]) # similar to find, but maybe index exception.
S.rindex(substr, [start, [end]])
S.count(substr, [start, [end]])
S.replace(oldstr, newstr, [count])
S.strip([chars])
S.lstrip([chars])
S.rstrip([chars])
S.expandtabs([tabsize])
S.lower()
S.upper()
S.swapcase()
S.capitalize()
S.title()
S.partition(substr)
S.rpartition(substr)
S.split([sep, [maxsplit]])
S.rsplit([sep, [maxsplit]])
S.splitlines([keepends])
S.join(sequence)
S.isalnum()
S.isalpha()
S.isdigit()
S.islower()
S.isupper()
S.isspace()
S.istitle()
S.startswith(prefix [, start, end])
S.endswith(suffix [, start, end])
S.center(width[, fillchar])
S.ljust(width[, fillchar])
S.rjust(width[, fillchar])
S.zfill(width)
S.translate(table[,deletechars])
S.encode([encoding,[errors]])
S.decode([encoding,[errors]])
string.atoi(s[,base])
string.atol(s[,base])
string.atof(s[,base])
Examples:
>>> st = string.maketrans("0123456789", "abcdefghij")
>>> "001".translate(st)
'aab'
allchars = string.maketrans('', '') # nothing is changed
keep = 'abcde'
alldel = allchars.translate(allchars, keep) # all execpt characters in keep
s.translater(allchars , alldel) # all in keep
>>> se = "007".encode('base64')
>>> se.decode('base64')
'007'
re library¶
https://docs.python.org/2/library/re.html
re.search(pattern, string, flags=0)
re.match(pattern, string, flags=0)
re.fullmatch(pattern, string, flags=0)
re.split(pattern, string, maxsplit=0, flags=0)
re.findall(pattern, string, flags=0)
re.finditer(pattern, string, flags=0)
re.sub(pattern, repl, string, count=0, flags=0)
re.escape(string)
textwrap module¶
https://docs.python.org/3/library/textwrap.html
textwrap.wrap(text, width=70, **kwargs)
textwrap.fill(text, width=70, **kwargs)
textwrap.shorten(text, width, **kwargs)
textwrap.dedent(text)
textwrap.indent
class textwrap.TextWrapper(**kwargs)
Sequences type cast¶
Integer to Binary string:
# Use built-in function bin()
bin(n)[2:]
Sting to list:
>>> list("Python")
['P', 'y', 't', 'h', 'o', 'n']
String to tuple:
>>> tuple("Python")
('P', 'y', 't', 'h', 'o', 'n')
List to string:
# Items for join function must be strings or characers
>>> ''.join(['P', 'y', 't', 'h', 'o', 'n'])
'Python'
List to set:
# unique the list after from list to set and back to list
list(set(l))
Dictionary to set:
>>> print set({'Hacker' : 'DOSHI', 'Rank' : 616 })
set(['Hacker', 'Rank'])
Map to set:
s = set(map(int, input().strip().split()))
List enumerate:
for idx, value in enumerate(['Spring', 'Summer', 'Fall', 'Winter'])
List build-in functions¶
append:
>>> numbers.append(10)
>>> numbers
[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
count:
>>> numbers[10]=9
>>> numbers
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 9]
>>> numbers.count(9) # count the number of value '9'
2
extend:
>>> numbers=[0,1,2,3,4,5,6,7,8,9]
>>> numbers.extend([10, 11, 12, 13]) # similar to a = a + b
>>> numbers
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]
Note
a + b will not change a，but a.extend(b) will change a
index:
>>> numbers=[0,1,2,3,4,5,6,7,8,9]
>>> numbers.index(4) # find the first one and return the index
4
insert:
>>> numbers.insert(4, 41)
>>> numbers
[0, 1, 2, 3, 41, 4, 5, 6, 7, 8, 9]
remove:
>>> numbers.remove(41) # remove the first one which is found
>>> numbers
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
pop:
#pop(n) remove and return the Nth item, default is '-1'
>>> numbers.pop()
9
>>> numbers
[0, 1, 2, 3, 4, 5, 6, 7, 8]
reverse:
>>> numbers.reverse()
>>> numbers
[9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
sort:
>>> numbers.sort() # no return value, just change the list.
>>> numbers
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> name=["John", "Stephanie", "Hasan"]
>>> name.sort(cmp) # use build-in function 'cmp' as compare function
>>> name
['Hasan', 'John', 'Stephanie']
>>> name.sort(key=len) # compare the len
>>> name
['John', 'Hasan', 'Stephanie']
>>> name.sort(key=len, reverse=True) # sort and also reverse the result
>>> name
['Stephanie', 'Hasan', 'John']
Operations supported by dictionary¶
https://docs.python.org/3/library/stdtypes.html#mapping-types-dict
Create:
d = {'name': 'Alex', 'age': 42}
d = dict(name='Alex', age=42)
items = [('name', 'Alex'), ('age', 42)]
d = dict(items)
Add/Set:
d['name'] = 'Alisa'
d['interest'] = 'reading'
Delete:
del d['interest']
Length:
len(d)
Member check:
‘name’ in d #true
Dictionary to Sting:
>>> str(d)
"{'age': 42, 'name': 'Alex'}"
Format by key-value paring in dictionary:
>>> print "Name is %(name)s, age is %(age)s" % d
Name is Alex, age is 42
Dictionary build-in functions¶
>>> dir(d)
['__class__', '__cmp__', '__contains__', '__delattr__', '__delitem__',
'__doc__', '__eq__', '__format__', '__ge__', '__getattribute__',
'__getitem__', '__gt__', '__hash__', '__init__', '__iter__', '__le__',
'__len__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__',
'__repr__', '__setattr__', '__setitem__', '__sizeof__', '__str__',
'__subclasshook__', 'clear', 'copy', 'fromkeys', 'get', 'has_key', 'items',
'iteritems', 'iterkeys', 'itervalues', 'keys', 'pop', 'popitem', 'setdefault',
'update', 'values', 'viewitems', 'viewkeys', 'viewvalues']
clear:
d.clear() # clear all key-value, no return value
copy:
>>> d['interest'] = ["reading", "chese", "movie"]
>>> d
{'age': 42, 'name': 'Alex', 'interest': ['reading', 'chese', 'movie']}
>>> d1 = d.copy() # just add the reference
>>> d1['name'] = 'John'
>>> d1['interest'].remove('movie')
>>> d
{'age': 42, 'name': 'Alex', 'interest': ['reading', 'chese']}
>>> d1
{'age': 42, 'name': 'John', 'interest': ['reading', 'chese']}
deepcopy:
>>> from copy import deepcopy
>>> d2 = deepcopy(d) # deepcopy from the copy module
>>> d2['interest'].append('movie')
>>> d
{'age': 42, 'name': 'Alex', 'interest': ['reading', 'chese']}
>>> d2
{'age': 42, 'name': 'Alex', 'interest': ['reading', 'chese', 'movie']}
fromkeys:
>>> d1 = d.fromkeys(['name', 'age'], 'Unknown')
>>> d1
{'age': 'Unknown', 'name': 'Unknown'}
#fromkeys is a dict class static method
>>> d1 = dict.fromkeys(['name', 'age'], 'Unknown')
>>> d1
{'age': 'Unknown', 'name': 'Unknown'}
get:
>>> print d.get('name')
Alex
>>> print d.get('country') # return None instead of exception if doesn't exist.
None
>>> print d['country']
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
KeyError: 'country'
>>> print d.get('country', 'Unkown') # provide a default value
Unkown
setdefault:
# It's to get value, but if key doesn't exist, return the default value.
# Will also add the key and the default value as new key-value paring.
>>> d.setdefault('interest', ['reading', 'chese'])
['reading', 'chese']
has_key:
>>> d.has_key('name')
True
items:
>>> d.items() # return the list of all the key-value tuple
[('age', 42), ('name', 'Alex'), ('interest', ['reading', 'chese'])]
>>> for key, value in d.items():
... print "[%s]:%s" %(key, value)
...
[age]:42
[name]:Alex
[interest]:['reading', 'chese']
iteritems:
>>> it = d.iteritems() # return the iterable items
>>> it
<dictionary-itemiterator object at 0x7f2382fae2b8>
>>> list(it)
[('age', 42), ('name', 'Alex'), ('interest', ['reading', 'chese'])]
keys:
>>> d.keys() # return the list of all the keys only
['age', 'name', 'interest']
iterkeys:
>>> d.iterkeys() # return a iterable keys
<dictionary-keyiterator object at 0x7f2382fae310>
values:
>>> d.values() # return the list of all the values only
[42, 'Alex', ['reading', 'chese']]
itervalues:
>>> d.itervalues() # return the iterable values
<dictionary-valueiterator object at 0x7f2382fae310>
pop:
# Remove the key-value specified by this key in the dictionary.
# And return the value of this key
>>> d.pop('interest')
['reading', 'chese']
>>> d
{'age': 42, 'name': 'Alex'}
# provide default return value if key doesn't exist
>>> d.pop('interest', 'No this key')
'No this key'
popitem:
# The same as pop, but using a random key instead of specified one
# Also return the value of this randome key-value paring.
>>>d.popitem()
update:
# Update if the same key exist, add new key-value paring if not.
>>> d1 = {'name': 'John', 'country': 'USA'}
>>> d.update(d1)
>>> d
{'country': 'USA', 'age': 42, 'name': 'John', 'interest': ['reading', 'chese']}
Operations supported by set¶
https://docs.python.org/3/library/stdtypes.html#set-types-set-frozenset
>>> x = set('python')
>>> y = {'p', 'o', 'i', 'n', 't'}
>>> x & y # x.intersection(y), in both x and y
set(['p', 't', 'o', 'n'])
>>> x | y # x.union(y), in x or y
set(['i', 'h', 'o', 'n', 'p', 't', 'y'])
>>> x - y # or x.difference(y), in x but ont in y
set(['y', 'h'])
>>> x ^ y #x.symmetric_difference(y), in x or y but not both
set(['i', 'h', 'y'])
>>> s = set()
>>> s.add('test') # add 'test' into set
>>> s.update('test') # add 't', 'e', 's', 't' into test
>>> s.update({1, 6}, [5, 3]) # add 1, 6, 5 ,3 into set
>>> s.discard(1) # remove 1 from set if exits.
>>> s.remove(1) # remove 1 from set and raise KeyError if doesn't exit.
Define and use of Enum¶
enum is standard after python 3.4, for older version, please try “pip install enum”.
from enum import Enum, IntEnum, unique
try:
@unique
class WEEKDAY(Enum):
MON = 1
TUS = 2
WEN = 3
THU = 4
FRI = 1
except ValueError as e:
print(e)
duplicate values found in <enum 'WEEKDAY'>: FRI -> MON
try:
class Color(IntEnum):
RED = 0
GREEN = 1
BLUE = 'b'
except ValueError as e:
print(e)
invalid literal for int() with base 10: 'b'
red = class(0)
print(red is Color.R) # True
print(red == 0) # False
Container datatypes¶
collections module: https://docs.python.org/3/library/collections.html
This module implements specialized container datatypes providing alternatives to Python’s general purpose built-in containers, dict, list, set, and tuple.
For example:
- namedtuple() factory function for creating tuple subclasses with named fields
- deque list-like container with fast appends and pops on either end
- ChainMap dict-like class for creating a single view of multiple mappings
- Counter dict subclass for counting hashable objects
- OrderedDict dict subclass that remembers the order entries were added
- defaultdict dict subclass that calls a factory function to supply missing values
- UserDict wrapper around dictionary objects for easier dict subclassing
- UserList wrapper around list objects for easier list subclassing
- UserString wrapper around string objects for easier string subclassing
Examples:
from collections import Counter, OrderedDict
class OrderedCounter(Counter, OrderedDict):
pass
[print(*c) for c in OrderedCounter(sorted('bbbaaccde').most_common(3)]
# output: (top 3 occurrence count and sorted when with the same count)
b 3
a 2
c 2
Python Expressions and Operators¶
https://docs.python.org/3/reference/expressions.html
http://www.runoob.com/python/python-operators.html
Overview¶
Arithmetic conversions:
- If either argument is a complex number, the other is converted to complex;
- otherwise, if either argument is a floating point number, the other is converted to floating point;
- otherwise, both must be integers and no conversion is necessary.
The power operator Unary arithmetic and bitwise operations Binary arithmetic operations Shifting operations Binary bitwise operations Comparisons Boolean operations Conditional expressions Lambdas Expression lists Evaluation order Operator precedence
Logical expressions¶
False:
False None 0 "" () [] {}
Examples:
name = input("Please enter your name: ") or 'Unknown'
x = A or B or C or default
x = 1 if 5<6 else 0
List Comprehensions¶
Examples:
[line.rstrip().replace(' ', '#') for line in open('testfile')]
>>> [a**2 + b**2 for (a, b) in zip(range(1,5),range(1,5))]
[2, 8, 18, 32]
[x[0]**2 + x[1]**2 for x in zip(range(1,5),range(1,5))] # zip两个序列
[2, 8, 18, 32]
[i**2 for i in range(0,5) if i > 2] # 附带条件判断的
[a**2 + b**2 for a in range(1,5) for b in range(1,5)] # 两个序列的全排列组合
[2, 5, 10, 17, 5, 8, 13, 20, 10, 13, 18, 25, 17, 20, 25, 32]
l = [(1,2,3), (4,5,6)]
[i for t in l for i in t]
Dictionary Comprehensions¶
# dict([(str(i), i**2) for i in range(0,5)]) >>> {str(i):i**2 for i in range(0,5)} {‘3’: 9, ‘4’: 16, ‘0’: 0, ‘2’: 4, ‘1’: 1}
Generator Expression¶
Examples:
total = sum(num * num for num in xrange(1, 101))
month_codes = dict((fn(i+1), code)
for i, code in enumerate('FGHJKMNQUVXZ')
for fn in (int, str))
Python Statements¶
https://docs.python.org/3/reference/simple_stmts.html
https://docs.python.org/3/reference/compound_stmts.html
Overview¶
Simple statements:
- Expression statements
- Assignment statements
- The assert statement
- The pass statement
- The del statement
- The return statement
- The yield statement
- The raise statement
- The break statement
- The continue statement
- The import statement
- The global statement
- The nonlocal statement
Compound statements:
- The if statement
- The while statement
- The for statement
- The try statement
- The with statement
- Function definitions
- Class definitions
- Coroutines
Assignment statement¶
All assignment expression will not return value like C language.
There is syntax error in this expression: 5 == (x = 5).
All assignment are reference copy, but not real deep copy. For example:
>>> L1 = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
>>> L2 = L1[::2] # L2 = [0, 2, 4, 6, 8]
>>> L1[2] = 20 # L1[2] point to reference of int '20', L2 is still not changed.
>>> L2
[0, 2, 4, 6, 8] # L2[1] is still point to int '2'
>>> L1[2] = [1, 2]
>>> L2 = L1[::2] # Both L1[2] and L2[1] point to the refrence of list [1, 2]
>>> L1[2][0] = 10 # The first value of list [1, 2] is changed to 10
>>> L1
[0, 1, [10, 2], 3, 4, 5, 6, 7, 8, 9]
>>> L2
[0, [10, 2], 4, 6, 8] # So both the value in L1 and L2 are changed, because of the same one reference.
Special assignment examples:
x, y, z = (1, 2, 3)
a,b,c,d = 'test'
a, b = ['111', '222']
a,b = c, d
[a, b] = [c, d]
for (a, b, c) in [(1,2,3), (4,5,6)]: pass
x = y = z = 1
for statement¶
Examples:
for x in List/Tuple/String: ... else: pass
for k in Dict: pass
for (a, b) in [(1, 2), (3, 4), (5, 6)]: pass
for line in open('test.txt'): pass # read one line each time
for line in open('test.txt').readlines(): pass # read all into list
for line in os.popen('ls -1 /'): pass
for i in range(0,10, 2): print L[i]
for l in L[::2]: print l # will copy L[::2] as new list, use more memory than above case.
Note
All object with __next__() method is iteriable, next(o) is equal to o.__next()
with statement¶
Syntax:
with A() as a:
with A() as a , B() as b: # Support this after python 3.1
# will close file automatically
with open(r'/home/test/test.txt') as myfile:
for line in myfile:
print(line)
Any class with __enter__(), __exit__() method can use with/as statement:
- with will call__enter__(), and return to variable behind as.
- __exit() will always be called at the end of the with/as statement.
Example:
class TraceBlock:
def message(self, arg):
print('running %s' %arg)
def __enter__(self):
print('starting trace block')
return self
def __exit__(self, exc_type, exc_value, exc_tb):
if exc_type is None:
print('exit normal')
else:
print('exit with execption %s' % exc_type)
return False
with TraceBlock() as action:
action.message('test1')
print('reached')
with TraceBlock() as action:
action.message('test2')
raise TypeError
print('not reached')
Python Exception and Errors¶
Exception Overview¶
- Some exceptions are not error, such as EOFError, KeyboardInterrupt.
- Catch the exception as exact as possible.
try/except/else/finally Syntax:
try:
...
except <ExceptionName1>:
...
except <ExceptionName2>:
...
else:
...
finally:
...
try/except/else/finally Example:
try:
a = int(raw_input("enter number"))
break
except (NameError, TypeError, RuntimeError):
print "err"
except: # equal to "except Exception:", base Exception is the base class of all exception.
pass
print "test"
raise <instance of class>:
raise IndexError('name')
assert <test>, <data>:
def f(x):
assert x<0, 'x must be negative'
Define New Exception¶
Examples:
import sys
class FormatError(Execption):
def __init__(self, line, file):
self.line = line
self.file = file
try:
raise FormatError(sys._getframe().f_lineno, __file__)
except FormatError as e:
print('Format error at: %s line %d' % (e.file, e.line))
Module: errno¶
Error Code to String:
import errno
import os
os.strerror(errno.E2BIG) # Convert error code E2BIG to string: 'Argument list too long'
print errno.errorcode # see also: linux/include/errno.h
{1: 'EPERM', 2: 'ENOENT', 3: 'ESRCH', 4: 'EINTR', 5: 'EIO', 6: 'ENXIO', 7: 'E2BIG', 8: 'ENOEXEC',
... }
POSIX Error Number:
e2big - argument list too long
eacces - permission denied
eaddrinuse - address already in use
eaddrnotavail - cannot assign requested address
eadv - advertise error
eafnosupport - address family not supported by protocol family
eagain - resource temporarily unavailable
ealign - EALIGN
ealready - operation already in progress
ebade - bad exchange descriptor
ebadf - bad file number
ebadfd - file descriptor in bad state
ebadmsg - not a data message
ebadr - bad request descriptor
ebadrpc - RPC structure is bad
ebadrqc - bad request code
ebadslt - invalid slot
ebfont - bad font file format
ebusy - file busy
echild - no children
echrng - channel number out of range
ecomm - communication error on send
econnaborted - software caused connection abort
econnrefused - connection refused
econnreset - connection reset by peer
edeadlk - resource deadlock avoided
edeadlock - resource deadlock avoided
edestaddrreq - destination address required
edirty - mounting a dirty fs w/o force
edom - math argument out of range
edotdot - cross mount point
edquot - disk quota exceeded
eduppkg - duplicate package name
eexist - file already exists
efault - bad address in system call argument
efbig - file too large
ehostdown - host is down
ehostunreach - host is unreachable
eidrm - identifier removed
einit - initialization error
einprogress - operation now in progress
eintr - interrupted system call
einval - invalid argument
eio - I/O error
eisconn - socket is already connected
eisdir - illegal operation on a directory
eisnam - is a named file
el2hlt - level 2 halted
el2nsync - level 2 not synchronized
el3hlt - level 3 halted
el3rst - level 3 reset
elbin - ELBIN
elibacc - cannot access a needed shared library
elibbad - accessing a corrupted shared library
elibexec - cannot exec a shared library directly
elibmax - attempting to link in more shared libraries than system limit
elibscn - .lib section in a.out corrupted
elnrng - link number out of range
eloop - too many levels of symbolic links
emfile - too many open files
emlink - too many links
emsgsize - message too long
emultihop - multihop attempted
enametoolong - file name too long
enavail - not available
enet - ENET
enetdown - network is down
enetreset - network dropped connection on reset
enetunreach - network is unreachable
enfile - file table overflow
enoano - anode table overflow
enobufs - no buffer space available
enocsi - no CSI structure available
enodata - no data available
enodev - no such device
enoent - no such file or directory
enoexec - exec format error
enolck - no locks available
enolink - link has be severed
enomem - not enough memory
enomsg - no message of desired type
enonet - machine is not on the network
enopkg - package not installed
enoprotoopt - bad proocol option
enospc - no space left on device
enosr - out of stream resources or not a stream device
enosym - unresolved symbol name
enosys - function not implemented
enotblk - block device required
enotconn - socket is not connected
enotdir - not a directory
enotempty - directory not empty
enotnam - not a named file
enotsock - socket operation on non-socket
enotsup - operation not supported
enotty - inappropriate device for ioctl
enotuniq - name not unique on network
enxio - no such device or address
eopnotsupp - operation not supported on socket
eperm - not owner
epfnosupport - protocol family not supported
epipe - broken pipe
eproclim - too many processes
eprocunavail - bad procedure for program
eprogmismatch - program version wrong
eprogunavail - RPC program not available
eproto - protocol error
eprotonosupport - protocol not supported
eprototype - protocol wrong type for socket
erange - math result unrepresentable
erefused - EREFUSED
eremchg - remote address changed
eremdev - remote device
eremote - pathname hit remote file system
eremoteio - remote i/o error
eremoterelease - EREMOTERELEASE
erofs - read-only file system
erpcmismatch - RPC version is wrong
erremote - object is remote
eshutdown - cannot send after socket shutdown
esocktnosupport - socket type not supported
espipe - invalid seek
esrch - no such process
esrmnt - srmount error
estale - stale remote file handle
esuccess - Error 0
etime - timer expired
etimedout - connection timed out
etoomanyrefs - too many references
etxtbsy - text file or pseudo-device busy
euclean - structure needs cleaning
eunatch - protocol driver not attached
eusers - too many users
eversion - version mismatch
ewouldblock - operation would block
exdev - cross-domain link
exfull - message tables full
nxdomain - the hostname or domain name could not be found
Python Functions¶
Overview¶
def is a statement, and function itself is a object, for examples:
if test-some-thing:
def func();
pass
func1 = lambda x:x+10
fun_name = func1
fun_name(args)
Return None by default if there is no return.
LEGB search variable: local def => parent def … => Global => Build in
Parameters¶
def func(arg, named_arg=None, *args, **kargs): pass
func(arg1_value, arg2_value, named_arg1=value1, named_arg2=value2)
# => arg=arg1_value, named_arg=value1, *args=(arg2_value,), kargs={named_arg2=value2}
Arguments¶
def test(a, b):
print a + ' ' + b
test(a="test", b="function")
param = ("Hello", "world")
test(*param)
dparam = {'a':"Very", 'b':"happy"}
test(**dparam)
test(*['hello', 'world']) # directly without via a variable
Similar to print function:
>>> print(*{"a":1, 'b':2, "c":3}) # '*' mean keys only
b c a
>>> print(*[1,2,3])
1 2 3
>>> print(*{1,2,3})
1 2 3
>>> print(*(1,2,3), sep='')
123
lambda expression¶
func = lambda: arg1, arg2 ... argN: expression using arguments 1~N
Decorator Function¶
func = decorator_func(func) cls = decorator_func(cls) func = decorator_cls().__call__(func) cls = decorator_cls().__call__(cls) func = decorator_func(args)(func) # decorator(args) will return sub decorator_func(func)
Function decorator:
import time
import functools
def timeit(func):
@functools.wraps(func)
def wrapper():
start = time.clock()
func()
end =time.clock()
print 'used:', end - start
return wrapper
@timeit
def foo():
print 'in foo()'
Class decorator:
class HTML(object):
"""Baking HTML Tags!"""
def __init__(self, tag="p"):
print("LOG: Baking Tag <{}>!".format(tag))
self.tag = tag
def __call__(self, func):
return lambda: "<{0}>{1}</{0}>".format(self.tag, func(), self.tag)
@HTML("div")
@HTML("p")
def body():
return "Hello"
print(body()) # HTML("div")__call__((HTML("div")__call__(body)))
LOG: Baking Tag <div>!
LOG: Baking Tag <p>!
<div><p>Hello</p></div>
Buildin Functions¶
Python Functional Programming¶
Built-in Functions¶
map/filter/reduce
Note
reduce() is in functools library but not a built-in function in python3.
Example:
>>> import operator, functools
>>> functools.reduce(operator.add, [1,2,3,4], 0)
10
Higher-Order Functions¶
Example:
def compose(*funcs):
"""Return a new function s.t. compose(f,g,...)(x) == f(g(...(x)))"""
def inner(data, funcs=funcs):
result = data
for f in reversed(funcs):
result = f(result)
return result
return inner
# >>> times2 = lambda x: x*2
# >>> minus3 = lambda x: x-3
# >>> mod6 = lambda x: x%6
# >>> f = compose(mod6, times2, minus3)
# >>> all(f(i)==((i-3)*2)%6 for i in range(1000000))
# True
Module: itertools¶
Infinite iterators:
| Iterator | Arguments | Example |
|---|---|---|
| count() | start, [step] | count(10) –> 10 11 12 13 14 … |
| cycle() | p | cycle(‘ABCD’) –> A B C D A B C D … |
| repeat() | elem [,n] | repeat(10, 3) –> 10 10 10 |
Iterators terminating on the shortest input sequence:
| Iterator | Arguments | Example |
|---|---|---|
| accumulate() | p [,func] | accumulate([1,2,3,4,5]) –> 1 3 6 10 15 |
| chain() | p, q, … | chain(‘ABC’, [1,2,3]) –> A B C 1 2 3 |
| compress() | data, selectors | compress(‘ABCDEF’, [1,0,1,0,1,1]) –> A C E F |
| takewhile() | pred, seq | takewhile(lambda x: x<5, [1,4,6,4,1]) –> 1 4 |
| dropwhile() | pred, seq | dropwhile(lambda x: x<5, [1,4,6,4,1]) –> 6 4 1 |
| filterfalse() | pred, seq | filterfalse(lambda x: x%2, range(10)) –> 0 2 4 6 8 |
| groupby() | iterable[,func] | sub-iterators grouped by value of keyfunc(v) |
| islice() | seq, … | islice(‘ABCDEFG’, 2, None) –> C D E F G |
| starmap() | func, seq | starmap(pow, [(2,5), (3,2), (10,3)]) –> 32 9 1000 |
| tee() | it, n | it1, it2, … itn splits one iterator into n |
| zip_longest() | p, q, … | zip_longest(‘ABCD’, ‘xy’, fillvalue=’-‘) –> Ax By C- D- |
Combinatoric generators:
| Iterator | Arguments | Example |
|---|---|---|
| product() | p,…[repeat=1] | product(“ABC”, “123”) =>(A,1),(A,2),(A,3),(B,1),..(C,2),(C,3) |
| permutations() | p[, r] | permutations([1,2,3], 3)=>(1,2,3),(1,3,2),(2,1,3),(2,3,1),… |
| combinations() | p, r | combinations([1,2,3], 2)=>(1,2),(1,3),(2,3) |
Example:
>>> list(itertools.tee([1,2,3,4], 2))
[<itertools.tee object at 0x7feff9d40128>, <itertools.tee object at 0x7feff9d40170>]
>>> list(itertools.tee([1,2,3,4], 2)[0])
[1, 2, 3, 4]
>>> list(itertools.tee([1,2,3,4], 2)[1])
[1, 2, 3, 4]
#----------------------------------------------------------------------
city_list = [('Decatur', 'AL'), ('Huntsville', 'AL'), ('Selma', 'AL'),
('Anchorage', 'AK'), ('Nome', 'AK'),
('Flagstaff', 'AZ'), ('Phoenix', 'AZ'), ('Tucson', 'AZ')]
def get_state(city_state):
return city_state[1]
itertools.groupby(city_list, get_state) =>
('AL', iterator-1),
('AK', iterator-2),
('AZ', iterator-3), ...
where
iterator-1 => ('Decatur', 'AL'), ('Huntsville', 'AL'), ('Selma', 'AL')
iterator-2 => ('Anchorage', 'AK'), ('Nome', 'AK')
iterator-3 => ('Flagstaff', 'AZ'), ('Phoenix', 'AZ'), ('Tucson', 'AZ')
#----------------------------------------------------------------------
>>> print(*[(k, list(g)) for k, g in groupby('1222233')], sep=' ')
('1', ['1']) ('2', ['2', '2', '2', '2']) ('3', ['3', '3'])
>>> list(itertools.combinations_with_replacement([1,2,3], 2))
[(1, 1), (1, 2), (1, 3), (2, 2), (2, 3), (3, 3)]
>>> list(itertools.product("01",repeat=3))
[('0', '0', '0'), ('0', '0', '1'), ('0', '1', '0'), ('0', '1', '1'),
('1', '0', '0'), ('1', '0', '1'), ('1', '1', '0'), ('1', '1', '1')]
Module: functools¶
@functools.lru_cache(maxsize=128, typed=False):
Decorator to wrap a function with a memoizing callable that saves up to the maxsize most recent calls.
@functools.total_ordering:
Decorator to wrap a class, provide at least __eq__() and one of __lt__(), __le__(), __gt__(), or __ge__(). The others will be included automatically based on above eq and at least one of others.
functools.partial(func, *args, **keywords):
functools.partialmethod(func, *args, **keywords):
Return a new partial object which when called will behave like func called with the positional arguments args and keyword arguments keywords.
@functools.singledispatch:
See also multipledispatch module
@functools.wraps(func):
Decorator to wrap the internal function of a decorator.
Example:
>>> from functools import partial
>>> basetwo = partial(int, base=2)
>>> basetwo.__doc__ = 'Convert base 2 string to an int.'
>>> basetwo('10010') # => 18
Module: operator¶
operators in module operator:
Syntax Function
a + b add(a, b)
seq1 + seq2 concat(seq1, seq2)
obj in seq contains(seq, obj)
a / b truediv(a, b)
a // b floordiv(a, b)
a & b and_(a, b)
a ^ b xor(a, b)
~a invert(a)
a | b or(a, b)
a ** b pow(a, b)
a is b is(a, b)
a is not b is_not(a, b)
obj[k] = v setitem(obj, k, v)
del obj[k] delitem(obj, k)
obj[k] getitem(obj, k)
a << b lshift(a, b)
a % b mod(a, b)
a * b mul(a, b)
a @ b matmul(a, b)
-a neg(a)
not a not_(a)
+a pos(a)
a >> b rshift(a, b)
seq[i:j] = values setitem(seq, slice(i, j), values)
del seq[i:j] delitem(seq, slice(i, j))
seq[i:j] getitem(seq, slice(i, j))
s % obj mod(s, obj)
a - b sub(a, b)
obj truth(obj)
a < b lt(a, b)
a <= b le(a, b)
a == b eq(a, b)
a != b ne(a, b)
a >= b ge(a, b)
a > b gt(a, b)
a += b a = iadd(a, b)
a &= b a = iand(a, b)
ListA += listB ListA = iconcat(ListA, listB)
a //= b a = ifloordiv(a, b)
a <<= b a = ilshift(a, b)
a %= b a = imod(a, b)
a *= b a = imul(a, b)
a @= b a = imatmul(a, b)
a |= b a = ior(a, b)
a **= b a = ipow(a, b)
a >>= b a = irshift(a, b)
a -= b a = isub(a, b)
a /= b a = itruediv(a, b)
a ^= b a = ixor(a, b)
Module: multipledispatch¶
Example:
from multipledispatch import dispatch
@dispatch(Rock, Rock)
def beats3(x, y): return None
@dispatch(Rock, Paper)
def beats3(x, y): return y
@dispatch(Rock, Scissors)
def beats3(x, y): return x
@dispatch(Paper, Rock)
def beats3(x, y): return x
@dispatch(Paper, Paper)
def beats3(x, y): return None
@dispatch(Paper, Scissors)
def beats3(x, y): return x
@dispatch(Scissors, Rock)
def beats3(x, y): return y
@dispatch(Scissors, Paper)
def beats3(x, y): return x
@dispatch(Scissors, Scissors)
def beats3(x, y): return None
@dispatch(object, object)
def beats3(x, y):
if not isinstance(x, (Rock, Paper, Scissors)):
raise TypeError("Unknown first thing")
else:
raise TypeError("Unknown second thing")
# >>> beats3(rock, paper)
# <__main__.DuckPaper at 0x103b894a8>
# >>> beats3(rock, 3)
# TypeError: Unknown second thing
Module: predicative_dispatch¶
Example:
from predicative_dispatch import predicate
@predicate(lambda x: x < 0, lambda y: True)
def sign(x, y):
print("x is negative; y is", y)
@predicate(lambda x: x == 0, lambda y: True)
def sign(x, y):
print("x is zero; y is", y)
@predicate(lambda x: x > 0, lambda y: True)
def sign(x, y):
print("x is positive; y is", y)
Python Class¶
Overview¶
In python, class can be used to create a instance, But still, it’s an object and therefore:
- you can assign it to a variable
- you can copy it
- you can add attributes to it
- you can pass it as a function parameter
Examples:
>>> class myClass(object):
... pass
...
>>> type(myClass)
<type 'type'>
>>> type(myClass())
<class '__main__.myClass'>
>>> print myClass
<class '__main__.myClass'>
>>> print myClass()
<__main__.myClass object at 0x7f5cdbfca210>
>>> new_class = myClass
>>> print new_class
<class '__main__.myClass'>
Create Class via type()¶
Besides define the class by “class” keyword, you can also use type() function:
var = type(classname, tuple of parent classes, dictionary containing attributes)
For example:
new_class = type('myClass', (object,), {'test': True})
>>> type(new_class)
<type 'type'>
>>> type(new_class())
<class '__main__.myClass'>
>>> print new_class.test
True
>>> type(myClass)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
NameError: name 'myClass' is not defined
Note
This is also what python do in the background when you define a class by “class” keyword.
What’s metaclasses¶
http://stackoverflow.com/questions/100003/what-is-a-metaclass-in-python
We learned that Python classes are objects. Well, metaclasses are what create these objects. They are the classes’ classes, you can picture them this way:
myClass = metaClass()
myObject = myClass()
As you know, ‘int’ is a normal class to create integers, ‘str’ is a normal class to create strings. The above “type” is also a class, which is used to create other classes, so, we call it metaclass.
Example for how to define a metaclass (normally, it overwrite __new__ to do some special things):
class UpperAttrMetaclass(type):
def __new__(cls, clsname, bases, dct):
uppercase_attr = {}
for name, val in dct.items():
if not name.startswith('__'):
uppercase_attr[name.upper()] = val
else:
uppercase_attr[name] = val
return super(UpperAttrMetaclass, cls).__new__(cls, clsname, bases, uppercase_attr)
class myClass(object):
__metaclass__ = UpperAttrMetaclass # Use this metaclass instead of default 'type'
class_attr = "Yes"
def __init__(self):
self.inst_attr = "right"
super(myClass, self).__init__()
if __name__ == '__main__':
c = myClass()
print(c.CLASS_ATTR) # The class attribute name is changed when call __new__
print(c.inst_attr) # instance attribute is not changed because __init__ is called after __new__.
How to search the metaclass:
- If there is __metaclass__ attribute in current class, use it
- Find the first __metaclass__ attribute in parent classes, use it.
- If there is the definiation of __metaclass__ variable in current or imported module(file), use it.
- Otherwise, use type by default.
Please note that __metaclass__ can also be a function, for example:
def upper_attr(future_class_name, future_class_parents, future_class_attr):
attrs = ((name, value) for name, value in future_class_attr.items() if not name.startswith('__'))
__metaclass__ = upper_attr # For all classes in this module which has no their own metaclass.
Most of the time, you don’t need metaclass. Instead of metaclass, you can also use “monkey patching” or “class decorators”
Monkey Patching¶
Monkey patching is used to:
- Replace methods/attributes/functions at runtime, e.g. to stub out a function during testing;
- Modify/extend behaviour of a third-party product without maintaining a private copy of the source code;
- Apply a patch at runtime to the objects in memory, instead of the source code on disk;
- Distribute security or behavioural fixes that live alongside the original source code.
Carelessly written or poorly documented monkey patches can lead to problems:
- They can lead to upgrade problems when the patch makes assumptions about the patched object that are no longer true; if the product you have changed changes with a new release it may very well break your patch. For this reason monkey patches are often made conditional, and only applied if appropriate.[4]
- If two modules attempt to monkey patch the same method, one of them (whichever one runs last) “wins” and the other patch has no effect, unless monkey patches are written with a pattern like alias_method_chain.
- They create a discrepancy between the original source code on disk and the observed behaviour that can be very confusing to anyone unaware of the patches’ existence.
The following Python example monkey patches the value of Pi from the standard math library:
>>> import math
>>> math.pi
3.141592653589793
>>> math.pi = 3
>>> math.pi
3
Class decorators¶
For example:
class classDecorator(object):
def __init__(self, func):
super(classDecorator, self).__init__()
self.func = func
def __call__(self, msg):
print("class decorator ...")
return self.func(msg)
# echo = classDecorator(echo)
@classDecorator
def echo(msg):
print(msg)
print(echo) # echo is a instance of classDecorator now.
echo('hello') # equal to classDecorator(echo).__call__('hello')
There is three built-in decorators for class
- @staticmethod
- @classmethod
- @property
Example for @property:
class Student(object):
def __init__(self):
super(Student, self).__init__()
self._score = 0
@property # this define a getter, and also define @score.setter
def score(self):
return self._score
@score.setter # Optional, proerty will be readonly without this setter.
def score(self, value):
if not isinstance(value, int):
raise ValueError('score must be an integer!')
if value < 0 or value > 100:
raise ValueError('score must between 0 ~ 100!')
self._score = value
s = Student()
print s.score # ==> 0
s.score = 98
print s.score # ==> 98
Please see also @abstractmethod in abc(Abstract Base Classes) module.
Python Modules¶
The Paths for Import¶
Specified by sys.path, the default path in order:
- Current directory
- PYTHONPATH (path1:path2…)
- Standard installation directories
- Specified by xxx.pth file (one line for each path)
Of course, we can append path in to sys.path like this:
sys.path.append('Path-for-mymoudule')
import mymodule
How to Import Modules¶
import xxx:import xxx, yyy:import <package-name>.<module-name>:Compile and execute all the code in the module, and import itself like a object into current name space.
import xxx as yyy:Same as import, but also rename the module object in current name space.
from xxx import yyy:Compile and execute all the code in the module, and only import yyy into current name space.
from xxx import *:Compile and execute all the code in the module, and import each in it into current name space.
from xxx import yyy as zzz:Same as “from xxx import yyy”, but also rename it to “zzz”.
from .xxx improt yyy:from ..xxx improt yyy:from . improt yyy:from .. import yyy:Require the current module and the “xxx” module are in the same package.
Note
variables/functions start with _X will not be imported by from xxx import *
When call the properties from outside of class, need to add class name like this ‘_className__x’.
Note
import requires using module name later, but from import the name directly, must be careful.
Note
import、from can also import pyc, package directory, c module, zip archive.
__init__.py¶
All code in __init__.py will be execute when first import the package.
List __all__ in __init__.py defines the module which will be imported when from xxx import *.
Otherwise, you must use ‘import’ to import the sub modules one by one manually.
Local Import¶
As the same as def, import and from also are statements, and can be used any where, such as in if/else, or inside of a function:
def square_root(a):
# This import is into the square_root functions local scope
import math
return math.sqrt(a)
Optional Import¶
try:
# Python 2.x
from urlparse import urljoin
from urllib2 import urlopen
except ImportError:
# Python 3.x
from urllib.parse import urljoin
from urllib.request import urlopen
Python Design Patterns¶
Creational Patterns:
| Pattern | Description |
|---|---|
| abstract_factory | use a generic function with specific factories |
| borg | a singleton with shared-state among instances |
| builder | instead of using multiple constructors, builder object receives parameters and returns constructed objects |
| factory_method | delegate a specialized function/method to create instances |
| lazy_evaluation | lazily-evaluated property pattern in Python |
| pool | preinstantiate and maintain a group of instances of the same type |
| prototype | use a factory and clones of a prototype for new instances (if instantiation is expensive) |
Structural Patterns:
| Pattern | Description |
|---|---|
| 3-tier | data<->business logic<->presentation separation (strict relationships) |
| adapter | adapt one interface to another using a white-list |
| bridge | a client-provider middleman to soften interface changes |
| composite | encapsulate and provide access to a number of different objects |
| decorator | wrap functionality with other functionality in order to affect outputs |
| facade | use one class as an API to a number of others |
| flyweight | transparently reuse existing instances of objects with similar/identical state |
| front_controller | single handler requests coming to the application |
| mvc | model<->view<->controller (non-strict relationships) |
| proxy | an object funnels operations to something else |
Behavioral Patterns:
| Pattern | Description |
|---|---|
| chain | apply a chain of successive handlers to try and process the data |
| catalog | general methods will call different specialized methods based on construction parameter |
| chaining_method | continue callback next object method |
| command | bundle a command and arguments to call later |
| mediator | an object that knows how to connect other objects and act as a proxy |
| memento | generate an opaque token that can be used to go back to a previous state |
| observer | provide a callback for notification of events/changes to data |
| publish_subscribe | a source syndicates events/data to 0+ registered listeners |
| registry | keep track of all subclasses of a given class |
| specification | business rules can be recombined by chaining the business rules together using boolean logic |
| state | logic is organized into a discrete number of potential states and the next state that can be transitioned to |
| strategy | selectable operations over the same data |
| template | an object imposes a structure but takes pluggable components |
| visitor | invoke a callback for all items of a collection |
Singleton¶
Examples:
# Method 1: only __new__() once
class Singleton(object):
def __new__(cls,*args,**kwargs):
if not hasattr(cls,'_inst'):
cls._inst=super(Singleton,cls).__new__(cls,*args,**kwargs)
return cls._inst
# Method 2: share the __dict__
class Borg(object):
_shared_state={}
def __new__(cls,*args,**kwargs):
obj=super(Borg,cls).__new__(cls,*args,**kwargs)
obj.__dict__=cls._shared_state
return obj
Note
The pythonic way is to define a module if you want to use singleton class.
Python Project¶
Typical Structure¶
foobar
|__docs
| |__conf.py
| |__quickstart.rst
| |__index.rst
|__foobar
| |__ __init__.py
| |__xxx.py
| |__tests
| |__ __init__.py
| |__ test_xxx.py
|__tools
|__data
|__setup.py
|__ReadMe.rst
|__requirements.txt # the dependencies
virtualenv¶
virtualenv + virtualenvwrapper:
sudo pip install virtualenv
sudo pip install virtualenvwrapper
[MAC OS] pyenv + pyenv-virtualenv:
brew install pyenv
brew install pyenvbrew install pyenv
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
if which pyenv > /dev/null; then eval "$(pyenv init -)"; fi
pyenv install -l
pyenv install 3.5.2
pyenv rehash
在任意目录， 如果想在这个目录及其子目录中使用不同的python版本
pyenv local 3.5.2
python --version #should be 3.5.2
pyenv virtualenv 3.5.2 venv3
pyenv shell venv3
pyenv activate venv3
pyenv virtualenv-delete venv3 # delete the virtualenv
pyenv <tab> # to show all sub commands
不建议使用 pyenv global， 可能对全局其他程序有影响。
pyenv: 安装切换不同的python版本，但是同一个版本还是共用一个python环境
virtualenv: 创建不同的python环境 （--python指定基于哪一个python版本，要求系统中先装好了对应版本）
virtualenvwrapper: 更加方便地创建和管理不同的virtualenv。
pyenv-virtualenv: 把virtualenv集成到pyenv中， 同时自带virtualenvwrapper 和autoenv 类似高阶管理功能。
如果系统已经自带多个python版本， 比如ubuntu16.04 自带python2 和python3,
或许virtualenv + virtualenvwrapper 足够。
如果还需要自己安装管理多个python版本，python环境, 比如 MacbootPro 2015上面，
那么 pyenv+pyenv-virtualenv 可能更适合。
Coding Style¶
- https://www.python.org/dev/peps/pep-0008/
- http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html
PEP8 online: http://www.pep8online.com/
pylint:
https://pylint.readthedocs.io/en/latest/
sudo apt-get install pylint
There are 5 kind of message types : * (C) convention, for programming standard violation * (R) refactor, for bad code smell * (W) warning, for python specific problems * (E) error, for much probably bugs in the code * (F) fatal, if an error occurred which prevented pylint from doing
Examples:
pylint --generate-rcfile >pylint.conf
pylint --rcfile=pylint.conf xxx.py
pylint --reports=n xxx.py
pep8(pycodestyle):
https://pypi.python.org/pypi/pep8
sudo pip install pep8, and run it like this: pep8 <source-file>.
Note
There is a pyflakes vim plugin: https://github.com/vim-scripts/pyflakes
flake8:
https://pypi.python.org/pypi/flake8
豆瓣内部要求使用这个, Include Pyflakes + Pycodestyle + McCabe … + other plugins。
Code complexity:
flake8 --max-complexity 10 app.py (#It's better not greater than 10)
PEP8 Auto-convert tool:
- autopep8: https://github.com/hhatto/autopep8
- Google python: https://github.com/google/yapf
Configuration¶
- Use a config.py file or the config module
- Module “configparser” for ini configuration file
- Module “json” for json configuration file
- Module “csv” for csv configuration file
- Module “xml.etree.ElementTree ” for xml configuration file
Examples for json:
with open('comments.json', 'r') as f: # see also json.load()
comments = json.loads(f.read())
with open('comments.json', 'w') as f: # see also json.dump()
f.write(json.dumps(comments, indent=4, separators=(',', ': ')))
Unittest¶
Module “unittest” for unit tests.
See also nose: - http://nose.readthedocs.org/en/latest/ - http://pythontesting.net/framework/nose/nose-introduction/
Documentation¶
Tips:
- Use some formal template of document
- Simple sentences
- One topic for each paragraph
- More use of present tenses
- Use real codes as the examples.
- Good code is always better than many documents.
HowTO:
- Document project: sphinx + reStructuredText
- doctest: https://docs.python.org/2/library/doctest.html
- pydoc: https://docs.python.org/2/library/pydoc.html
- epydoc tool: http://epydoc.sourceforge.net/
Epydoc is a tool for generating API documentation for Python modules, based on their docstrings. For an example of epydoc’s output, see the API documentation for epydoc itself.
A lightweight markup language called epytext can be used to format docstrings, and to add information about specific fields, such as parameters and instance variables. Epydoc also understands docstrings
License¶
The lice tool: https://github.com/licenses/lice
lice [opts ...] ('afl3', 'agpl3', 'apache', 'bsd2', 'bsd3', 'cc0',
'cc_by', 'cc_by_nc', 'cc_by_nc_nd', 'cc_by_nc_sa', 'cc_by_nd',
'cc_by_sa', 'cddl', 'epl', 'gpl2', 'gpl3', 'isc', 'lgpl', 'mit',
'mpl', 'wtfpl', 'zlib')
Package¶
Module ‘setuptools’
Continuous Integration¶
buildbot¶
Quick start:
sudo pip install buildbot
sudo pip install buildbot-www
sudo pip install buildbot-waterfall_view
sudo pip install buildbot-console_view
sudo pip install buildbot-worker
buildbot create-master /tmp/experimental_buildmaster
cp /tmp/experimental_buildmaster/master.cfg.sample /tmp/experimental_buildmaster/master.cfg
buildbot upgrade-master /tmp/experimental_buildmaster # each time when change the conf file
# Check the worker name and password from master.cfg
buildbot-worker create-worker /tmp/experimental_buildslave localhost:9989 example-worker pass
buildbot-worker start /tmp/experimental_buildslave
firefox http://localhost:8010/ &
Builds => runtests => force => Input "Your name"(email) and click "Start Build"
Jenkins¶
Project Template¶
https://github.com/audreyr/cookiecutter
Create new project base on a template:
pip install cookiecutter
cookiecutter https://github.com/sloria/cookiecutter-flask.git
All templates here: https://github.com/audreyr/cookiecutter#a-pantry-full-of-cookiecutters
Other code check tool¶
https://wiki.python.org/moin/PythonTestingToolsTaxonomy
pychecker:
http://pychecker.sourceforge.net/
Install: sudo apt-get install pychecker
clonedigger:
Clone Digger aimed to detect similar code in Python and Java programs. The synonyms for the term “similar code” are “clone” and “duplicate code”.
http://clonedigger.sourceforge.net/
Install: sudo pip install clonedigger
Examples:
clonedigger source_file_1 source_file_2 ...
clonedigger path_to_source_tree
coverage:
Install: sudo pip install coverage
Examples:
/usr/local/bin/coverage run test.py # the output is in file "./.coverage" by default
/usr/local/bin/coverage report -m # line range show the code which is not covered.
/usr/local/bin/coverage html # the outputs are in folder "./htmlcov" by default
firefox htmlcov/index.html
profile:
https://docs.python.org/3/library/profile.html
Examples:
python -m cProfile test.py
pymetrics: The complexity test tool
http://www.traceback.org/2008/03/31/measuring-cyclomatic-complexity-of-python-code/
/tmp/pymetrics-0.8.1/pymetrics ./send-pull-request
Command-line interfaces¶
argparse:
https://docs.python.org/3/library/argparse.html#module-argparse
docopt:
Python PEP8 in Chinese¶
代码编排¶
- 缩进。4个空格的缩进（编辑器都可以完成此功能），不使用Tap，更不能混合使用Tap和空格。
- 每行最大长度79，换行可以使用反斜杠，最好使用圆括号。换行点要在操作符的后边敲回车。
- 类和top-level函数定义之间空两行；类中的方法定义之间空一行 函数内逻辑无关段落之间空一行；其他地方尽量不要再空行。
文档编排¶
- 模块内容的顺序：模块说明和docstring—import—globals&constants—其他定义。 其中import部分，又按标准、三方和自己编写顺序依次排放，之间空一行。
- 不要在一句import中多个库，比如import os, sys不推荐。
- 如果采用from XX import XX引用库，可以省略‘module.’，都是可能出现命名冲突，这时就要采用import XX。
空格的使用¶
总体原则，避免不必要的空格。
- 各种右括号前不要加空格。
- 逗号、冒号、分号前不要加空格。
- 函数的左括号前不要加空格。如Func(1)。
- 序列的左括号前不要加空格。如list[2]。
- 操作符左右各加一个空格，不要为了对齐增加空格。
- 函数默认参数使用的赋值符左右省略空格。
- 不要将多句语句写在同一行，尽管使用‘；’允许。
- if/for/while语句中，即使执行语句只有一句，也必须另起一行。
注释¶
总体原则：
- 避免无谓的注释。
- 错误的注释不如没有注释。所以当一段代码发生变化时，第一件事就是要修改注释！
注释必须使用英文，最好是完整的句子，首字母大写，句后要有结束符。 如果是短语，可以省略结束符。
块注释，在一段代码前增加的注释。在‘#’后加一空格。段落之间以只有‘#’的行间隔。比如:
# Description : Module config. # # Input : None # # Output : None
行注释，在一句代码后加注释，这种方式尽量少使用。比如：
x = x + 1 # Increment x
文档描述¶
为所有的共有模块、函数、类、方法写docstrings；非共有的没有必要，但是可以写注释（在def的下一行）。
如果docstring要换行，参考如下例子,详见PEP 257:
"""Return a foobang Optional plotz says to frobnicate the bizbaz first. """
命名规范¶
总体原则，新编代码必须按下面命名风格进行，现有库的编码尽量保持风格。
- 尽量单独使用小写字母‘l’，大写字母‘O’等容易混淆的字母。
- 模块命名尽量短小，使用全部小写的方式，可以使用下划线。
- 包命名尽量短小，使用全部小写的方式，不可以使用下划线。
- 类的命名使用CapWords的方式，模块内部使用的类采用_CapWords的方式。
- 异常命名使用CapWords+Error后缀的方式。
- 全局变量尽量只在模块内有效，类似C语言中的static。实现方法有两种，一是__all__机制;二是前缀一个下划线。
- 函数命名使用全部小写的方式，可以使用下划线。
- 常量命名使用全部大写的方式，可以使用下划线。
- 类的属性（方法和变量）命名使用全部小写的方式，可以使用下划线。
- 类的私有属性(方法和变量）以一条下划线作为前缀。
- 类的属性若与关键字名字冲突，后缀一下划线，尽量不要使用缩略等其他方式。
- 为避免与子类属性命名冲突，在类的一些属性前，前缀两条下划线。 比如：类Foo中声明__a,访问时，只能通过Foo._Foo__a，避免歧义。如果子类也叫Foo，那就无能为力了。
- 类的方法第一个参数必须是self，而静态方法第一个参数必须是cls。
编码建议¶
编码中考虑到其他python实现的效率等问题，比如运算符‘+’在CPython（Python）中效率很高 都是Jython中却非常低，所以应该采用.join()的方式。
尽可能使用‘is’‘is not’取代‘==’，比如if x is not None 要优于if x。
使用基于类的异常，每个模块或包都有自己的异常类，此异常类继承自Exception。
异常中不要使用裸露的except，except后跟具体的exceptions。
异常中try的代码尽可能少。比如:
try: value = collection[key] except KeyError: return key_not_found(key) else: return handle_value(value) 优于 try: return handle_value(collection[key]) except KeyError: # Will also catch KeyError raised by handle_value() return key_not_found(key)
使用startswith() and endswith()代替切片进行序列前缀或后缀的检查。比如:
if foo.startswith('bar'): 优于 if foo[:3] == 'bar':
使用isinstance()比较对象的类型。比如:
if isinstance(obj, int): 优于 if type(obj) is type(1):
判断序列空或不空，有如下规则:
if not seq: if seq: 优于 if len(seq) if not len(seq)
字符串不要以空格收尾。
二进制数据判断使用 if boolvalue的方式。
超长行字符串定义:
"this is line one, and then" "this is line two to avoid long line"
Python Tox¶
unittest Quickstart¶
pytest is better than the unittest standard library.
https://docs.python.org/2.7/library/unittest.html
All test case methods must begin with test, and run in the order of name.
Example(test.py):
#!/usr/bin/python
# -*- coding:utf-8 -*-
import unittest
class TestStringMethods(unittest.TestCase): #must based on unitest.TestCase
def test_upper(self):
self.assertEqual('foo'.upper(), 'FOO')
def test_isupper(self):
self.assertTrue('FOO'.isupper())
self.assertFalse('Foo'.isupper())
def test_split(self):
s = 'hello world'
self.assertEqual(s.split(), ['hello', 'world'])
# check that s.split fails when the separator is not a string
with self.assertRaises(TypeError):
s.split(2)
if __name__ == '__main__':
#unittest.main() # run all test cases by default
#suite = unittest.TestLoader().loadTestsFromTestCase(TestStringMethods)
suite = unittest.TestSuite() # create test suite to test part of the cases
suite.addTest(TestStringMethods('test_upper'))
suite.addTest(TestStringMethods('test_isupper'))
unittest.TextTestRunner(verbosity=2).run(suite)
Run the test case:
./test.py
python -m unittest -v test # Run special test module
python -m unittest -v test.TestStringMethods # Run special test class in the module
python -m unittest -v test.TestStringMethods.test_split #run special test case
python -m unittest discover ./ "test*.py" # Discover all test cases.
Important concepts¶
- test fixture:
- A test fixture represents the preparation needed to perform one or more tests, and any associate cleanup actions. This may involve, for example, creating temporary or proxy databases, directories, or starting a server process.
- test case:
- A test case is the smallest unit of testing. It checks for a specific response to a particular set of inputs. unittest provides a base class, TestCase, which may be used to create new test cases.
- test suite:
- A test suite is a collection of test cases, test suites, or both. It is used to aggregate tests that should be executed together.
- test runner:
- A test runner is a component which orchestrates the execution of tests and provides the outcome to the user. The runner may use a graphical interface, a textual interface, or return a special value to indicate the results of executing the tests.
Fixtures¶
setUp/tearDown() for test case level setUpClass()/tearDownClass for class level, need the @classmethod decorator. setUpModule()/tearDownModule() for module level
Skip/ExpectFaiure¶
The following decorators implement test skipping and expected failures:
@unittest.skip(reason): Unconditionally skip the decorated test. reason should describe why the test is being skipped.
@unittest.skipIf(condition, reason): Skip the decorated test if condition is true.
@unittest.skipUnless(condition, reason): Skip the decorated test unless condition is true.
@unittest.expectedFailure(): Mark the test as an expected failure. If the test fails when run, the test is not counted as a failure.
Skipped tests will not have setUp() or tearDown() run around them. Skipped classes will not have setUpClass() or tearDownClass() run.
Asserts¶
| Method | Checks that |
|---|---|
| assertEqual(a, b) | a == b |
| assertNotEqual(a, b) | a != b |
| assertTrue(x) | bool(x) is True |
| assertFalse(x) | bool(x) is False |
| assertIs(a, b) | a is b |
| assertIsNot(a, b) | a is not b |
| assertIsNone(x) | x is None |
| assertIsNotNone(x) | x is not None |
| assertIn(a, b) | a in b |
| assertNotIn(a, b) | a not in b |
| assertIsInstance(a, b) | isinstance(a, b) |
| assertNotIsInstance(a, b) | not isinstance(a, b) |
| assertAlmostEqual(a, b) | round(a-b, 7) == 0 |
| assertNotAlmostEqual(a, b) | round(a-b, 7) != 0 |
| assertGreater(a, b) | a > b |
| assertGreaterEqual(a, b) | a >= b |
| assertLess(a, b) | a < b |
| assertLessEqual(a, b) | a <= b |
| assertRegexpMatches(s, r) | r.search(s) |
| assertNotRegexpMatches(s, r) | not r.search(s) |
| assertItemsEqual(a, b) | sorted(a) == sorted(b) |
| assertDictContainsSubset(a, b) | all the key/value pairs in a exist in b |
Python pytest¶
Python Shell¶
sh¶
sh (previously pbs) is a full-fledged subprocess interface for Python that allows you to call any program as if it were a function:
from sh import ifconfig
print ifconfig("eth0")
They may be executed on the sh namespace, or imported directly from sh:
import sh
print(sh.ifconfig("eth0"))
For commands that have dashes in their names, for example /usr/bin/google-chrome, substitute the dash for an underscore:
import sh
sh.google_chrome("http://google.com")
For commands that take multiple arguments, need to be invoked using separate string for each argument rather than single string for all:
from sh import ls
ls("-l", "./") # ls("-l ./") will not work
For commands with more exotic characters in their names, like ., or if you just don’t like the “magic”-ness of dynamic lookups, you may use sh’s Command wrapper and pass in the command name or the absolute path of the executable:
import sh
run = sh.Command("/home/amoffat/run.sh") # Absolute path
run()
lscmd = sh.Command("ls") # Absolute path not needed
lscmd()
lscmd('./')
Note
Similarly, sh.Command(“ls ./”) will not work! Must wrap command only and pass parameters later one by one.
Commands support short-form -a and long-form –arg arguments as keyword arguments:
# resolves to "curl http://duckduckgo.com/ -o page.html --silent"
curl("http://duckduckgo.com/", o="page.html", silent=True)
# or if you prefer not to use keyword arguments, this does the same thing:
curl("http://duckduckgo.com/", "-o", "page.html", "--silent")
# resolves to "adduser amoffat --system --shell=/bin/bash --no-create-home"
adduser("amoffat", system=True, shell="/bin/bash", no_create_home=True)
# or
adduser("amoffat", "--system", "--shell", "/bin/bash", "--no-create-home")
Shell style piping is performed using function composition. Just pass one command as the input to another, and sh will create a pipe between the two:
# sort this directory by biggest file
print(sort(du(glob("*"), "-sb"), "-rn"))
# print(the number of folders and files in /etc
print(wc(ls("/etc", "-1"), "-l"))
You can redirect the stdin, stdout and stderr streams of a process to a file or file-like object by the special _in, _out and _err keyword argument:
ls(_out="files.list")
ls("nonexistent", _err="error.txt")
print(tr("[:lower:]", "[:upper:]", _in="sh is awesome")) # SH IS AWESOME
The exit code can be seen through a command’s exit_code property:
output = ls("/")
print(output.exit_code) # should be 0
For other more advance features, please refer to: http://amoffat.github.io/sh/
os.system()¶
Return 0 for success and others for failure.
test.py:
import os
retcode = os.system("cp ./test.py ./test1.py")
print("retcode is: %d" % retcode);
retcode = os.system("cp ./not_exist_file.py ./test1.py")
print("retcode is: %d" % retcode);
$ python ./test.py:
retcode is: 0
cp: cannot stat ‘./not_exist_file.py’: No such file or directory
retcode is: 256
os.popen()¶
# get the result only
fouput = os.popen("ls /home/test")
print fouput.readlines()
# or use the input only
finput = os.popen("python TestInput.py", "w")
finput.write("how are you\n")
subprocess¶
It’s better to use subprocess instead of os.popen() and os.system()
Python Expect¶
Python SSH¶
pxssh¶
https://pexpect.readthedocs.org/en/stable/api/pxssh.html
Example:
import pxssh
import getpass
try:
s = pxssh.pxssh()
hostname = raw_input('hostname: ')
username = raw_input('username: ')
password = getpass.getpass('password: ')
s.login (hostname, username, password, original_prompt='[$#>]')
s.sendline ('uptime')
s.prompt()
print s.before
s.sendline ('ls -l')
s.prompt()
print s.before
s.logout()
except pxssh.ExceptionPxssh, e:
print "pxssh failed on login."
print str(e)
parmmiko¶
Installation:
sudo pip install pycrypto
sudo pip install paramiko
Example:
import paramiko
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
ssh.connect(host, 22, user, passwd, timeout=7200)
stdin, stdout, stderr = ssh.exec_command("uptime")
print(stdout.readlines())
stdin, stdout, stderr = ssh.exec_command("uname -a")
print(stdout.readlines())
ssh.close()
fabric¶
Python Web Scraping¶
beautifulsoup¶
lxml¶
The lxml XML toolkit is a Pythonic binding for the C libraries libxml2 and libxslt. The speed is more fast than beautifulsoup. Maybe some problem in multi-thread, so it’s better to use the multi-process.
sudo apt-get install python-lxml sudo pip install cssselect
Example:
from lxml.etree import HTML
import requests
url = 'https://segmentfault.com/'
res = requests.get(url)
page = HTML(res.text)
for title in page.cssselect('.title>a'):
#print(title.text.encode('UTF-8')) # For python2.7
print title.text # For python3.x
urllib¶
Selenium¶
pip3 install selenium
Example:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
from time import sleep
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("http://www.baidu.com")
driver.find_element_by_id("kw").send_keys("Selenium2")
driver.find_element_by_id("su").click() # search "Selenium2"
sleep(1)
driver.get_screenshot_as_file('./search_result.png')
driver.implicitly_wait(10)
driver.find_element_by_xpath(".//*[@id='1']/h3/a").click() # goto the first result
sleep(3)
driver.quit()
Note
Scrapy¶
Python Email¶
Example:
import os
import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from .config import Config
from .utils import Logger
from .git import Git
class RREmail(object):
def __init__(self, user=None, from_email=None, to_email=None, cc_email=None):
super(RREmail, self).__init__()
self.from_email = from_email or Git.get_user_email()
self.to_email = to_email or Config.GateKeeper
self.cc_email = cc_email or Config.CHECK_IN_CC
self.user = user or Config.EMAIL_USER
self.server = Config.EMAIL_SMTP_SERVER
self.port = Config.EMAIL_PORT
self.sm = smtplib.SMTP(self.server, self.port, timeout=20)
# Not neccessary
def login(self, user='', password=''):
#self.sm.set_debuglevel(1)
#self.sm.ehlo()
#self.sm.starttls()
#self.sm.ehlo()
self.sm.login(user or self.user,
password or raw_input('Please input the email password: ').strip())
def send(self, subject, body, body_html=None, attachments=None, in_reply_to=None):
message = MIMEMultipart('related')
message['Subject'] = subject
message['From'] = self.from_email
message['To'] = self.to_email
if self.cc_email.strip():
message['CC'] = self.cc_email
if in_reply_to is not None:
message['In-Reply-To'] = in_reply_to
msg_body = MIMEMultipart('alternative')
message.attach(msg_body)
body_plain = MIMEText(body, 'plain')
msg_body.attach(body_plain)
if body_html is not None:
body_html = MIMEText(body_html, 'html', 'utf-8')
msg_body.attach(body_html)
if attachments is not None:
for patch in attachments:
f = open(patch)
att = MIMEText(f.read(), 'plain', 'utf-8')
att["Content-Type"] = 'text/plain'
att["Content-Disposition"] = 'attachment;filename="%s"' % os.path.basename(patch)
message.attach(att)
f.close()
return self.sm.sendmail(self.from_email,
self.to_email,
message.as_string())
def quit(self):
self.sm.quit()
References:
Python Excel¶
Overview¶
openpyxl The recommended package for reading and writing Excel 2010 files (ie: .xlsx)
https://openpyxl.readthedocs.io/en/default/
xlwd This package is for reading data and formatting information from older Excel files (ie: .xls)
http://xlrd.readthedocs.io/en/latest/
xlwt This package is for writing data and formatting information to older Excel files (ie: .xls)
openpyxl¶
Install: sudo pip install openpyxl
Workboot in Memory¶
Create new workbook:
from openpyxl import Workbook
wb = Workbook()
# Write-only mode
wb = Workbook(write_only=True)
Get the worksheet:
ws = wb.active # Get the active sheet (The first by default)
ws = wb['Mysheet'] # Get the sheet by name
Change the worksheet name:
ws.title = "New Title" #
Get all the sheet names:
print(wb.sheetnames) # or
for sheet in wb:
print(sheet.title)
Get the column letter:
from openpyxl.utils import get_column_letter
get_column_letter(1) # 'A' for column 1
Create new worksheet:
ws1 = wb.create_sheet("Mysheet") # insert at the end (default)
ws2 = wb.create_sheet("Mysheet", 0) # insert at first position
Copy sheet within a single workbook:
source = wb.active
target = wb.copy_worksheet(source)
The background color of the tab holding this title is white by default. You can change this via the sheet_properties.tabColor property:
ws.sheet_properties.tabColor = "1072BA" # RRGGBB
Access one cell:
c = ws['A1']
ws['A1'] = 'Name'
d = ws.cell(row=4, column=2) # return cell, d.value contains the value
d = ws.cell(row=4, column=2, value=10) # change/assign to new value
cell_range = ws['A1':'C2'] # cells from A1 to C2
colC = ws['C'] # The whole column
ow10 = ws[10] # The whole row
col_range = ws['C:D'] # Two columns
row_range = ws[5:10] # Two rows
# Cells in each rows
>>> for row in ws.iter_rows(min_row=1, max_col=3, max_row=2):
... for cell in row:
... print(cell)
<Cell Sheet1.A1>
<Cell Sheet1.B1>
<Cell Sheet1.C1>
<Cell Sheet1.A2>
<Cell Sheet1.B2>
<Cell Sheet1.C2>
# Cells in each columns
>>> for col in ws.iter_cols(min_row=1, max_col=3, max_row=2):
... for cell in col:
... print(cell)
<Cell Sheet1.A1>
<Cell Sheet1.A2>
<Cell Sheet1.B1>
<Cell Sheet1.B2>
<Cell Sheet1.C1>
<Cell Sheet1.C2>
# all cells in row
>>> tuple(ws.rows)
((<Cell Sheet.A1>, <Cell Sheet.B1>, <Cell Sheet.C1>),
(<Cell Sheet.A2>, <Cell Sheet.B2>, <Cell Sheet.C2>),
...)
>>> tuple(ws.columns)
((<Cell Sheet.A1>, <Cell Sheet.A2>, ...),
(<Cell Sheet.B1>, <Cell Sheet.B2>, ...),
(<Cell Sheet.C1>, <Cell Sheet.C2>, ...))
ws['A1'].row # 1
ws['A1'].column # 'A'
ws['A1'].coordinate # 'A1'
Write-only cell:
from openpyxl.writer.write_only import WriteOnlyCell
from openpyxl.styles import Font
cell = WriteOnlyCell(ws, value="hello world")
cell.comment = Comment(text="A comment", author="Author's Name")
cell.font = Font(name='Courier', size=36)
ws.append([cell, 3.14, None])
Working with style:
See also: https://openpyxl.readthedocs.io/en/default/styles.html
Builtin styles: https://openpyxl.readthedocs.io/en/default/styles.html#using-builtin-styles
Merge/Unmerge cells:
ws.merge_cells('A1:B1')
ws.unmerge_cells('A1:B1')
ws.merge_cells(start_row=2,start_column=1,end_row=2,end_column=4)
ws.unmerge_cells(start_row=2,start_column=1,end_row=2,end_column=4)
Set the width of column:
ws.column_dimensions['A'].width = 16
Using formulae:
ws['D1'] = '=SUM(A1:C1)'
Using number format:
ws['A2'] = datetime.datetime(2010, 7, 21)
ws['A2'].number_format # 'yyyy-mm-dd h:mm:ss'
ws['A2'].number_format # 'General'
ws['A2'] = 0.12 # 0.12 when open by MS office
ws['A2'].number_format = '0%'
ws['A2'].value # value still 0.12, but shown as 12%
Insert an image:
from openpyxl.drawing.image import Image
img = Image('logo.png')
ws.add_image(img, 'A1')
Adding a comment to a cell:
# One comment can only be assgined to one cell
from openpyxl.comments import Comment
ws["A1"].comment = Comment("Text", "Author")
Create a chart:
>>> for i in range(1, 11):
... ws['A%d'%i] = i
>>> from openpyxl.chart import BarChart, Reference, Series
>>> values = Reference(ws, min_col=1, min_row=1, max_col=1, max_row=10)
>>> chart = BarChart()
>>> chart.add_data(values)
>>> ws.add_chart(chart, "E15")
>>> wb.save("SampleChart.xlsx")
See also: https://openpyxl.readthedocs.io/en/default/charts/introduction.html
Fold columns:
ws.column_dimensions.group('A','D', hidden=True)
Using filters and sorts:
ws.auto_filter.ref = "A1:B15"
ws.auto_filter.add_filter_column(0, ["Kiwi", "Apple", "Mango"])
ws.auto_filter.add_sort_condition("B2:B15")
Save to a file:
wb.save('myfile.xlsx')
Loading from a file¶
- ::
>>> from openpyxl import load_workbook >>> wb2 = load_workbook('test.xlsx') >>> print wb2.get_sheet_names() ['Sheet2', 'New Title', 'Sheet1']
# Read-only mode wb = load_workbook(filename=’large_file.xlsx’, read_only=True)
Python Performance and Debug¶
- https://wiki.python.org/moin/PythonTestingToolsTaxonomy
- http://www.jb51.net/article/63244.htm
- http://blog.jobbole.com/51062/
pdb¶
python -m pdb xxx.py
time¶
time.time() timeit /usr/bin/time –verbose python test.py
trace¶
https://docs.python.org/3/library/trace.html
The trace module allows you to trace program execution, generate annotated statement coverage listings, print caller/callee relationships and list functions executed during a program run. It can be used in another program or from the command line.
python -m trace –count -C . somefile.py
pycallgraph tool¶
memory_profiler¶
pip install psutil memory_profiler
https://pypi.python.org/pypi/psutil#downloads https://pypi.python.org/pypi/memory_profiler/
# line by line python -m memory_profiler test.py
# timing sequence mprof run test.py #output to mprofile_xxxx.dat by default mprof plot mprofile_xxx.dat
line_profiler¶
https://pypi.python.org/pypi/line_profiler/ https://github.com/rkern/line_profiler
sudo pip install line_profiler
@profile
def show_function():
...
kernprof.py -l -v test.py
objgraph¶
https://pypi.python.org/pypi/objgraph/ http://mg.pov.lt/objgraph/
pip install xdot objgraph
guppy¶
http://guppy-pe.sourceforge.net/
Install: sudo pip install guppy
Example:
from guppy import hpy
def random_sort3(n):
hp = hpy()
print "Heap at the beginning of the functionn", hp.heap()
l = [random.random() for i in range(n)]
l.sort()
print "Heap at the end of the functionn", hp.heap()
return l
if __name__ == "__main__":
random_sort3(2000000)
#hp.heap().more.more for more defails
Python Template Libraries¶
jinja2¶
from jinja2 import Environment, PackageLoader
def template_render(name, args):
env = Environment(loader=PackageLoader('myApp', 'templates'))
template = env.get_template(name)
return template.render(args=args)
Python Internationalization¶
gettext¶
Define Messages by _() function:
"""test.py"""
import gettext
try:
gettext.translation('myapp', localedir='locale', languages=['zh_CN']).install()
except:
_ = lambda s: s
# ...
print(_('This is a translatable string.'))
Generate the pot file:
mkdir -p ./locale/zh_CN/LC_MESSAGES/
xgettext -o locale/zh_CN/LC_MESSAGES/myapp.pot ./test.py
Generate the po and mo files:
# edit po file via poedit tool.
sudo apt-get install poedit
# /usr/bin/poedit
# create new translation or edit exist po file.
# add/edit/save
Python Charts¶
Data format and libraries¶
csv: [Standard Library] csv json: [Standard Library] json excel: openpyxl database: sqlite3/pymongodb
Aligned binary data: [Standard Library] struct:
>>> import struct
>>> struct.unpack_from('5s10s', b'abcde 0123456789')
(b'abcde', b' 012345678')
matplotlib¶
Installation:
sudo apt-get install python-numpy python-matplotlib python-scipy
Overview¶
Path for matplotlibrc:
./matplotlibrc
~/.config/matplotlib # get by matplotlib.get_configdir()
xxx/site-packages/matplotlib/mpl-data/matplotlibrc # installation path
#matplotlib.matplotlib_fname() get the matplitlibrc file name in use.
set rc parameters:
import matplotlib as mpl
mpl.rcParams['lines.linewidth'] = 2
mpl.rcParams['lines.color'] = 'r'
# call the rc() function
mpl.rc('lines', linewidth=2, color='r')
mpl.rcdefaults()
plot¶
Examples:
import numpy as np
import matplotlib.pyplot as plt
x = np.linspace(-np.pi, np.pi, 256, endpoint=True)
plt.rcParams['lines.linewidth'] = 2
plt.rcParams['lines.color'] = 'r'
plt.plot(x, np.sin(x))
plt.plot(x, np.cos(x), linewidth=3, color='g')
plt.title('Function $\sin$ and $\cos$')
plt.xlim(-np.pi, np.pi)
plt.xticks([-np.pi, -np.pi/2, 0, np.pi/2, np.pi],
[r'$-\pi$', r'$-\pi$', r'$0$', r'$+\pi/2$', r'$+\pi$',])
#plt.grid()
plt.show()
date index¶
Examples:
import datetime
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.dates as mdates
start = datetime.datetime(2016, 10, 1)
stop = datetime.datetime(2016, 10, 8)
delta = datetime.timedelta(days=1)
dates = mdates.drange(start, stop, delta)
values = np.random.rand(len(dates))
fig, ax = plt.subplots()
ax.plot_date(dates, values, linestyle='-')
date_format = mdates.DateFormatter('%Y-%m-%d')
ax.xaxis.set_major_formatter(date_format)
fig.autofmt_xdate()
plt.show()
pie¶
Examples:
import matplotlib.pyplot as plt
from matplotlib.font_manager import FontProperties
labels = ['项目1', '项目2', '项目3', '项目4']
fracs = [60, 25, 5, 10]
explode = (0.01, 0, 0, 0)
pathes, texts, autotexts = plt.pie(fracs, labels=labels, explode=explode, counterclock=False,
startangle=90, autopct='%3.1f%%', radius=0.5)
# Make the labels on the small plot easier to read.
font_zh = FontProperties(fname='/usr/share/fonts/truetype/arphic/uming.ttc')
for t in texts:
t.set_size('smaller')
t.set_fontproperties(font_zh) # show labels in zh_CN
for t in autotexts:
t.set_size('x-small')
autotexts[0].set_color('y')
plt.axis('equal') # change ellipse to standard circle
plt.show() # show in tkinter window
# save figure (eps, pdf, pgf, png, ps, raw, rgba, svg, svgz)
fig = plt.gcf()
fig.set_size_inches(6.4, 6.4)
fig.savefig('test2png.png', dpi=100)
fig.savefig('/tmp/test.png')
stacked bar¶
Examples:
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.font_manager import FontProperties
font_zh = FontProperties(fname='/usr/share/fonts/truetype/arphic/uming.ttc')
N = 5
means_1 = (20, 35, 30, 35, 27)
means_2 = (25, 32, 34, 20, 25)
means_3 = (1, 3, 4, 2 ,4)
bottom_2 = means_1
bottom_3 = [i + j for i, j in zip(means_1, means_2)]
ind = np.arange(N) # the x locations for the groups
width = 0.9 # the width of the bars: can also be len(x) sequence
p1 = plt.bar(ind, means_1, width, color='#d62728')
p2 = plt.bar(ind, means_2, width, color='#10e010', bottom=bottom_2)
p3 = plt.bar(ind, means_3, width, color='#1111e1', bottom=bottom_3)
plt.ylabel('Values')
plt.title('This Is The Title')
plt.xlabel('范围', fontproperties=font_zh)
plt.ylabel('项目1 -- 项目2 -- 项目3', fontproperties=font_zh)
plt.xticks(ind, ('X1', 'X2', 'X3', 'X4', 'X5'), fontproperties=font_zh)
plt.yticks(np.arange(0, 81, 10))
#plt.legend((p1[0], p2[0], p3[0]), ('Item1', 'Item2', 'Item3'))
plt.show()
layout¶
Examples:
import numpy as np
import matplotlib.pyplot as plt
t = np.arange(0.0, 1.0, 0.01)
plt.subplot(121)
plt.plot(t, np.sin(2 * np.pi * t))
plt.subplot(122)
plt.plot(t, np.cos(2 * np.pi * t))
plt.show()
Python Data Analysis¶
Installations¶
Canopy(epdfree)¶
sudo bash ./canopy-1.7.4-rh5-64.sh
# Agree on the license and select installation directory
# start
/home/jxiao1/Canopy/canopy
ipython¶
sudo apt-get install ipython ipython-notebook
# start
ipython
ipython noteboot --pylab=inline
numpy¶
- https://docs.scipy.org/doc/numpy-1.10.1/contents.html
- http://www.numpy.org/
- https://docs.scipy.org/doc/
- https://docs.scipy.org/doc/numpy-dev/user/quickstart.html
Overview¶
NumPy’s main object is the homogeneous multidimensional array(class ndarray). It is a table of elements (usually numbers), all of the same type, indexed by a tuple of positive integers. In Numpy dimensions are called axes. The number of axes is rank.
In the below example below, the array has rank 2 (it is 2-dimensional). The first dimension (axis) has a length of 2, the second dimension has a length of 3.
- ::
- [[ 1., 0., 0.],
- [ 0., 1., 2.]]
Create ndarray¶
Examples:
import numpy as np
a = np.arange(15).reshape(3, 5) # arange(0, 15, 1)
b = np.linspace(0, 2, 9) # 9 numbers from 0 to 2.0, including 2.0
c = np.random.random((2,3)) # 2 X 3 array
a = np.array([2,3,4])
b = np.array([(1.5,2,3), (4,5,6)])
np.empty((2,3)) # value is not initialized, what in memory.
np.zeros((3,4)) # value is initialized to '0'
np.ones((2,3,4), dtype=np.int16) # value is initialized to '1'
np.eye(4) # 4x4 array I
f= lambda x, y: 10*x+y
np.fromfunction(f,(5,4),dtype=int)
Out[176]:
array([[ 0, 1, 2, 3],
[10, 11, 12, 13],
[20, 21, 22, 23],
[30, 31, 32, 33],
[40, 41, 42, 43]])
Note
reshape and slice only return new view of the array, but not copied the data.
change the new view will change the sharing data buffer. If you want to copy
data, copy() is required.
d = a.copy()
Data types¶
https://docs.scipy.org/doc/numpy-1.10.1/user/basics.types.html
dtype is np.float64 in most cases by default.
============= ============== =======================================================================
Data type Character Code Description
============= ============== =======================================================================
object O Python Object
string_ SN String with fixed length of N, e.g. S10
unicode_ UN Unicode with fixed length of N, e.g. U10
bool_ ? Boolean (True or False) stored as a byte
int8 i1 Byte (-128 to 127)
int16 i2 Integer (-32768 to 32767)
int32 i4 Integer (-2147483648 to 2147483647)
int64 i8 Integer (-9223372036854775808 to 9223372036854775807)
uint8 u1 Unsigned integer (0 to 255)
uint16 u2 Unsigned integer (0 to 65535)
uint32 u4 Unsigned integer (0 to 4294967295)
uint64 u8 Unsigned integer (0 to 18446744073709551615)
float16 f2 Half precision float: sign bit, 5 bits exponent, 10 bits mantissa
float32 f4/f Single precision float: sign bit, 8 bits exponent, 23 bits mantissa
float64 f8/d Double precision float: sign bit, 11 bits exponent, 52 bits mantissa
float128 f16/d Extend double precision float.
complex64 c8 Complex number, represented by two 32-bit floats (real and imaginary)
complex128 c16 Complex number, represented by two 64-bit floats (real and imaginary)
complex256 c32 Represented by two 128-bit floats (real and imaginary components)
============= ============== =======================================================================
Examples:
a = np.array([1.0, 2, 3]) # dtype('float64')
a = np.array([1.0, 2, 3], dtype=np.int32) # dtyoe('int32')
a = a.astype(np.int64) # dtype('int64')
a.astype(int).dtype # dtype('int64'), convert pythyon type to numpy type.
a.astype('f').dtype # dtype('float32'), use the character codes
Array Operations¶
- https://docs.scipy.org/doc/numpy-1.10.1/reference/routines.math.html
- https://docs.scipy.org/doc/numpy-1.10.1/reference/routines.logic.html
basic:
a = np.array( [20,30,40,50] )
b = np.arange( 4 )
c = a-b
b**2
b > 10 # return the dtype=bool array
b += a
a *= 3
index:
a = np.array(12).reshape(4, 3)
a[0][0] # the element at raw 0 column 0
a[1:3] # raw from 1 to 3 (not included)
a[::-1] # raw in inverted sequence
a[1] # the second raw
a[-1] # the last raw
a[:, 1] # column 1 in each raw
a[1:3, [1, 2]] # column 1, 2 in raw 1, 2
a[1:5:2, ::3] # 1:5:2 for raw and ::3 for column
x = np.arange(10,1,-1)
x[np.array([3, 3, 1, 8])] # array([7, 7, 9, 2])
a_idx = np.array([1, 2, 1])
a[a_idx] # select item at index 1, 2 ,1
a_idx = np.array([[1, 3], [2, 4]])
a[a_idx] # new array ([[a[1], a[3]], [a[2], a[4]])
y = np.arange(35).reshape(5,7)
y[np.array([0,2,4])] # array([[ 0, 1, 2, 3, 4, 5, 6],
[14, 15, 16, 17, 18, 19, 20],
[28, 29, 30, 31, 32, 33, 34]])
y[np.array([0,2,4]), np.array([0,1,2])] # array([ 0, 15, 30])
y[[0,2,4]][:, [0,1,2]] # array([[ 0, 1, 2],
[14, 15, 16],
[28, 29, 30]])
y[y>20] # array([21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34])
y[np.array([False, False, False, True, True])] # the raws 3 and 4, which is true
y[y<10] = 10 # set all elements which are little than 10 to 10
x = np.arange(30).reshape(2,3,5)
b = np.array([[True, True, False], [False, True, True]])
x[b] # raw 0, 1 in x[0], raw 1, 2 in x[1], total 4x5
y[np.array([0,2,4]),1:3] # raw 0, 2, 4 column 1~2
newaxis:
x = np.arange(5)
xx = x[:,np.newaxis] # 5x1 array([[0],[1],[2],[3],[4]])
xy = x[np.newaxis, :] # 1x5 array([[0, 1, 2, 3, 4]])
xx + xy # 5x5
functions:
#a.ufunc or np.ufunc(a)
abs
fabs
sqrt
square
exp
log/log10/log2
sign # return 1 (>0), 0(=0), -1(<0)
ceil # min integer that >= e for e in elements in a
floor # max integer that <= e for e in elements in a
rint
modf
isnan
isfinite
isinf
cos/sin
tan
logical_not
#np.ufunc(a, b)
add
subtract
multiply
divide/floor_divide
power
maximum, fmax
minimum, fmin
mod
copysign
greater/greater_equal
less/less_equal
equal/not_equal
logical_and/logical_or
logical_xor
# functions for bool
any()
all()
# functions for set
unique(x)
intersect1d(x, y) # in both x and y
union1d(x, y) # in x or y
in1d(x, y) # bool arrary than x[i] in y
setdiff1d(x, y) # in x and not in y
setxor1d(x, y) # only in x or only in y
#function for statistics
a.sum() # sum of all values
a.sum(axis=0) # sum of each axis
a.min()
a.max()
a.mean() # average
a.std()
a.argmin() # index of min
a.argmax() # index of max
a.cumsum() # 累计和
a.cumprod() # 累计积
# examples
(a > 0).sum() # number of element which is greater than 0
np.add(a, b)
for row in b:
for element in b.flat:
list(b.flat)
a = np.floor(10*np.random.random((3,4)))
a.ravel()
a.reshape(6,-1) # -1 means automaticall get 12/6 =2
b1 = np.array([False,True,True])
a[b1,:] # a[b1], show second and third raws which is True
a = np.floor(10*np.random.random((2,12)))
np.hsplit(a,3) # Split a into 3
np.hsplit(a,(3,4)) # Split a after the third and the fourth column
x = np.arange(0,10,2) # x=([0,2,4,6,8])
y = np.arange(5) # y=([0,1,2,3,4])
m = np.vstack([x,y]) # m=([[0,2,4,6,8],
# [0,1,2,3,4]])
xy = np.hstack([x,y]) # xy =([0,2,4,6,8,0,1,2,3,4])
b = np.sort(a)
Data input and output¶
https://docs.scipy.org/doc/numpy-1.10.1/reference/routines.io.html
np.save('filename.npy', a) # save array a as filename.npy dat file
a = np.load('filename.npy') # load filename.npy to array a
np.savez('filename.npz', a=arr_a, b=arr_b) # save array arr_a and arr_b in one file as filename.npz
arch = np.load('filename.npy') # load filename.npz to dictionary {'a': arr_a, 'b': arr_b}
np.savetxt('filename.txt', a, fmt='%4d', delimiter=' ', newline='\n', header='', footer='')
a = np.loadtxt('filename.txt', delimiter=',')
Random sampling¶
https://docs.scipy.org/doc/numpy-dev/reference/routines.random.html https://docs.scipy.org/doc/numpy-1.10.1/reference/routines.random.html#simple-random-data
Examples:
np.random.rand(10) # 10 samples in [0, 1)
np.random.rand(2, 4) # 2 X 4 array, values in [0,1)
np.random.randn(2, 4) # Return samples from “standard normal” distribution.
np.random.randint(0, 10, 10) # int in [0, 10)
np.random.random_integers(0, 10, 10) # int in [0, 10]
np.random.random(10) # floats in [0.0, 1.0)
np.random.choice([1, 2, 3, 4], 10) # 10 samples in the give array [1, 2, 3, 4]
pandas¶
http://pandas.pydata.org/pandas-docs/stable/index.html
Data Structures¶
http://pandas.pydata.org/pandas-docs/stable/dsintro.html
Series
Series is similar as the dictionary, but support more operations.
Example:
# the common import way
import numpy as np
import pandas as pd
from pandas import Series, DataFrame
pd.Series(np.random.randn(5))
pd.Series(np.random.randn(5), index=['a', 'b', 'c', 'd', 'e'])
pd.Series({'a' : 0., 'b' : 1., 'c' : 2.})
pd.Series(5, index=['a', 'b', 'c', 'd', 'e'])
# like a ndarray
s = pd.Series({'a' : 0., 'b' : 1., 'c' : 2.})
s[0] # 0.0
s[1:] # slice from 1 to end, 'b' and 'c' are left
s[s > s.median()] # equal to s[[False, False, True]], only 'c' is left
s[[0, 0, 1]] # select 'a', 'a', 'b'
s[[2, 1]] # select 'c' and 'b'
# like a dictionary
s['a'] # 0.0
s.get('e', np.nan) # support get function too.
np.square(s) # square for each
s * 2 # i * 2 for i in s
s.name = 'something'
s.rename('otherthing')
DataFrame
DataFrame is a 2-dimensional labeled data structure with columns of potentially different types. You can think of it like a spreadsheet or SQL table, or a dict of Series objects. It is generally the most commonly used pandas object. Like Series, DataFrame accepts many different kinds of input:
- Dict of 1D ndarrays, lists, dicts, or Series
- 2-D numpy.ndarray
- Structured or record ndarray
- A Series
- Another DataFrame
Examples:
# From dictionary of lists
#---------------------------------------------------------------------------
d = {'Column1': [1, 2, 3, 4],
'Column2': [1., 2., 3., 4.]}
pd.DataFrame(d, index=['a', 'b', 'c', 'd'])
Out[51]:
Column1 Column2
a 1 1.0
b 2 2.0
c 3 3.0
d 4 4.0
pd.DataFrame({'a' : [1, 0, 1], 'b' : [0, 1, 1] }, dtype=bool)
Out[74]:
a b
0 True False
1 False True
2 True True
# From ndarrays
#---------------------------------------------------------------------------
dates = pd.date_range('2013-01-01', periods=3)
pd.DataFrame(np.random.randn(3,2), index=dates, columns=list('AB'))
Out[63]:
A B
2013-01-01 1.017122 -0.509179
2013-01-02 -0.165409 -0.185033
2013-01-03 -0.108724 -0.775507
# From dictionary for series
#---------------------------------------------------------------------------
d = {'Column1': pd.Series([1., 2., 3.], index=['a', 'b', 'c']),
'Column2': pd.Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}
pd.DataFrame(d)
# Select part of the data, and indexs
pd.DataFrame(d, index=['c', 'b', 'a'], columns=['Column1', 'Column3'])
Out[47]:
Column1 Column3
c 3.0 NaN
b 2.0 NaN
a 1.0 NaN
# From dict of objects that can be converted to series-like.
#---------------------------------------------------------------------------
In [10]: df2 = pd.DataFrame({ 'A' : 1.,
....: 'B' : pd.Timestamp('20130102'),
....: 'C' : pd.Series(1,index=list(range(4)),dtype='float32'),
....: 'D' : np.array([3] * 4,dtype='int32'),
....: 'E' : pd.Categorical(["test","train","test","train"]),
....: 'F' : 'foo' })
....:
In [11]: df2
Out[11]:
A B C D E F
0 1.0 2013-01-02 1.0 3 test foo
1 1.0 2013-01-02 1.0 3 train foo
2 1.0 2013-01-02 1.0 3 test foo
3 1.0 2013-01-02 1.0 3 train foo
#From structured or record array
#---------------------------------------------------------------------------
data = np.zeros((2,), dtype=[('A', 'i4'),('B', 'f4'),('C', 'a10')])
data[:] = [(1,2.,'Hello'), (2,3.,"World")]
pd.DataFrame(data)
Out[44]:
A B C
0 1 2.0 Hello
1 2 3.0 World
pd.DataFrame(data, index=['first', 'second'])
Out[45]:
A B C
first 1 2.0 Hello
second 2 3.0 World
pd.DataFrame(data, columns=['C', 'A'])
Out[46]:
C A
0 Hello 1
1 World 2
pd.DataFrame.from_records(data, index='C')
Out[53]:
A B
C
Hello 1 2.0
World 2 3.0
#From a list of dicts
#---------------------------------------------------------------------------
data2 = [{'a': 1, 'b': 2}, {'a': 5, 'b': 10, 'c': 20}]
pd.DataFrame(data2, index=['R1', 'R2'])
Out[48]:
a b c
R1 1 2 NaN
R2 5 10 20.0
Series Operations¶
Examples:
s1 = pd.Series(np.random.randn(4))
s1[s1>0]
s1 * 2
s1 + 5
s1.index
s1.values
s2 = pd.Series(s1.values,index=['norm_'+ str(i) for i in xrange(4)])
s2['norm_1':'norm_3'] # raws from 'norm_1' to 'norm_3' (included)
s2[['norm_1','norm_3']] # raws 'norm_1' and 'norm_3'
data = pd.Series([1, 2.1, np.nan, None, '', 0])
data.isnull() # only np.nan and None is null.
DataFrame Operations¶
Indexing/Selection:
df['A']
df[['A', 'B']]
df[0:3]
df.loc['bar':'kar']
df.loc['2013-01-03', 'A']
df.loc['2013-01-03':'2013-01-06', ['A', 'B']]
df.iloc[1,1]
df.iloc[3:5,0:2]
df.iloc[[1,2,4],[0,2]]
df.iloc[:,1:3]
df.ix('2013-01-03', 'A')
df.at[dates[0],'A']
df.iat[1,1]
df[df.A > 0]
df.loc[(df.A > 0) & (df.C >0), 'B']
crit1 = df.A > 0
crit2 = df.B > 0
crit3 = df.C > 0
crit = crit1 & crit2 & crit3
df[crit]
df[~((df.A <= 6) & (df.index.isin([0,2,4])))] # ~ means not
| Operation | Syntax | Result |
|---|---|---|
| Select column | df[‘col’]/df.col | Series |
| Select row by label | df.loc[label] | Series |
| Select row by integer location | df.iloc[loc] | Series |
| Slice rows | df[5:10] | DataFrame |
| Select Columns | df[[‘A’, ‘B’]] | DataFrame |
| Select rows by boolean vector | df[bool_vec] | DataFrame |
Deletion:
del df['Column1']
Addtion:
df['Status'] = 'ready' # Set the whole new column 'Status' to 'ready'
df['ColumnN'] = df['Column1'] * df['Column2']
df['ColumnN'] = df['Status'] == 'ready'
df.insert(1, 'Column1-copy', df['Column1'])
Copy:
df2 = df.copy()
Data view:
df.index
df.columns
df.values
df.head(2) # first 5 by default
df.tail(3) # last 5 by default
df.describe() # Quick statistic summary of your data
df.sort_index(axis=1, ascending=False)
df.sort_values(by='B')
Missing Data
df1.dropna(how=’all’, axis=1) df1.fillna(value=5, inplace=True) df1.fillna({1:10, 2:20}) # 10 for column 1 and 20 for column 2 pd.isnull(df1) pd.notnull(df1)
Data alignment and arithmetic:
df - df.iloc[0]
df.sub(df.iloc[0], axis=1)
df.sub(df['A'], axis=0)
df*5 + 2
df.T
np.square(df)
Join:
pd.merge(df1, df2, on='key_index')
Concat:
pd.concat(list_of_dfs)
Append:
pd.append(df, df.iloc[3], ignore_index=True) # Generate new index
Group:
df.groupby('A').sum()
Sort:
df.sort_values(by='A')
if/then:
df.ix[df.A >= 0,'B'] = True; df
df_mask = pd.DataFrame({'A' : [True] * 4, 'B' : [False] * 4,'C' : [True,False] * 2})
df.where(df_mask, 1000) # Set all 'False' to the value 1000
# add new column 'result' to show status of value 'A', 'high' if >5, otherwise 'low'.
df['result'] = np.where(df['A'] > 5,'high','low'); df
Plot in Pandas¶
Both Series and DataFrame support plot function.
Examples:
ts = pd.Series(np.random.randn(10, index=pd.date_range('1/1/2000', periods=10))
ts1.cumsum() # the sum of all previous values
ts1.plot()
df = pd.DataFrame(np.random.randn(10, 4), index=ts.index, columns=['A', 'B', 'C', 'D'])
df1 = df.cumsum()
pf = plt.figure()
pf.legend(loc='best')
df.plot()
df1.plot(kind='bar')
Data import and export¶
csv:
pd.read_csv('foo.csv')
df.to_csv('foo.csv')
excel:
pd.read_excel('foo.xlsx', 'Sheet1', index_col=None, na_values=['NA'])
df.to_excel('foo.xlsx', sheet_name='Sheet1')
binary file:
# offsets are larger than the size of the type because of struct padding
names = 'count', 'avg', 'scale'
formats = 'i4', 'f8', 'f4'
offsets = 0, 8, 16
dt = np.dtype({'names': names, 'offsets': offsets, 'formats': formats}, align=True)
df = pd.DataFrame(np.fromfile('binary.dat', dt))
Python GUI¶
Phoenix(wxPython)¶
Phoenix is the code name of for the new version of wxPython, which runs on the python3.
colorama¶
Cross-platform colored terminal text: - https://pypi.python.org/pypi/colorama - https://github.com/tartley/colorama
Python Differences Between Versions¶
Mainly about the differences between Python2.7 and Python3.5
- Rename raw_input() to input(), and always return string。Remove original input()
- Rename xrange() to range(), and remove original range()
- ‘a/b’ return integer in python2, but float in python3 by default.
Python Opensource¶
How to read the open source codes¶
frame stacks + pygraphviz: https://www.zhihu.com/question/26766601
Github open source projects¶
https://github.com/kennethreitz/requests
https://github.com/kennethreitz/python-guide
https://github.com/kennethreitz/tablib (dataset library, output Json, excel, html, yaml …)
https://github.com/kennethreitz/hypermark (markdown to html)
https://github.com/kennethreitz/inbox.py (smtp email)
https://github.com/kennethreitz/envoy (subprocesses for humans)
https://github.com/mitsuhiko/pluginbase
https://github.com/mitsuhiko/platter/ (simplifies deployments)
https://github.com/bottlepy/bottle
https://github.com/crsmithdev/arrow (dates, times, and timestamps)
Flask Beginner¶
Installation¶
http://flask.pocoo.org/docs/0.10/installation/#installation
virtualenv:
Virtualenv enables multiple side-by-side installations of Python, one for each project. It doesn’t actually install separate copies of Python, but it does provide a clever way to keep different project environments isolated.
$ sudo easy_install virtualenvOr even better:
$ sudo pip install virtualenvMaybe it’s even in your package manager. If you use Ubuntu, try:
$ sudo apt-get install python-virtualenvOnce you have virtualenv installed, just fire up a shell and create your own environment:
$ mkdir myproject $ cd myproject $ virtualenv [--python=python3] venv # default is python2Now, whenever you want to work on a project, you only have to activate the corresponding environment:
$ . venv/bin/activateor$ source venv/bin/activateAfter all, if you want to exit the virtualenv:
(venv) $ deactivate
flask:
Now you can just enter the following command to get Flask activated in your virtualenv:
(venv) $ pip install flask jinja2 werkzeugAny time, you can generate the required packages list like this:
(venv) $ pip freeze > ./requirements.txtAnd set up the same virtual environment in other place list this:
(venv) $ pip install -r requirements.txt
Quickstart¶
http://flask.pocoo.org/docs/0.10/quickstart/#quickstart
A minimal Flask application looks something like this:
Create hello.py:
from flask import Flask
app = Flask(__name__)
@app.route('/user/<name>')
def hello(name):
return 'Hello %s!' % name
if __name__ == '__main__':
# Listen any client host
# Never use debug mode in product project
app.run(host='0.0.0.0', debug=True)
Note
If you enable debug support the server will reload itself on code changes, and it will also provide you with a helpful debugger if things go wrong. Because of security risk, never use debug mode on production machines.
Start the service:
$ python hello.py >/dev/null 2>&1 &
Now head over to http://127.0.0.1:5000/user/flask, and you should see your ‘hello flask!’ greeting. Or in command line mode:
$ curl -i http://127.0.0.1:5000/user/flask
Project Template¶
Deploying to a Web Server¶
Books¶
Flask Web Development: Developing Web Application with Python
Author: Miguel Grinberg github: https://github.com/miguelgrinberg Example source code: https://github.com/miguelgrinberg/flasky Blog: http://blog.miguelgrinberg.com
Flask Routing¶
@app.route('/')
def index(): pass
@app.route('/hello')
def hello(): pass
Variable in the URL¶
<variable_name> or <converter:variable_name>
| Converters | Description |
|---|---|
| string | accepts any text without a slash (the default) |
| int | accepts integers |
| float | like int but for floating point values |
| path | like the default but also accepts slashes |
@app.route('/user/<username>')
def show_user_profile(username): pass
@app.route('/post/<int:post_id>')
def show_post(post_id): pass
# Support variable and also default value
@app.route('/hello/')
@app.route('/hello/<name>')
def hello(name=None): pass
@app.route('/<name>/test/<age>')
def user(name, age):
return '<h1>Hello %s, age is %s!</h1>' %(name, age)
Trailing slash in the URL¶
Flask’s URL rules are based on Werkzeug’s routing module.
# '/projects' is redirected to '/projects/'
@app.route('/projects/')
def projects(): pass
# '/about/' will result in 404 error
@app.route('/about')
def about(): pass
URL Building¶
To build a URL to a specific function you can use the url_for() function. It accepts the name of the function as first argument and a number of keyword arguments, each corresponding to the variable part of the URL rule. Unknown variable parts are appended to the URL as query parameters.
Here are some examples:
from flask import Flask, url_for
app = Flask(__name__)
@app.route('/')
def index(): pass
@app.route('/login')
def login(): pass
@app.route('/user/<username>')
def profile(username): pass
with app.test_request_context():
print url_for('index')
print url_for('index', _external=True)
print url_for('login')
print url_for('login', next='/')
print url_for('profile', username='John Doe')
The result:
/
http://localhost:5000/
/login
/login?next=/
/user/John%20Doe
Note
The_request_context() method tells Flask to behave as though it is handling a request, even though we are interacting with it through a Python shell. (see also: http://flask.pocoo.org/docs/0.10/quickstart/#context-locals).
HTTP Methods¶
By default, a route only answers to GET requests, but that can be changed by providing the methods argument to the route() decorator.
@app.route('/login', methods=['GET', 'POST'])
def login():
if request.method == 'POST':
do_the_login()
else:
show_the_login_form()
Static files¶
Dynamic web applications also need static files. That’s usually where the CSS and JavaScript files are coming from. Ideally your web server is configured to serve them for you, but during development Flask can do that as well. Just create a folder called static in your package or next to your module and it will be available at /static on the application.
As static is the default routing function created when app is initialized. To generate URLs for static files, use the special ‘static’ endpoint name:
url_for('static', filename='style.css')
The file has to be stored on the filesystem as static/style.css.
Redirects¶
To redirect a user to another endpoint, use the redirect() function; to abort a request early with an error code, use the abort() function:
from flask import abort, redirect, url_for
@app.route('/')
def index():
return redirect(url_for('login'))
@app.route('/login')
def login():
abort(401)
this_is_never_executed()
Error Pages¶
By default a black and white error page is shown for each error code. You can use the errorhandler() decorator to customize the error page:
from flask import render_template
@app.errorhandler(404)
def page_not_found(error):
return render_template('page_not_found.html'), 404
Note
The 404 after the render_template() call. This tells Flask that the status code of that page should be 404 which means not found. By default 200 is assumed which translates to: all went well.
Flask Bluepoint¶
A Blueprint object works similarly to a Flask application object, but it is not actually an application. It is a set of operations which can be registered on an application, even multiple times. We can group views and resources in many subclass blueprint and flexibly register them to application later.
Basic example¶
from flask import Blueprint, render_template, abort
from jinja2 import TemplateNotFound
simple_page = Blueprint('simple_page', __name__,
template_folder='templates')
@simple_page.route('/', defaults={'page': 'index'})
@simple_page.route('/<page>')
def show(page):
try:
return render_template('pages/%s.html' % page)
except TemplateNotFound:
abort(404)
Registering Blueprints¶
from flask import Flask
from yourapplication.simple_page import simple_page
app = Flask(__name__)
app.register_blueprint(simple_page)
If you check the rules registered on the application, you will find these:
[<Rule '/static/<filename>' (HEAD, OPTIONS, GET) -> static>,
<Rule '/<page>' (HEAD, OPTIONS, GET) -> simple_page.show>,
<Rule '/' (HEAD, OPTIONS, GET) -> simple_page.show>]
Blueprints however can also be mounted at different locations:
app.register_blueprint(simple_page, url_prefix='/pages')
And sure enough, these are the generated rules:
[<Rule '/static/<filename>' (HEAD, OPTIONS, GET) -> static>,
<Rule '/pages/<page>' (HEAD, OPTIONS, GET) -> simple_page.show>,
<Rule '/pages/' (HEAD, OPTIONS, GET) -> simple_page.show>]
On top of that you can register blueprints multiple times though not every blueprint might respond properly to that. In fact it depends on how the blueprint is implemented if it can be mounted more than once.
Blueprint Resources¶
Sometimes you might want to introduce a blueprint only for the resources.
Like for regular applications, blueprints are considered to be contained in a folder. While multiple blueprints can originate from the same folder, it does not have to be the case and it’s usually not recommended.
The folder is inferred from the second argument to Blueprint which is usually __name__. This argument specifies what logical Python module or package corresponds to the blueprint. If it points to an actual Python package that package (which is a folder on the filesystem) is the resource folder. If it’s a module, the package the module is contained in will be the resource folder. The Blueprint.root_path property show what the resource folder is:
>>> simple_page.root_path
'/Users/username/TestProject/yourapplication'
To quickly open sources from this folder you can use the open_resource() function:
with simple_page.open_resource('static/style.css') as f:
code = f.read()
Static Files¶
A blueprint can expose a folder with static files by providing a path to a folder on the filesystem via the static_folder keyword argument. It can either be an absolute path or one relative to the folder of the blueprint:
admin = Blueprint('admin', __name__, static_folder='static')
By default the rightmost part of the path is where it is exposed on the web. Because the folder is called static here it will be available at the location of the blueprint + /static. Say the blueprint is registered for /admin the static folder will be at /admin/static.
The endpoint is named blueprint_name.static so you can generate URLs to it like you would do to the static folder of the application:
url_for('admin.static', filename='style.css')
Templates¶
If you want the blueprint to expose templates you can do that by providing the template_folder parameter to the Blueprint constructor:
admin = Blueprint('admin', __name__, template_folder='templates')
As for static files, the path can be absolute or relative to the blueprint resource folder. The template folder is added to the searchpath of templates but with a lower priority than the actual application’s template folder. That way you can easily override templates that a blueprint provides in the actual application.
So if you have a blueprint in the folder yourapplication/admin and you want to render the template ‘admin/index.html’ and you have provided templates as a template_folder you will have to create a file like this: yourapplication/admin/templates/admin/index.html.
Building URLs¶
If you want to link from one page to another you can use the url_for() function just like you normally would do just that you prefix the URL endpoint with the name of the blueprint and a dot (.):
url_for('admin.index')
Additionally if you are in a view function of a blueprint or a rendered template and you want to link to another endpoint of the same blueprint, you can use relative redirects by prefixing the endpoint with a dot only:
url_for('.index')
This will link to admin.index for instance in case the current request was dispatched to any other admin blueprint endpoint.
Flask Templates¶
Generating HTML from within Python is not fun, and actually pretty cumbersome because you have to do the HTML escaping on your own to keep the application secure. Because of that Flask configures the Jinja2 template engine for you.
http://jinja.pocoo.org/docs/dev/
Rendering Templates¶
To render a template you can use the render_template() method. All you have to do is provide the name of the template and the variables you want to pass to the template engine as keyword arguments.
Here’s a simple example of how to render a template:
from flask import render_template
@app.route('/hello/')
@app.route('/hello/<name>')
def hello(name=None):
return render_template('hello.html', name=name)
Flask will look for templates in the templates folder. So if your application is a module, this folder is next to that module, if it’s a package, it’s actually inside your package:
Case 1: a module:
/application.py
/templates
/hello.html
Case 2: a package:
/application
/__init__.py
/templates
/hello.html
Example of the template:
<!doctype html>
<title>Hello from Flask</title>
{% if name %}
<h1>Hello {{ name }}!</h1>
{% else %}
<h1>Hello World!</h1>
{% endif %}
Note
Inside templates you also have access to the request, session and g objects as well as the get_flashed_messages() function.
Automatic escaping is enabled, so if name contains HTML it will be escaped automatically. If you can trust a variable and you know that it will be safe HTML you can mark it as safe by using the Markup class or by using the safe filter in the template.
Global Variables within templates¶
The following global variables are available within Jinja2 templates by default.
config:
The current configuration object (flask.config) This is now always available, even in imported templates.
request:
The current request object (flask.request). This variable is unavailable if the template was rendered without an active request context.
session:
The current session object (flask.session). This variable is unavailable if the template was rendered without an active request context.
g:
The request-bound object for global variables (flask.g). This variable is unavailable if the template was rendered without an active request context.
url_for():
The flask.url_for() function.
get_flashed_messages()
The flask.get_flashed_messages() function.
Context Processors¶
To inject new variables automatically into the context of a template, context processors exist in Flask. Context processors run before the template is rendered and have the ability to inject new values into the template context. A context processor is a function that returns a dictionary. The keys and values of this dictionary are then merged with the template context, for all templates in the app:
@app.context_processor
def inject_user():
return dict(user=g.user)
The context processor above makes a variable called user available in the template with the value of g.user. This example is not very interesting because g is available in templates anyways, but it gives an idea how this works.
Variables are not limited to values; a context processor can also make functions available to templates:
@app.context_processor
def utility_processor():
def format_price(amount, currency=u'€'):
return u'{0:.2f}{1}'.format(amount, currency)
return dict(format_price=format_price)
The context processor above makes the format_price function available to all templates:
{{ format_price(0.33) }}
You could also build format_price as a template filter, but this demonstrates how to pass functions in a context processor.
Syntax of Jinja templates¶
http://jinja.pocoo.org/docs/dev/templates/
The default Jinja delimiters are configured as follows:
- {% … %} for Statements
- {{ … }} for Expressions to print to the template output
- {# … #} for Comments not included in the template output
- # … ## for Line Statements
Examples:
{# join string and variable value to one string #}
{{ "Hello " ~ name ~ "!" }}
Filters¶
http://jinja.pocoo.org/docs/dev/templates/#builtin-filters
Variables can be modified by filters. Filters are separated from the variable by a pipe symbol (|) and may have optional arguments in parentheses. Multiple filters can be chained. The output of one filter is applied to the next.
For example, {{ name|striptags|title }} will remove all HTML Tags
from variable name and title-case the output (title(striptags(name))).
Filters that accept arguments have parentheses around the arguments, just like
a function call. For example: {{ listx|join(', ') }} will join a list with
commas (str.join(‘, ‘, listx)).
If you want to register your own filters in Jinja2 you have two ways to do that. You can either put them by hand into the jinja_env.filter of the application:
def reverse_filter(s):
return s[::-1]
app.jinja_env.filters['reverse'] = reverse_filter
Or use the template_filter() decorator:
@app.template_filter('reverse')
def reverse_filter(s):
return s[::-1]
List of Popular Builtin Filters:
| Filter Name | Description |
|---|---|
| abs | Return the absolute value of the argument. |
| capitalize | The first character will be uppercase, all others lowercase |
| title | Return a titlecased version of the value. |
| lower | Convert a value to lowercase. |
| upper | Convert a value to uppercase. |
| indent | Return a copy of string each line indented by 4 spaces. |
| center | Centers the value in a field of a given width. |
| trim | Strip leading and trailing whitespace. |
| wordwrap | Wrap string, line length is 79 characters by default. |
| default | If the value is undefined it will return the default value. |
| dictsort | Sort a dict and yeild (key, value) pairs. (‘for xx in xx’) |
| escape | Convert &, <, >, ‘, and ” in string to HTML-safe sequences. |
| filesizeformat | Format the value like a ‘human-readable’ file size (13 kB). |
| first | Return the first item of a sequence. |
| last | Return the last item of a sequence. |
| random | Return a random item from the sequence. |
| format | {{ "%s - %s"|format("Hello?", "Foo!") }} |
| groupby | Group items in dict with the same value to a specified key. |
| int | Convert the value into an integer. |
| float | Convert the value into a floating point number. |
| list | Convert the value into a list. |
| string | Make a string unicode if it isn’t already. |
| join | Return the concatenation of the strings in the sequence. |
| length | Return the number of items of a sequence or mapping. |
| wordcount | Count the words in that string. |
| map() | Filte on a sequence of objects or looks up an attribute. |
| pprint | Pretty print a variable. Useful for debugging. |
| select | Filters a sequence of objects. {{ numbers|select(“odd”) }} |
| selectattr | Filters a sequence of objects by attribute. |
| reject | Filters a sequence, {{ numbers|reject(“odd”) }} |
| rejectattr | Filters a sequence/objects by applying test to an attribute |
| replace | Return a copy of the value after replaced old by new. |
| reverse | Reverse the object. |
| round | Round a given precision. {{ 42.55|round}} ->43.0 |
| safe | Safe means this variable will not be escaped. |
| striptags | Strip XML tags and replace adjacent whitespace by one space |
| slice | Slice an iterator |
| sort | Sort an iterable. |
| sum | {{ items|sum(attribute='price') }} |
| truncate | {{ "foo bar baz"|truncate(9, True) }} -> “foo ba…” |
| urlencode | Escape strings for use in URLs (uses UTF-8 encoding). |
| urlize | Converts URLs in plain text into clickable links. |
| xmlattr | Create an XML attribute string based on the items in a dict |
Tests¶
{% if variable is defined %}
value of variable: {{ variable }}
{% else %}
variable is not defined
{% endif %}
{% if foo.expression is equalto 42 %}
the foo attribute evaluates to the constant 42
{% endif %}
{{ users|selectattr("email", "equalto", "foo@bar.invalid") }}
List of Builtin Tests
| Test Name | Description |
|---|---|
| callable | Return whether the object is callable (has __call__ method) |
| defined | Return true if the variable is defined. |
| undefined | Like defined() but the other way round. |
| divisibleby | Check if a variable is divisible by the specified number. |
| iterable | Check if it’s possible to iterate over an object. |
| escaped | Check if the value is escaped. |
| equalto | Check if an object has the same value as another object. |
| sameas | Check if the objects points to the same memory address |
| odd | Return true if the variable is odd. |
| even | Return true if the variable is even. |
| lower | Return true if the variable is lowercased. |
| upper | Return true if the variable is uppercased. |
| mapping | Return true if the object is a mapping (dict etc.). |
| none | Return true if the variable is none. |
| number | Return true if the variable is a number. |
| sequence | Return true if the variable is a sequence. |
| string | Return true if the object is a string. |
Template Inheritance¶
extends:
{% extends "layout/default.html" %}
{% extends layout_template if layout_template is defined else 'master.html' %}
include:
{% include 'header.html' %}
Body
{% include 'footer.html' %}
{% include "sidebar.html" ignore missing %}
{% include "sidebar.html" ignore missing with context %}
{% include "sidebar.html" ignore missing without context %}
{% include ['page_detailed.html', 'page.html'] %}
{% include ['special_sidebar.html', 'sidebar.html'] ignore missing %}
import:
Imagine we have a helper module that renders forms (called forms.html):
{% macro input(name, value='', type='text') -%}
<input type="{{ type }}" value="{{ value|e }}" name="{{ name }}">
{%- endmacro %}
{%- macro textarea(name, value='', rows=10, cols=40) -%}
<textarea name="{{ name }}" rows="{{ rows }}" cols="{{ cols
}}">{{ value|e }}</textarea>
{%- endmacro %}
The easiest and most flexible way to access a template’s variables and macros is to import the whole template module into a variable:
{% import 'forms.html' as forms %}
<dl>
<dt>Username</dt>
<dd>{{ forms.input('username') }}</dd>
<dt>Password</dt>
<dd>{{ forms.input('password', type='password') }}</dd>
</dl>
<p>{{ forms.textarea('comment') }}</p>
You can import specific names from a template into the current namespace:
{% from 'forms.html' import input as input_field, textarea %}
<dl>
<dt>Username</dt>
<dd>{{ input_field('username') }}</dd>
<dt>Password</dt>
<dd>{{ input_field('password', type='password') }}</dd>
</dl>
<p>{{ textarea('comment') }}</p>
Note
By default, included templates are passed the current context and imported templates are not. The reason for this is that imports, unlike includes, are cached; as imports are often used just as a module that holds macros.
This is only the default behaviour, you can change it if you want to:
{% from 'forms.html' import input with context %}
{% include 'header.html' without context %}
block¶
{% block sidebar %}
{{ super() }}
{% for item in seq %}
<li>{% block loop_item scoped %}{{ item }}{% endblock %}</li>
{% endfor %}
{% endblock sidebar %}
Note
supper block is to inherit the contents in the pararnt block.
Note
Blocks can be nested for more complex layouts. However, per default blocks may not access variables from outer scopes. ‘item’ variable is out of item in above case because it defined out of the block. ‘scoped’ is to fix this kind of issue.
macro block¶
Macros are comparable with functions in regular programming languages. They are useful to put often used idioms into reusable functions.
Here’s a small example of a macro that renders a form element:
{% macro input(name, value='', type='text', size=20) -%}
<input type="{{ type }}" name="{{ name }}" value="{{
value|e }}" size="{{ size }}">
{%- endmacro %}
The macro can then be called like a function in the namespace:
<p>{{ input('username') }}</p>
<p>{{ input('password', type='password') }}</p>
Note
If the macro was defined in a different template, you have to import it first.
Note
If a macro name starts with an underscore, it’s not can’t be exported/imported.
call block¶
Here’s an example of how a call block can be used with arguments:
{% macro dump_users(users) -%}
<ul>
{%- for user in users %}
<li><p>{{ user.username|e }}</p>{{ caller(user) }}</li>
{%- endfor %}
</ul>
{%- endmacro %}
{# define the caller(user) in above macro block #}
{% call(user) dump_users(list_of_user) %}
<dl>
<dl>Realname</dl>
<dd>{{ user.realname|e }}</dd>
<dl>Description</dl>
<dd>{{ user.description }}</dd>
</dl>
{% endcall %}
filter block¶
Filter sections allow you to apply regular Jinja2 filters on a block of template data. Just wrap the code in the special filter section:
{% filter upper %}
This text becomes uppercase
{% endfilter %}
assignments block¶
{% set navigation = [('index.html', 'Index'), ('about.html', 'About')] %}
{% set key, value = call_something() %}
{% set navigation %}
<li><a href="/">Index</a>
<li><a href="/downloads">Downloads</a>
{% endset %}
List of Control Structures¶
for: Loop over each item in a sequence.
Inside of a for-loop block, you can access some special variables:
| Variable | Description |
|---|---|
| loop.index | The current iteration of the loop. (1 indexed) |
| loop.index0 | The current iteration of the loop. (0 indexed) |
| loop.revindex | The number of iterations from the end of the loop (1 indexed) |
| loop.revindex0 | The number of iterations from the end of the loop (0 indexed) |
| loop.first | True if first iteration. |
| loop.last | True if last iteration. |
| loop.length | The number of items in the sequence. |
| loop.cycle | A helper function to cycle between a list of sequences. See the explanation below. |
| loop.depth | Indicates how deep in deep in a recursive loop the rendering currently is. Starts at level 1 |
| loop.depth0 | Indicates how deep in deep in a recursive loop the rendering currently is. Starts at level 0 |
For example:
{% for user in users %}
<li>{{ user.username|e }}</li>
{% endfor %}
<dl>
{% for key, value in my_dict.iteritems() %}
<dt>{{ key|e }}</dt>
<dd>{{ value|e }}</dd>
{% endfor %}
</dl>
# will get the value 'odd', 'even' in turn.
{% for row in rows %}
<li class="{{ loop.cycle('odd', 'even') }}">{{ row }}</li>
{% endfor %}
{% for user in users if not user.hidden %}
<li>{{ user.username|e }}</li>
{% endfor %}
{% for user in users %}
{%- if loop.index >= 10 %}{% break %}{% endif %}
{%- endfor %}
{% for user in users %}
{%- if loop.index is even %}{% continue %}{% endif %}
...
{% endfor %}
Note
Note, however, that Python dicts are not ordered; so you might want to either pass a sorted list or a collections.OrderedDict to the template, or use the dictsort filter.
Note
Unlike in Python, it’s not possible to break or continue in a loop. You can, however, filter the sequence during iteration, which allows you to skip items.
If no iteration took place because the sequence was empty or the filtering removed all the items from the sequence, you can render a default block by using else:
<ul>
{% for user in users %}
<li>{{ user.username|e }}</li>
{% else %}
<li><em>no users found</em></li>
{% endfor %}
</ul>
Note
In Python, else blocks are executed whenever the corresponding loop did not break. Since Jinja loops cannot break anyway, a slightly different behavior of the else keyword was chosen.
if: The if statement in Jinja is comparable with the Python if statement.
For example:
{% if users %}
<ul>
{% for user in users %}
<li>{{ user.username|e }}</li>
{% endfor %}
</ul>
{% endif %}
{% if kenny.sick %}
Kenny is sick.
{% elif kenny.dead %}
You killed Kenny! You bastard!!!
{% else %}
Kenny looks okay --- so far
{% endif %}
autoescape expression¶
{% autoescape true %}
Autoescaping is active within this block
{% endautoescape %}
With Statement¶
{% with %}
{% set foo = 42 %}
{{ foo }} foo is 42 here
{% endwith %}
foo is not visible here any longer
i18n¶
{% trans book_title=book.title, author=author.name %}
This is {{ book_title }} by {{ author }}
{% endtrans %}
{# `_` is the alias of the gettext function #}
{{ _('Hello World!') }}
{{ gettext('Hello %(name)s!', name='World') }}
Flask Request Object¶
Overview Example¶
from flask import request
@app.route('/login', methods=['POST', 'GET'])
def login():
error = None
if request.method == 'POST':
if valid_login(request.form['username'],
request.form['password']):
return log_the_user_in(request.form['username'])
else:
error = 'Invalid username/password'
# the code below is executed if the request method
# was GET or the credentials were invalid
return render_template('login.html', error=error)
Response to Request:
- Plain text string
- response object
- redirect(‘<ohter-url>’)
- abort(<error_code>)
dir(request)
"['__class__', '__delattr__', '__dict__', '__dir__', '__doc__',
'__enter__', '__eq__', '__exit__', '__format__', '__ge__',
'__getattribute__', '__gt__', '__hash__', '__init__',
'__le__', '__lt__', '__module__', '__ne__', '__new__',
'__reduce__', '__reduce_ex__', '__repr__', '__setattr__',
'__sizeof__', '__str__', '__subclasshook__', '__weakref__',
'_get_file_stream', '_get_stream_for_parsing', '_is_old_module',
'_load_form_data', '_parse_content_type',
'accept_charsets', 'accept_encodings', 'accept_languages', 'accept_mimetypes', 'access_route',
'application', 'args', 'authorization', 'base_url', 'blueprint', 'cache_control',
'charset', 'close', 'content_encoding', 'content_length', 'content_md5', 'content_type',
'cookies', 'data', 'date', 'dict_storage_class', 'disable_data_descriptor',
'encoding_errors', 'endpoint', 'environ', 'files',
'form', 'form_data_parser_class', 'from_values', 'full_path',
'get_data', 'get_json', 'headers', 'host', 'host_url',
'if_match', 'if_modified_since', 'if_none_match', 'if_range', 'if_unmodified_since',
'input_stream', 'is_json', 'is_multiprocess', 'is_multithread',
'is_run_once', 'is_secure', 'is_xhr', 'json', 'list_storage_class',
'make_form_data_parser', 'max_content_length', 'max_form_memory_size', 'max_forwards',
'method', 'mimetype', 'mimetype_params', 'module', 'on_json_loading_failed',
'parameter_storage_class', 'path', 'pragma', 'query_string', 'range', 'referrer',
'remote_addr', 'remote_user', 'routing_exception', 'scheme', 'script_root',
'shallow', 'stream', 'trusted_hosts', 'url', 'url_charset', 'url_root', 'url_rule',
'user_agent', 'values', 'view_args', 'want_form_data_parsed']"
Get the args in the URL¶
To access parameters submitted in the URL (?key=value):
searchword = request.args.get('key', '')
Callback Functions¶
Decorator to define callback function:
Global variable ‘g’ is valid in above functions. For example, we can use g.user to share login information among these functions.
Flask Response Object¶
The return value from a view function is automatically converted into a response object for you. If the return value is a string it’s converted into a response object with the string as response body, an 200 OK error code and a text/html mimetype.
The logic that Flask applies to converting return values into response objects is as follows:
- If a response object of the correct type is returned it’s directly returned from the view.
- If it’s a string, a response object is created with that data and the default parameters.
- If a tuple is returned the items in the tuple can provide extra information. Such tuples have to be in the form (response, status, headers) where at least one item has to be in the tuple. The status value will override the status code and headers can be a list or dictionary of additional header values.
- If none of that works, Flask will assume the return value is a valid WSGI application and convert that into a response object.
If you want to get hold of the resulting response object inside the view, you can use the make_response() function. Imagine you have a view like this:
@app.errorhandler(404)
def not_found(error):
return render_template('error.html'), 404
You just need to wrap the return expression with make_response() and get the response object to modify it, then return it:
@app.errorhandler(404)
def not_found(error):
resp = make_response(render_template('error.html'), 404)
resp.headers['X-Something'] = 'A value'
return resp
Flask Cookies¶
To access cookies you can use the cookies attribute. To set cookies you can use the set_cookie method of response objects. The cookies attribute of request objects is a dictionary with all the cookies the client transmits. If you want to use sessions, do not use the cookies directly but instead use the Sessions in Flask that add some security on top of cookies for you.
Reading cookies:
from flask import request
@app.route('/')
def index():
username = request.cookies.get('username')
# use cookies.get(key) instead of cookies[key] to not get a
# KeyError if the cookie is missing.
Storing cookies:
from flask import make_response
@app.route('/')
def index():
resp = make_response(render_template(...))
resp.set_cookie('username', 'the username')
return resp
Note
Cookies are set on response objects. Since you normally just return strings from the view functions Flask will convert them into response objects for you. If you explicitly want to do that you can use the make_response() function and then modify it.
If you want to set a cookie at a point where the response object does not exist yet. This is possible by utilizing the Deferred Request Callbacks pattern.
Flask Sessions¶
Session allows you to store information specific to a user from one request to the next. This is implemented on top of cookies for you and signs the cookies cryptographically. What this means is that the user could look at the contents of your cookie but not modify it, unless they know the secret key used for signing.
In order to use sessions you have to set a secret key.
Example¶
from flask import Flask, session, redirect, url_for, escape, request
app = Flask(__name__)
@app.route('/')
def index():
if 'username' in session:
return 'Logged in as %s' % escape(session['username'])
return 'You are not logged in'
@app.route('/login', methods=['GET', 'POST'])
def login():
if request.method == 'POST':
session['username'] = request.form['username']
return redirect(url_for('index'))
return '''
<form action="" method="post">
<p><input type=text name=username>
<p><input type=submit value=Login>
</form>
'''
@app.route('/logout')
def logout():
# remove the username from the session if it's there
session.pop('username', None)
return redirect(url_for('index'))
# set the secret key. keep this really secret:
app.secret_key = 'A0Zr98j/3yX R~XHH!jmN]LWX/,?RT'
Note
The escape() mentioned here does escaping for you if you are not using the template engine (as in this example).
How to generate good secret keys¶
The problem with random is that it’s hard to judge what is truly random. And a secret key should be as random as possible. Your operating system has ways to generate pretty random stuff based on a cryptographic random generator which can be used to get such a key:
>>> import os
>>> os.urandom(24)
'\xfd{H\xe5<\x95\xf9\xe3\x96.5\xd1\x01O<!\xd5\xa2\xa0\x9fR"\xa1\xa8'
Just take that thing and copy/paste it into your code and you’re done.
Note
Flask will take the values you put into the session object and serialize them into a cookie. If you are finding some values do not persist across requests, cookies are indeed enabled, and you are not getting a clear error message, check the size of the cookie in your page responses compared to the size supported by web browsers.
Flask AJAX¶
Flask Uploading Files¶
http://flask.pocoo.org/docs/0.10/patterns/fileuploads/#uploading-files
Example:
from flask import request
from werkzeug import secure_filename
@app.route('/upload', methods=['GET', 'POST'])
def upload_file():
if request.method == 'POST':
f = request.files['the_file']
f.save('/var/www/uploads/' + secure_filename(f.filename))
...
Flask Message Flashing¶
http://flask.pocoo.org/docs/0.10/patterns/flashing/#message-flashing-pattern
Good applications and user interfaces are all about feedback. If the user does not get enough feedback they will probably end up hating the application.
Flask provides a really simple way to give feedback to a user with the flashing system. The flashing system basically makes it possible to record a message at the end of a request and access it (only) on the next request. This is usually combined with a layout template to expose the message.
To flash a message use the flash() method, to get hold of the messages you can use get_flashed_messages() which is also available in the templates.
Flask Logging¶
http://flask.pocoo.org/docs/0.10/errorhandling/
Sometimes you might be in a situation where you deal with data that should be correct, but actually is not. For example you may have some client-side code that sends an HTTP request to the server but it’s obviously malformed. This might be caused by a user tampering with the data, or the client code failing. Most of the time it’s okay to reply with 400 Bad Request in that situation, but sometimes that won’t do and the code has to continue working.
You may still want to log that something fishy happened. This is where loggers come in handy. As of Flask 0.3 a logger is preconfigured for you to use.
Here are some example log calls:
app.logger.debug('A value for debugging')
app.logger.warning('A warning occurred (%d apples)', 42)
app.logger.error('An error occurred')
The attached logger is a standard logging Logger, so head over to the official logging documentation for more information.
Logger handlers¶
There are a couple of handlers provided by the logging system out of the box, but not all of them are useful for basic error logging. The most interesting are probably the following: | | SMTPhandler - logs messages by sending out a email | FileHandler - logs messages to a file on the filesystem. | RotatingFileHandler - logs messages to a file on the filesystem | and will rotate after a certain number of messages. | SysLogHandler - logs messages to a UNIX syslog. |
Example codes for SMTPhandler:
if not app.debug:
import logging
from logging.handlers import SMTPHandler
mail_handler = SMTPHandler('host-ip-or-name',
'from@example.com',
[to-list1@example.com],
'Email-title')
mail_handler.setLevel(logging.ERROR)
app.logger.addHandler(mail_handler)
...
app.logger.error('Some content in the E-mail!')
Note
The from address will not be checked, so it doesn’t need to be a valid address. However, if it’s invalid, it can not be replied.
Better format¶
Email Format:
from logging import Formatter
mail_handler.setFormatter(Formatter('''
Message type: %(levelname)s
Location: %(pathname)s:%(lineno)d
Module: %(module)s
Function: %(funcName)s
Time: %(asctime)s
Message:
%(message)s
'''))
Log file Format:
from logging import Formatter
file_handler.setFormatter(Formatter(
'%(asctime)s %(levelname)s: %(message)s '
'[in %(pathname)s:%(lineno)d]'
))
Flask Sqlite3¶
Using flask-sqlalchemy¶
(venv) $ pip install flask-sqlalchemy
Using Sqlite3 with Flash¶
http://flask.pocoo.org/docs/0.10/patterns/sqlite3/#sqlite3
Python sqlite3 standard library : https://docs.python.org/2/library/sqlite3.html
Module functions and constants:
connect(db-path[, timeout, ...])
Connection Objects:
cursor()
commit()
rollback()
close()
Cursor Objects:
execute(sql[, parameters])
executemany(sql, seq_of_parameters)
executescript(sql_script)
fetchone()
fetchmany([size=cursor.arraysize])
fetchall()
Example:
# -*- coding: utf-8 -*-
import os
import sqlite3
from flask import g
class Sqlite3DB():
app = None
@classmethod
def bindapp(cls, app):
cls.app = app
@classmethod
def connect(cls):
if not hasattr(g, 'sqlite_db'):
assert cls.app is not None, 'Application is not binded to DB'
path = cls.app.config.get('DATABASE')
assert path is not None, 'DB path is not set!'
if not os.path.isfile(path):
cls.app.logger.debug('Initialize the DB')
db = sqlite3.connect(path)
with cls.app.open_resource('../database/schema.sql', mode='r') as f:
db.cursor().executescript(f.read())
db.commit()
else:
cls.app.logger.debug('Connect the DB')
db = sqlite3.connect(path)
db.row_factory = sqlite3.Row
g.sqlite_db = db
return g.sqlite_db
@classmethod
def fetchall(cls, sql):
cls.app.logger.debug('Fetch all entries in the DB')
db = cls.connect()
cur = db.cursor().execute(sql)
return cur.fetchall()
@classmethod
def addone(cls, sql, data=None):
db = cls.connect()
db.cursor().execute(sql, data)
db.commit()
@classmethod
def close(cls):
db = cls.connect()
db.close()
Flask Unit Test¶
Flask Extension¶
Flask extensions are listed on the Flask Extension Registry and can be downloaded with easy_install or pip. If you add a Flask extension as dependency to your requirements.rst or setup.py file they are usually installed with a simple command or when your application installs.
Using Extensions¶
Extensions typically have documentation that goes along that shows how to use it. There are no general rules in how extensions are supposed to behave but they are imported from common locations.
If you have an extension called Flask-Foo or Foo-Flask it will be always importable from flask.ext.foo:
from flask.ext import foo
Popular Extensions¶
http://www.phperz.com/article/15/0901/153299.html
How to install extensions: sudo pip install extension-name.
flask-script¶
http://flask-script.readthedocs.io/en/latest/
Start server with some options in command line, such as host/port/debug.
Examples:
# "from flask_script import Manager" for python3
from flask.ext.script import Manager
from project.main import app
manager = Manager(app)
manager.run()
flask-bootstrap¶
Please refer to the examples here: http://getbootstrap.com
Please extends the bootstrap/base.html, and fill your blocks.
After using flask-bootstrap, you can use bootstrap/wtf.html for extending wtf form automatically.
For example:
{% import "bootstrap/wtf.html" as wtf %}
{% block page_content %}
<div class="page-header">
<h2>登录</h2>
</div>
<div class="col-md-4">
{{ wtf.quick_form(form) }}
...
</div>
{% endblock %}
flask-sqlalchemy¶
http://flask-sqlalchemy.pocoo.org/2.1/
Manage database such as MySQL, Postgres, SQLite
Please see also: http://pythonhosted.org/Flask-MongoAlchemy/
Example:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from datetime import datetime
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:////tmp/test.db'
db = SQLAlchemy(app)
db.create_all()
class Post(db.Model):
id = db.Column(db.Integer, primary_key=True)
title = db.Column(db.String(80))
body = db.Column(db.Text)
pub_date = db.Column(db.DateTime)
category_id = db.Column(db.Integer, db.ForeignKey('category.id'))
category = db.relationship('Category',
backref=db.backref('posts', lazy='dynamic'))
def __init__(self, title, body, category, pub_date=None):
self.title = title
self.body = body
if pub_date is None:
pub_date = datetime.utcnow()
self.pub_date = pub_date
self.category = category
def __repr__(self):
return '<Post %r>' % self.title
class Category(db.Model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(50))
def __init__(self, name):
self.name = name
def __repr__(self):
return '<Category %r>' % self.name
py = Category('Python')
p = Post('Hello Python!', 'Python is pretty cool', py)
db.session.add(py)
db.session.add(p)
db.session.commit()
flask-login¶
https://flask-login.readthedocs.io/en/latest/
Login authentication management, login information can be saved in cookie in browser’s access session.
Example:
from flask_login import LoginManager
login_manager = LoginManager()
login_manager.session_protection = 'Strong'
login_manager.login_view = 'auth.login'
login_manager.init_app(app)
#We must provide user_loader callback, too.
@login_manager.user_loader
def load_user(user_id):
return User.get(user_id)
#Where the User is a special class which must provide four properties or methods.
# - is_authenticated
# - is_active
# - is_anonymous
# - get_id()
#To make implementing a user class easier, you can inherit from UserMixin,
#which provides default implementations for all of these properties or methods.
class User(UserMixin, db.Model):
pass
from flask_login import login_required
@main.route('/')
@login_required
def index():
return render_template('main/index.html')
flash-wtf¶
For create form based on Python language APIs.
flask-httpauth¶
http://flask-httpauth.readthedocs.io/en/latest/
HTTP authentication for RESTful API via token.
Basic Password Example:
from flask_httpauth import HTTPBasicAuth
from flask_restful import Api
http_auth = HTTPBasicAuth()
api = Api(prefix='/api/v1.0', decorators=[http_auth.login_required])
@http_auth.verify_password
def verify_pw(username, password):
user = User.query.filter_by(username=username).first()
if not user or not user.confirmed:
return False
return user.verify_password(password)
@http_auth.error_handler
def auth_error():
return make_response(401, {'error': 'unauthorized'})
Token Authentication Example:
# app/models.py
from itsdangerous import TimedJSONWebSignatureSerializer as Serializer
from flask import current_app
class User(UserMixin, db.modle):
...
def generate_auth_token(self, expiration):
s = Serializer(current_app.config['SECRET_KEY'],
expires_in=expiration)
return s.dumps({'id': self.id}).decode('ascii')
@staticmethod
def verify_auth_token(token):
s = Serializer(current_app.config['SECRET_KEY'])
try:
data = s.loads(token)
except:
return None
return User.query.get(data['id'])
# app/api/authentication.py
from flask import g
from ..extensions import http_auth
from ..database import User
from ..utils import make_response
from ..auth.views import auth
@http_auth.verify_password
def verify_pw(username_or_token, password):
if username_or_token == '':
return False
if password == '':
g.current_user = User.verify_auth_token(username_or_token)
g.token_used = True
return g.current_user is not None
user = User.query.filter_by(username=username_or_token).first()
if not user or not user.confirmed:
return False
g.current_user = user
g.token_used = False
return user.verify_password(password)
# We can define a '/token' routing in app/blueprint or Api instance.
@auth.route('/token')
@http_auth.login_required
def get_token():
if g.current_user.is_anonymous or g.token_used:
return make_response(401, {'error': 'unauthorized'})
return make_response(200, {
'token': g.current_user.generate_auth_token(expiration=3600),
'expiration': 3600})
# For define /token in Api
class apiToken(Resource):
def get(self):
if g.current_user.is_anonymous or g.token_used:
return make_response(401, {'error': 'unauthorized'})
return make_response(200, {
'token': g.current_user.generate_auth_token(expiration=3600),
'expiration': 3600})
api.add_resource(apiToken, '/token')
Need to get token firstly like this:
curl -s -u admin:admin -H "Content-Type: application/json" \
http://127.0.0.1:5000/token
And then you can use token for authentication now:
curl -s -u <TOKEN>: -H "Content-Type: application/json" \
http://127.0.0.1:5000/api/v1.0/xxx
flask-mail¶
For sending e-mail in flask. Please note that it’s not flask-email
Example:
from flask import Flask
from flask_mail import Mail, Message
import os
app = Flask(__name__)
app.config.update(
DEBUG = True,
MAIL_SERVER='smtp.live.com',
MAIL_PROT=25,
MAIL_USE_TLS = True,
MAIL_USE_SSL = False,
MAIL_USERNAME = 'username@hotmail.com',
MAIL_PASSWORD = 'password',
MAIL_DEBUG = True
)
mail = Mail(app)
subject = 'hello'
msg = Message(subject,
sender='example@example.com',
recipients=['example@example.com'])
msg.body = "This is the content of test email"
with app.open_resource("./test.jpg") as f:
msg.attach("image.jpg", "image/jpg", f.read())
mail.send(msg)
flask-restful¶
Flask-RESTful is an extension for Flask that adds support for quickly building REST APIs.
Example:
from flask import Flask, request
from flask.ext.restful import Resource, Api
app = Flask(__name__)
api = Api(app)
todos = {}
class TodoSimple(Resource):
def get(self, todo_id):
return {todo_id: todos[todo_id]}
def put(self, todo_id):
todos[todo_id] = request.form['data']
return {todo_id: todos[todo_id]}
api.add_resource(TodoSimple, '/<string:todo_id>')
if __name__ == '__main__':
app.run(debug=True)
Flask-SSLify¶
https://github.com/kennethreitz/flask-sslify
This is a simple Flask extension that configures your Flask application to redirect all incoming requests to HTTPS(only when app.debug is False).
install: pip install Flask-SSLify
See also: http://flask.pocoo.org/snippets/111/
- Example::
from flask import Flask from flask_sslify import SSLify
app = Flask(__name__) sslify = SSLify(app)
context = (‘ssl_keys/ca.crt’, ‘ssl_keys/ca.key’) app.run(ssl_context=context)
Create the certificate and key:
openssl genrsa -out ca.key 2048
openssl req -new -x509 -days 36500 -key ca.key -out ca.crt -subj \
"/C=CN/ST=Beijing/L=Beijing/O=MyOrganization/OU=Hoxm"
openssl genrsa -out server.key 2048
openssl req -new -key server.key -out server.csr -subj \
"/C=CN/ST=Beijing/L=Beijing/O=MyOrganization/OU=Hoxm/CN=localhost"
openssl x509 -req -in server.csr -out server.crt \
-CA ca.crt -CAkey ca.key -CAcreateserial -days 3650
This is for ssl in flask directly. If use nginx for proxy, we can use https in nginx but http at the background.
Fixme: url_for for ssl: http://segmentfault.com/q/1010000000167396
flask-debugtoolbar¶
Example:
from flask import Flask
from flask_debugtoolbar import DebugToolbarExtension
app = Flask(__name__)
# the toolbar is only enabled in debug mode:
app.debug = True
# set a 'SECRET_KEY' to enable the Flask session cookies
app.config['SECRET_KEY'] = '<replace with a secret key>'
toolbar = DebugToolbarExtension()
toolbar.init_app(app)
flask-moment¶
To handle the local date/time issue in flask.
flask-babel¶
http://pythonhosted.org/Flask-Babel/
Flask-Babel is an extension to Flask that adds i18n and l10n support to any Flask application. It has builtin support for date formatting with timezone support as well as a very simple and friendly interface to gettext translations.
flask-cache¶
http://pythonhosted.org/Flask-Cache/
Cache the result of view or non-view related functions
flask-assets¶
http://flask-assets.readthedocs.io/en/latest/
Flask-Assets helps you to integrate webassets, which is to merge and compress JavaScript and CSS files.
falsk-testing¶
http://pythonhosted.org/Flask-Testing/
The Flask-Testing extension provides unit testing utilities for Flask.
flask-themes¶
http://pythonhosted.org/Flask-Themes/
Flask-Themes makes it easy for your application to support a wide range of appearances.
flask-user¶
http://pythonhosted.org/Flask-User/
Flask-User offers the following(and more) features out-of-the-box:
- Registrations and Email Confirmations
- Change Usernames, Change Passwords, and Forgotten Passwords
- Role-based Authorization
- Remember-me cookies
- Multiple emails per user
- Internationalization
It uses the following amazing offerings:
- Flask-Babel
- Flask-Login
- Flask-Mail
- SQLAlchemy and Flask-SQLAlchemy
- WTForms and Flask-WTF
flask-security¶
Similar to flask-user, just integrate some security things together for you:
- Session based authentication
- Role management
- Password encryption
- Basic HTTP authentication
- Token based authentication
- Token based account activation (optional)
- Token based password recovery / resetting (optional)
- User registration (optional)
- Login tracking (optional)
- JSON/Ajax Support
By integrating various Flask extensions and libraries:
- Flask-Login
- Flask-Mail
- Flask-Principal
- Flask-Script
- Flask-WTF
- itsdangerous
- passlib
Flask-Security supports the following Flask extensions out of the box for data persistence:
- Flask-SQLAlchemy
- Flask-MongoEngine
- Flask-Peewee
frozen-flask¶
http://pythonhosted.org/Frozen-Flask/
Frozen-Flask freezes a Flask application into a set of static files. The result can be hosted without any server-side software other than a traditional web server.
Flask Project¶
mega_tutorial¶
EN: http://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-i-hello-world CN: http://www.pythondoc.com/flask-mega-tutorial/index.html
Typical Strucature¶
The typical project directory tree:
.
├── app
│ ├── hello.py
│ ├── static
│ └── templates
├── config.py
├── requirements.txt
├── run.py
├── tests
│ └── test_basic.py
└── venv
Examples:
Project Template¶
https://github.com/sloria/cookiecutter-flask
Install cookiecutter:
$ sudo pip install cookiecutter
$ cookiecutter https://github.com/sloria/cookiecutter-flask.git
$ sudo apt-get install libpq-dev python-dev
$ sudo pip install -r requirements/dev.txt (or do this in virtualenv)
Configuration Handling¶
http://flask.pocoo.org/docs/0.10/config/
The config attribute of the Flask object is the place where Flask itself puts certain configuration values and also where extensions can put their configuration values. This is also where you can have your own configuration.
The config is a subclass of a dictionary and can be modified just like any dictionary:
app = Flask(__name__)
app.config['DEBUG'] = True
Certain configuration values are also forwarded to the Flask object so you can read and write them from there:
app.debug = True
To update multiple keys at once you can use the dict.update() method:
app.config.update(
DEBUG=True,
SECRET_KEY='...'
)
Note
The SERVER_NAME key is used for the subdomain support. Because Flask cannot guess the subdomain part without the knowledge of the actual server name, this is required if you want to work with subdomains. This is also used for the session cookie.
Please keep in mind that not only Flask has the problem of not knowing what subdomains are, your web browser does as well. Most modern web browsers will not allow cross-subdomain cookies to be set on a server name without dots in it. So if your server name is ‘localhost’ you will not be able to set a cookie for ‘localhost’ and every subdomain of it. Please chose a different server name in that case, like ‘myapplication.local’ and add this name + the subdomains you want to use into your host config or setup a local bind.
Configuring from Files¶
Configuration becomes more useful if you can store it in a separate file, ideally located outside the actual application package. This makes packaging and distributing your application possible via various package handling tools (Deploying with Distribute) and finally modifying the configuration file afterwards.
So a common pattern is this:
app = Flask(__name__)
app.config.from_object('yourapplication.default_settings')
app.config.from_envvar('YOURAPPLICATION_SETTINGS')
This first loads the configuration from the yourapplication.default_settings module and then overrides the values with the contents of the file the YOURAPPLICATION_SETTINGS environment variable points to. This environment variable can be set on Linux or OS X with the export command in the shell before starting the server:
$ export YOURAPPLICATION_SETTINGS=/path/to/settings.cfg
$ python run-app.py
* Running on http://127.0.0.1:5000/
* Restarting with reloader...
The configuration files themselves are actual Python files. Only values in uppercase are actually stored in the config object later on. So make sure to use uppercase letters for your config keys.
Here is an example of a configuration file:
# Example configuration
DEBUG = False
SECRET_KEY = '?\xbf,\xb4\x8d\xa3"<\x9c\xb0@\x0f5\xab,w\xee\x8d$0\x13\x8b83'
Make sure to load the configuration very early on, so that extensions have the ability to access the configuration when starting up. There are other methods on the config object as well to load from individual files. For a complete reference, read the Config object’s documentation.
Other Examples:
app.config.from_pyfile('yourconfig.cfg')
DEBUG = True
SECRET_KEY = 'development key'
app.config.from_object(__name__)
Configuration Best Practices¶
The downside with the approach mentioned earlier is that it makes testing a little harder. There is no single 100% solution for this problem in general, but there are a couple of things you can keep in mind to improve that experience:
- create your application in a function and register blueprints on it. That way you can create multiple instances of your application with different configurations attached which makes unittesting a lot easier. You can use this to pass in configuration as needed.
- Do not write code that needs the configuration at import time. If you limit yourself to request-only accesses to the configuration you can reconfigure the object later on as needed.
Development / Production¶
Most applications need more than one configuration. There should be at least separate configurations for the production server and the one used during development. The easiest way to handle this is to use a default configuration that is always loaded and part of the version control, and a separate configuration that overrides the values as necessary as mentioned in the example above:
app = Flask(__name__)
app.config.from_object('yourapplication.default_settings')
app.config.from_envvar('YOURAPPLICATION_SETTINGS')
Then you just have to add a separate config.py file and export YOURAPPLICATION_SETTINGS=/path/to/config.py and you are done. However there are alternative ways as well. For example you could use imports or subclassing.
An interesting pattern is to use classes and inheritance for configuration:
class Config(object):
DEBUG = False
TESTING = False
DATABASE_URI = 'sqlite://:memory:'
class ProductionConfig(Config):
DATABASE_URI = 'mysql://user@localhost/foo'
class DevelopmentConfig(Config):
DEBUG = True
class TestingConfig(Config):
TESTING = True
To enable such a config you just have to call into from_object():
app.config.from_object('configmodule.ProductionConfig')
There are many different ways and it’s up to you how you want to manage your configuration files. However here a list of good recommendations:
- keep a default configuration in version control repository. Either populate the config with this default configuration or import it in your own configuration files before overriding values.
- use an environment variable to switch between the configurations. You can quickly and easily switch between different configs without having to touch the code at all. If you are working often on different projects you can even create your own script for sourcing that activates a virtualenv and exports the development configuration for you.
Instance Folders¶
With Flask 0.8 a new attribute was introduced: Flask.instance_path. It refers to a new concept called the “instance folder”. The instance folder is designed to not be under version control and be deployment specific. It’s the perfect place to drop things that either change at runtime or configuration files.
You can either explicitly provide the path of the instance folder when creating the Flask application or you can let Flask autodetect the instance folder. For explicit configuration use the instance_path parameter:
app = Flask(__name__, instance_path='/path/to/instance/folder')
Note
Please keep in mind that this path must be absolute when provided.
If the instance_path parameter is not provided the following default locations are used:
Uninstalled module:
/myapp.py
/instance
Uninstalled package:
/myapp
/__init__.py
/instance
Installed module or package:
$PREFIX/lib/python2.X/site-packages/myapp
$PREFIX/var/myapp-instance
Note
$PREFIX is the prefix of your Python installation. This can be /usr or the path to your virtualenv. You can print the value of sys.prefix to see what the prefix is set to.
Since the config object provided loading of configuration files from relative filenames we made it possible to change the loading via filenames to be relative to the instance path if wanted. The behavior of relative paths in config files can be flipped between “relative to the application root” (the default) to “relative to instance folder” via the instance_relative_config switch to the application constructor:
app = Flask(__name__, instance_relative_config=True)
Here is a full example of how to configure Flask to preload the config from a module and then override the config from a file in the config folder:
app = Flask(__name__, instance_relative_config=True)
app.config.from_object('yourapplication.default_settings')
app.config.from_pyfile('application.cfg', silent=True)
The path to the instance folder can be found via the Flask.instance_path. Flask also provides a shortcut to open a file from the instance folder with Flask.open_instance_resource().
Example usage for both:
filename = os.path.join(app.instance_path, 'application.cfg')
with open(filename) as f:
config = f.read()
# or via open_instance_resource:
with app.open_instance_resource('application.cfg') as f:
config = f.read()
Flask RESTful API¶
Definition: hypermedia for hypertext-driven(HATEOAS) + POST/GET/PUT/DELETE for CRUD
HATEOAS: hypermedia as the engine of application state
Flask Addon Project¶
JavaScript Basic¶
http://www.w3school.com.cn/b.asp http://www.w3school.com.cn/js/index_pro.asp https://developer.mozilla.org/en-US/docs/Web/JavaScript
JavaScript Overview¶
- JavaScript programs are written using the Unicode character set.
- JavaScript is a case-sensitive language. (However, HTML is not case-sensitive.)
- JavaScript ignores spaces that appear between tokens in programs and ignores line breaks.
- JavaScript supports two styles of comments, the same as C language.
- JavaScript uses the semicolon ( ; ) to separate statements (see Chapter 5) from each other.
- JavaScript identifier names must match “^[a-zA-z_$][a-zA-z0-9_$]*”
- JavaScript interpreter performs automatic garbage collection for memory management.
- JavaScript is an object-oriented language.
- JavaScript variables are untyped, you can assign a value of any type to a variable.
- All numbers in JavaScript are represented as floating-point values.
- JavaScript numbers can approximate 0.1 very closely, but cannot be represented exactly.
- JavaScript code behaves as if all variable declarations in a function (but not any associated assignments) are “hoisted” to the top of the function.
JavaScript Reserved Words¶
JavaScript reserves a number of identifiers as the keywords of the language itself. You cannot use these words as identifiers in your programs:
break case catch continue debugger default
delete do else false finally for
function if in instanceof new null
return switch this throw true try
typeof var void while with
JavaScript also reserves certain keywords that are not currently used by the language but which might be used in future versions. ECMAScript 5 reserves the following words:
class const enum export extends import super
The following words are legal in ordinary JavaScript code, but reserved in strict mode:
implements interface let package private protected
public static yield
Strict mode also imposes restrictions on the use of the following identifiers. They are not fully reserved, but they are not allowed as variable, function, or parameter names:
arguments eval
ECMAScript 3 reserved all the keywords of the Java language, and although this has been relaxed in ECMAScript 5, you should still avoid all of these identifiers if you plan to run your code under an ECMAScript 3 implementation of JavaScript:
abstract boolean byte char class const
double enum export extends final float
goto implements import int interface long
native package private protected public short
static super synchronized throws transient volatile
JavaScript predefines a number of global variables and functions, and you should avoid using their names for your own variables and functions:
arguments Array Boolean Date
decodeURI decodeURIComponent encodeURI encodeURIComponent
Error eval EvalError Function
Infinity isFinite isNaN JSON
Math NaN Number Object
parseFloat parseInt RangeError ReferenceError
RegExp String SyntaxError TypeError
undefined URIError
JavaScript Types and Variables¶
Examples:
const pi = 3.1415926; // const value cannot be changed
var a = 'test';
var b = 'test';
a == b ; // true, Primitive type only compare value
var a = {x: 1};
var a = {x: 1};
a == b // false
c = a;
c.x = 2;
a.x; // => 2, object is reference type
a === c; // true
// 'let' defines block level values in loop or function.
for (let i=1; i<len; i++) { ...}
[x, y] = [1, 2] // x=1, y=2
[x, y] = [y, x] // switch x and y
[a, b, c] = [1, 2] // a=1, b=2, c=undefined
[a, [b, c]] = [1, [2,3], 4] // a=1, b=2, c=3
{r: red, g:green, b:blue} = {r:0, g:255, b:0}; // red=0, green=255, blue=0
var data=[1, 2, -3];
var squares = [x*x for each (x in data) if (x>=0)] // squares = [1, 4]
Primitive/Object Type¶
Primitive Type:
- number
- string
- boolean
- null
- undefined
Object Type:
- object/class
- array
- function
- date
- regexp
- error
Mutable/Immutable Type¶
Mutable Type:
- object
- array
Immutable Type:
- number
- string
- boolean
- null
- undefined
Number¶
0 === -0 # true
(0.3 - 0.2) == (0.2 -0.1) # False
Math.pow(2,53) // => 9007199254740992: 2 to the power 53
Math.round(.6) // => 1.0: round to the nearest integer
Math.ceil(.6) // => 1.0: round up to an integer
Math.floor(.6) // => 0.0: round down to an integer
Math.abs(-5) // => 5: absolute value
Math.max(x,y,z) // Return the largest argument
Math.min(x,y,z) // Return the smallest argument
Math.random() // Pseudo-random number x where 0 <= x < 1.0
Math.PI // π: circumference of a circle / diameter
Math.E // e: The base of the natural logarithm
Math.sqrt(3) // The square root of 3
Math.pow(3, 1/3) // The cube root of 3
Math.sin(0) // Trigonometry: also Math.cos, Math.atan, etc.
Math.log(10) // Natural logarithm of 10
Math.log(100)/Math.LN10 // Base 10 logarithm of 100
Math.log(512)/Math.LN2 // Base 2 logarithm of 512
Math.exp(3) // Math.E cubed
Date¶
var then = new Date(2010, 0, 1); // The 1st day of the 1st month of 2010
var later = new Date(2010, 0, 1, 17, 10, 30);
var now = new Date(); // The current date and time
var elapsed = now - then; // Date subtraction: interval in milliseconds
later.getFullYear() // => 2010
later.getMonth() // => 0: zero-based months
later.getDate() // => 1: one-based days
later.getDay() // => 5: day of week. 0 is Sunday 5 is Friday.
later.getHours() // => 17: 5pm, local time
later.getUTCHours() // -> hours in UTC time; depends on timezone
later.toString() // => "Fri Jan 01 2010 17:10:30 GMT-0800 (PST)"
later.toUTCString() // => "Sat, 02 Jan 2010 01:10:30 GMT"
later.toLocaleDateString() // => "01/01/2010"
later.toLocaleTimeString() // => "05:10:30 PM"
later.toISOString() // => "2010-01-02T01:10:30.000Z"; ES5 only
String¶
http://www.w3school.com.cn/jsref/jsref_obj_string.asp
In ECMAScript 3, string literals must be written on a single line. In ECMAScript 5, however, you can break a string literal across multiple lines by ending each line but the last with a backslash ( ). In ECMAScript 5, strings can be treated like read-only arrays, and you can access individual characters from a string using square brackets.
Basic Functions:
var s = "Hello, " + "world";
s.charAt(0) // => "h": the first character.
s.charAt(s.length-1) // => "d": the last character.
s.substring(1,4) // => "ell": the 2nd, 3rd and 4th
s.slice(1,4) // => "ell": same thing
s.slice(-3) // => "rld": last 3 characters
s.indexOf("l") // => 2: position of first letter
s.lastIndexOf("l") // => 10: position of last letter
s.indexOf("l", 3) // => 3: position of first "l" at or after 3
s.split(", ") // => ["hello", "world"] split into substrings
s.replace("h", "H") // => "Hello, world": replaces all instances
s.toUpperCase() // => "HELLO, WORLD"
s[0] // => "h"
s[s.length-1] // => "d"
s.search(/world/); // => 7, first position
s.match(/l+/g) // => ["ll", "l"], 'g' means return all matches
Advance Usage:
"1, 2, 3, 4, 5".split(/\s*,\s*/); // Returns ["1","2","3","4","5"]
var s = 'My name is "John"' ; // $1 refer to the first matches
s.replace(/"([^'"]*)"/g, "'$1'") // => My name is 'John'
"1 plus 2 equals 3".match(/\d+/g) // returns ["1", "2", "3"]
"1 plus 2 equals 3".match(/\d+/) // returns ["1"]
"1 plus 2 equals 3".match(/(\d+)/g) // returns ["1", "2", "3"]
"1 plus 2 equals 3".match(/(\d+)/) // returns ["1", "1"], the first matches and content in "()" in turn
var url_pattern = /(\w+):\/\/([\w.]+)\/(\S*)/;
var text = "Visit my blog at http://www.example.com/~david";
var result = text.match(url_pattern);
if (result != null) {
var fullurl = result[0]; // Contains "http://www.example.com/~david"
var protocol = result[1]; // Contains "http"
var host = result[2]; // Contains "www.example.com"
var path = result[3]; // Contains "~david"
}
Boolean¶
All other values, including all objects (and arrays) convert to, and work like, true. Except the following false values:
undefined null 0 -0 NaN ""
null == undefined // true
0 == false // true
"0" == 0 // true
Note
null is a language keyword and undefined is a predefined global variable. You might consider undefined to represent a system-level, unexpected, or error-like absence of value and null to represent program-level, normal, or expected absence of value.
Type Conversions¶
10 + " objects" // "10 objects"
"7" * "4" // 28
1 - "" // 1
1 - "x" // NaN
Number("3") // 3
String(false) // "false"
Boolean([]) // true
Object(3) // new Number(3)
x + "" // equal to String(x)
+x // qeual to Number(x)
!!x // equal to Boolean(x)
var n = 17;
n.toString(2) // "10001"
n.toString(16) // "11"
var n = 123456.789
n.toFixed(0) // 123457
n.toFixed(2) // 123456.79
n.toFixed(5) // 123456.78900
n.toExponential(1) // "1.2e+5"
n.toPrecision(4) // "1.235e+5"
parseInt("3 blind mice") //3
parseFloat(" 3.14 meters") // 3.14
parseInt("0xFF") //255
parseInt(".1") // NaN
parseInt("0.1") // 0
parseFloat(".1") // 0.1
parseFloat("$12.34") // NaN
parseInt("11", 2) // 3
parseInt("11", 16) // 17
[1,2,3].toString() // "1,2,3"
Special Global Number Variables¶
Infinity
- Number.POSITITVE_INFINITY
- 1/0
- Number.MAX_VALUE + 1
-Infinity
- Number.NEGATIVE_INFINITY
- -1/0
- -Number.MAX_VALUE - 1
NaN
- 0/0
Note
JavaScript Expression¶
Operator Overview¶
The following operators re arranged in order from high precedence to low precedence, with horizontal lines separating groups of operators at the same precedence level:
============ ============================ ==== ==== ==================
Operator Operation A N Types
============ ============================ ==== ==== ==================
++ Pre- or post-increment R 1 lval -> num
-- Pre- or post-decrement R 1 lval -> num
- Negate number R 1 num -> num
+ Convert to number R 1 num -> num
~ Invert bits R 1 int -> int
! Invert boolean value R 1 bool -> bool
delete Remove a property R 1 lval -> bool
typeof Determine type of operand R 1 any -> str
void Return undefined value R 1 any -> undef
----------------------------------------------------------------------
* / % Multiply, divide, remainder L 2 num,num -> num
----------------------------------------------------------------------
+ - Add, subtract L 2 num,num -> num
+ Concatenate strings L 2 str,str -> str
----------------------------------------------------------------------
<< Shift left L 2 int,int -> int
>> Shift right with sign L 2 int,int -> int
>>> Shift right with zero L 2 int,int -> int
----------------------------------------------------------------------
< <= > >= Compare in numeric order L 2 num,num -> bool
< <= > >= Compare in alphabetic order L 2 str,str -> bool
instanceof Test object class L 2 obj,func-> bool
in Test whether property exists L 2 str,obj -> bool
----------------------------------------------------------------------
== Test for equality L 2 any,any -> bool
!= Test for inequality L 2 any,any -> bool
=== Test for strict equality L 2 any,any -> bool
!== Test for strict inequality L 2 any,any -> bool
----------------------------------------------------------------------
& Compute bitwise AND L 2 int,int -> int
----------------------------------------------------------------------
^ Compute bitwise XOR L 2 int,int -> int
----------------------------------------------------------------------
| Compute bitwise OR L 2 int,int -> int
----------------------------------------------------------------------
&& Compute logical AND L 2 any,any -> any
----------------------------------------------------------------------
|| Compute logical OR L 2 any,any -> any
----------------------------------------------------------------------
?: Choose 2nd or 3rd operand R 3 bool,any,any->any
----------------------------------------------------------------------
= Variable or property assign R 2 lval,any-> any
*= /= %= += Operate and assign R 2 lval,any-> any
-= &= ^= |= Operate and assign R 2 lval,any-> any
<<= >>= >>>= Operate and assign R 2 lval,any-> any
----------------------------------------------------------------------
, Discard 1st, return 2nd L 2 any,any -> any
============ ============================ ==== ==== ==================
Note
=== and ==¶
The strict equality operator === evaluates its operands, and then compares the two values as follows, performing no type conversion:
- If the two values have different types, they are not equal.
- If both values are null or both values are undefined , they are equal.
- If both values are the boolean value(true or false), they are equal.
- If one or even both values is NaN , they are not equal.
- If both values are numbers and have the same value, they are equal.
- If one value is 0 and the other is -0 , they are also equal.
- If both values are strings and have same content, they are equal.
- If both values refer to the same object, array, or function, they are equal.
- If they refer to different object instances, they are not equal.
The equality operator == is like the strict equality operator, but it is less strict:
If the two values have the same type, test them for strict equality as described above. If they are strictly equal, they are equal. If they are not strictly equal, they are not equal.
If the two values do not have the same type, the == operator attempts some type conversions and tries the comparison again:
- If one value is null and the other is undefined , they are equal.
- If one value is a number and the other is a string, convert the string to a number and try the comparison again, using the converted value.
- If either value is true , convert it to 1 and try the comparison again.
- If either value is false , convert it to 0 and try the comparison again.
- If one value is an object and the other is a number or string, convert the object to a primitive value and try the comparison again. (toString/valueOf)
- Any other combinations of values are not equal.
typeof¶
======================= ===========
x typeof x
======================= ===========
undefined "undefined"
null "object"
true or false "boolean"
any number or NaN "number"
any string "string"
any function "function"
any native object "object"
======================= ===========
JavaScript Statements¶
Overview¶
JavaScript does not have block scope and variables declared within a statement block are not private to the block.
Variables are defined throughout the script or function in which they are declared. Their declaration is “hoisted” up to the start of the script or function. Initialization, however, occurs at the location of the var statement, and the value of the variable is undefined before that point in the code.
Function declaration statements may appear in top-level JavaScript code, or they may be nested within other functions. When nested, however, function declarations may only appear at the top level of the function they are nested within. That is, function definitions may not appear within if statements, while loops, or any other statements.
With var, only the variable declaration is hoisted, the variable initialization code remains where you placed it. With function declaration statements, however, both the function name and the function body are hoisted: all functions in a script or all nested functions in a function are declared before any other code is run.
Like the var statement, function declaration statements create variables that cannot be deleted.
Finally, note that a break statement, with or without a label, can not transfer control across function boundaries. You cannot label a function definition statement, for ex- ample, and then use that label inside the function.
Statements Structure¶
statements:
// Initialize an array a
for(i = 0; i < a.length; a[i++] = 0) /* empty */ ;
var f = function(x) { return x+1; } // Expression assigned to a variable
function f(x) { return x+1; } // Function declaration statement
if (expression) {
statements
}
else if (expression) {
statements
}
else {
statements
}
// compare by "==="
switch(expression) {
case value1:
statements
break;
case value2:
statements
break;
default:
statements
break;
}
while (expression) {
statements
}
do {
statements
} while (expression) ; // Must always be terminated with a semicolon.
for(initialize; test; increment) {
statements
}
for (var p in o) { // Enumerable properties in object
statements
}
var o = {x:1, y:2, z:3}, a = [], i = 0;
for (a[i++] in o) { // Store properties of object into arrray
statements
}
lablename: loop-statements {
loop-statement {
...
continue lablename;
...
break lablename;
}
}
throw new Error("x must not be negative");
try {
// Normally, this code runs from the top of the block to the bottom
// without problems. But it can sometimes throw an exception,
// either directly, with a throw statement, or indirectly, by calling
// a method that throws an exception.
}
catch (e) {
// The statements in this block are executed if, and only if, the try
// block throws an exception. These statements can use the local variable
// e to refer to the Error object or other value that was thrown.
// This block may handle the exception somehow, may ignore the
// exception by doing nothing, or may rethrow the exception with throw.
}
finally {
// This block contains statements that are always executed, regardless of
// what happens in the try block. They are executed whether the try
// block terminates:
// 1) normally, after reaching the bottom of the block
// 2) because of a break, continue, or return statement
// 3) with an exception that is handled by a catch clause above
// 4) with an uncaught exception that is still propagating
}
if (o === undefined) debugger;
examples:
var o = {one: 1, two: 2, three: 3};
for (let p in o) console.log(p); // indexes: one, two, three
for each (let v in o) console.log(v) // values: 1, 2, 3
try {
// more than one type of exception where
throw 1;
}
catch(e if e instanceof ReferenceError) {
//do something here
}
catch(e if e === "quit") {
// do something here
}
catch(e){ //all other cases
// do something here
}
finally {
// do something here
}
‘use strict’ Directive¶
It can appear only at the start of a script or at the start of a function body, before any real statements have appeared. It need not be the very first thing in the script or function.
The differences between strict mode and non-strict mode are the following:
- The with statement is not allowed in strict mode.
- In strict mode, all variables must be declared: a ReferenceError is thrown if you assign a value to an identifier that is not a declared variable, function, function parameter, catch clause parameter, or property of the global object. (In non-strict mode, this implicitly declares a global variable by adding a new property to the global object.)
- In strict mode, functions invoked as functions (rather than as methods) have a
this value of undefined . (In non-strict mode, functions invoked as functions are
always passed the global object as their this value.) This difference can be used to
determine whether an implementation supports strict mode:
var hasStrictMode = (function() { "use strict"; return this===undefined}()); - Also, in strict mode, when a function is invoked with call() or apply() , the this value is exactly the value passed as the first argument to call() or apply() . (In nonstrict mode, null and undefined values are replaced with the global object and non-object values are converted to objects.)
- In strict mode, assignments to nonwritable properties and attempts to create new properties on nonextensible objects throw a TypeError. (In non-strict mode, these attempts fail silently.)
- In strict mode, code passed to eval() cannot declare variables or define functions in the caller’s scope as it can in non-strict mode. Instead, variable and function definitions live in a new scope created for the eval() . This scope is discarded when the eval() returns.
- In strict mode, the arguments object (§8.3.2) in a function holds a static copy of the values passed to the function. In non-strict mode, the arguments object has “magical” behavior in which elements of the array and named function parameters both refer to the same value.
- In strict mode, a SyntaxError is thrown if the delete operator is followed by an unqualified identifier such as a variable, function, or function parameter. (In non- strict mode, such a delete expression does nothing and evaluates to false .)
- In strict mode, an attempt to delete a nonconfigurable property throws a TypeError. (In non-strict mode, the attempt fails and the delete expression eval- uates to false .)
- In strict mode, it is a syntax error for an object literal to define two or more prop- erties by the same name. (In non-strict mode, no error occurs.)
- In strict mode, it is a syntax error for a function declaration to have two or more parameters with the same name. (In non-strict mode, no error occurs.)
- In strict mode, octal integer literals (beginning with a 0 that is not followed by an 0x) are not allowed. (In non-strict mode, some implementations allow octal literals.)
- In strict mode, the identifiers eval and arguments are treated like keywords, and you are not allowed to change their value. You cannot assign a value to these identifiers, declare them as variables, use them as function names, use them as function parameter names, or use them as the identifier of a catch block.
- In strict mode, the ability to examine the call stack is restricted. arguments.caller and arguments.callee both throw a TypeError within a strict mode function. Strict mode functions also have caller and arguments properties that throw TypeError when read. (Some implementations define these nonstandard properties on non-strict functions.)
JavaScript Objects¶
JavaScript’s fundamental datatype is the object. An object is an unordered collection of properties, each of which has a name and a value.
Overview¶
JavaScript objects are dynamic(properties can usually be added and deleted), but they can be used to simulate the static objects and “structs” of statically typed languages.
The most common things to do with objects are create them and to set, query, delete, test, and enumerate their properties.
In addition to its properties, every object has three associated object attributes: - An object’s prototype is a reference to another object from which properties are inherited. - An object’s class is a string that categorizes the type of an object. - An object’s extensible flag specifies whether new properties may be added to the object.
Finally, here are some terms we’ll use to distinguish among three broad categories of JavaScript objects and two types of properties:
Types of objects:
- A native object is an object or class of objects defined by the ECMAScript specification. Arrays, functions, dates, and regular expressions (for example) are native objects.
- A host object is an object defined by the host environment (such as a web browser) within which the JavaScript interpreter is embedded. The HTMLElement objects that represent the structure of a web page in client-side JavaScript are host objects. Host objects may also be native objects, as when the host environment defines methods that are normal JavaScript Function objects.
- A user-defined object is any object created by the execution of JavaScript code.
Types of properties:
- An own property is a property defined directly on an object.
- An inherited property is a property defined by an object’s prototype object.
property attributes¶
- The writable attribute specifies whether the value of the property can be set.
- The enumerable attribute specifies whether the property name is returned by a for/in loop.
- The configurable attribute specifies whether the property can be deleted and whether its attributes can be altered.
// Returns {value: 1, writable:true, enumerable:true, configurable:true}
Object.getOwnPropertyDescriptor({x:1}, "x");
// Now query the octet property of the random object defined above.
// Returns { get: /*func*/, set:undefined, enumerable:true, configurable:true}
Object.getOwnPropertyDescriptor(random, "octet");
// Returns undefined for inherited properties and properties that don't exist.
Object.getOwnPropertyDescriptor({}, "x"); // undefined, no such property
Object.getOwnPropertyDescriptor({}, "toString"); // undefined, inherited
// Does not have to include all four attributes at the same time.
Object.defineProperty(o, "x", { value : 1,
writable: true,
enumerable: false,
configurable: true});
var p = Object.defineProperties({}, {
x: { value: 1, writable: true, enumerable:true, configurable:true },
y: { value: 1, writable: true, enumerable:true, configurable:true },
r: {
get: function() { return Math.sqrt(this.x*this.x + this.y*this.y) },
enumerable:true,
configurable:true
}
});
Note
Here are the complete rules. Calls to Object.defineProperty() or Object.defineProperties() that attempt to violate them throw TypeError:
- If an object is not extensible, you can edit its existing own properties, but you cannot add new properties to it.
- If a property is not configurable, you cannot change its configurable or enumerable attributes.
- If an accessor property is not configurable, you cannot change its getter or setter method, and you cannot change it to a data property.
- If a data property is not configurable, you cannot change it to an accessor property.
- If a data property is not configurable, you cannot change its writable attribute from false to true , but you can change it from true to false .
- If a data property is not configurable and not writable, you cannot change its value. You can change the value of a property that is configurable but nonwritable. (making it writable, then changing the value, then converting it back to nonwritable).
Objects Operators¶
Creating Objects
var point = {x:0, y:0};
var o = Object.create(Object.prototype)
var o = new Object();
var a = new Arrays();
var d = new Date();
var r = new RegExp();
Querying and Setting Properties
The fact that inheritance occurs when querying properties but not when setting them is a key feature because it allows us to selectively override inherited properties.
Property assignment examines the prototype chain to determine whether the assignment is allowed. If o inherits a read-only property named x , for example, then the assignment is not allowed.
It is not an error to query a property that does not exist, undefined will be return. It is an error, however, to attempt to query a property of an object that does not exist.
An attempt to set a property p of an object o fails in these circumstances:
- o has an own property p that is read-only: it is not possible to set read-only prop- erties. (See the defineProperty() method, however, for an exception that allows configurable read-only properties to be set.)
- o has an inherited property p that is read-only: it is not possible to hide an inherited read-only property with an own property of the same name.
- o does not have an own property p ; o does not inherit a property p with a setter method, and o ’s extensible attribute is false . If p does not already exist on o , and if there is no setter method to call, then p must be added to o . But if o is not extensible, then no new properties can be defined on it.
/* If an object has properties whose name is a reserved
word, you must use square bracket notation to access them */
var author = book.author; // Get the "author" property of the book.
var name = author.surname; // Get the "surname" property of the author.
var title = book["main title"]; // Get the "main title" property of the book.
book.edition = 6; // Create an "edition" property of book.
book["main title"] = "ECMAScript"; // Set the "main title" property.
var len = book && book.subtitile && book.subtitle.length; // May doesn't exist
Deleting Properties
The delete operator only deletes own properties, not inherited ones. The delete operator does not remove properties that have a configurable attribute of false .
delete book.author; // The book object now has no author property.
delete book["main title"]; // Now it doesn't have "main title", either.
Testing Properties
var o = { x: 1 }
"x" in o; // true: o has an own property "x"
"y" in o; // false: o doesn't have a property "y"
"toString" in o; // true: o inherits a toString property
o.hasOwnProperty("x"); // true: o has an own property x
o.hasOwnProperty("y"); // false: o doesn't have a property y
o.hasOwnProperty("toString"); // false: toString is an inherited property
/* Only is property is an own property and its enumerable attribute is true */
o.propertyIsEnumerable("x"); // true: o has an own enumerable property x
Object.prototype.propertyIsEnumerable("toString"); // false: not enumerable
o.x !== undefined // It's equal to 'in' execpt property is explicitly set to undefined
if (o.x) o.x *=2 ; // If exist and is not undefined, null, false, "", 0, NaN
Enumerating Properties
for(p in o) {
if (!o.hasOwnProperty(p)) continue;
}
for(p in o) {
if (typeof o[p] === "function") continue; // Skip methods
}
Object.keys() // returns an array of the names of the enumerable own properties
Object.getOwnPropertyNames() // returns the names of all the own properties
Property Getters and Setters
var p = {
x: 1.0,
y: 1.0,
// r is a read-write accessor property with getter and setter.
get r() {
return Math.sqrt(this.x*this.x + this.y*this.y);
}, // Don't forget to put a comma after accessor methods.
set r(newvalue) {
var oldvalue = Math.sqrt(this.x*this.x + this.y*this.y);
var ratio = newvalue/oldvalue;
this.x *= ratio;
this.y *= ratio;
}
};
Object Attributes¶
Every object has associated prototype, class, and extensible attributes.
The prototype Attribute
var p = {x:1};
var o = Object.create(p);
p.isPrototypeOf(o) // => true: o inherits from p
Object.prototype.isPrototypeOf(o) // => true: p inherits from Object.prototype
Object.getPrototypeOf(o) ; // => Object { x=1}
The class Attribute
function classof(o) {
if (o === null) return "Null";
if (o === undefined) return "Undefined";
return Object.prototype.toString.call(o).slice(8,-1);
}
classof(null) // => "Null"
classof(1) // => "Number"
classof("") // => "String"
classof(false) // => "Boolean"
classof({}) // => "Object"
classof([]) // => "Array"
classof(/./) // => "Regexp"
classof(new Date()) // => "Date"
classof(window) // => "Window" (a client-side host object)
function f() {};
classof(new f()); // => "Object"
The extensible Attribute
var p = {x:1};
Object.isExtensible(p) // true
Object.preventExtensions(p) // make object nonextensible
Object.isExtensible(p) // false
p.y = 1 // will not report error, but y is not added
p.y // undefined
Object.seal(p) // object nonextensible, and all the own properties nonconfigurable.
Object.isSealed(p) // true
delete p.x // return false means cannot delete it
p.x // still exist with value 1
Object.freeze(p) // besides seal, all data properties will be read only.
Object.isFrozen() // true
Object Methods¶
var o = {x:1, y:1};
s = o.toString() ; // "[object object]"
s = JSON.stringify(o) // "{"x":1, "y":1}"
p = JSON.parse(s) // p is the deepcopy of o
p = new Number(3)
p.valueOf() // 3
JavaScript Array¶
Overview¶
JavaScript arrays are a specialized form of JavaScript object, and array indexes are really little more than property names that happen to be integers. What is special about arrays is that when you use property names that are non-negative integers less than 2^32 , the array automatically maintains the value of the length property for you. (JavaScript arrays are zero-based and use 32-bit indexes.)
JavaScript arrays are untyped: an array element may be of any type, and different elements of the same array may be of different types.
Array elements may even be objects or other arrays, which allows you to create complex data structures, such as arrays of objects and arrays of arrays.
JavaScript arrays are dynamic: they grow or shrink as needed and there is no need to declare a fixed size for the array when you create it or to reallocate it when the size changes.
JavaScript arrays may be sparse: the elements need not have contiguous indexes and there may be gaps.
Every JavaScript array has a length property. For nonsparse arrays, this property specifies the number of elements in the array. For sparse arrays, length is larger than the index of all elements.
Array Operators¶
Creating Arrays
var empty = []; // An array with no elements
var primes = [2, 3, 5, 7, 11]; // An array with 5 numeric elements
var misc = [ 1.1, true, "a", ]; // 3 elements of various types + trailing comma
var b = [[1,{x:1, y:2}], [2, {x:3, y:4}]];
var base = 1024;
var table = [base, base+1, base+2, base+3];
var count = [1,,3]; // An array with 3 elements, the middle one undefined.
var undefs = [,,]; // Normal array with 2 elements, both undefined.
var a = new Array(10) // Sparse array, length is 10, indexs/elements are all undefined
var a = new Array(5, 4, 3, 2, 1, "testing, testing"); // array with these elements
Reading and Writing Array Elements
var a = ["world"];
var value = a[0];
a[1] = 3.14;
i = 2;
a[i] = 3;
a[i + 1] = "hello";
a[-1.23] = true; // Invalid index number, this creates a property named "-1.23"
a["1000"] = 0; // Valid index string, this the 1001st element of the array
a[1.000] // Valid index float, same as a[1]
Adding and Deleting Array Elements
a = []
a[0] = "zero"; // a = ["zero"]
a.push("one") // Add a value at the end. a = ["zero", "one"]
a.push("two", "three") // Add two more values. a = ["zero", "one", "two", "three"]
delete a[2] // Delete will not change lenght, lenght is still 4
a.length = 2 // Delete elements, a = ["zero", "one"]
Iterating Arrays
for(var i = 0, len = a.length ; i < len; i++) {
if (!a[i]) continue; // Skip null, undefined, and nonexistent elements
if (a[i] === undefined) continue; // Skip undefined + nonexistent elements
if (!(i in a)) continue ; // Skip nonexistent elements
// loop body here
}
Array Type
Array.isArray([]) // => true
Array.isArray({}) // => false
Array.isArray{""} // => false
Array Methods¶
join():
var a = [1, 2, 3]; // Create a new array with these three elements
a.join(); // => "1,2,3"
a.join(" "); // => "1 2 3"
var b = new Array(10); // An array of length 10 with no elements
b.join('-') // => '---------': a string of 9 hyphens
reverse():
a.reverse() // a is now [3,2,1]
sort:
/* Always in dictionary order by default. Numbers will also be converted to string
If an array contains undefined elements, they are sorted to the end of the array.
*/
["one", "two", undefined, "four"].sort() // => ["four", "one", "two", undefined]
[11, 444, 2].sort(); // => [11, 2, 444]
[11, 444, 2].sort(function(a,b) {return a-b;}); // => [2, 11, 444]
concat():
/* concat() will always create new array, but not to change the original array.
If any of these arguments is itself an array, then it is the array elements that
are concatenated, not the array itself.
*/
var a = [1,2,3];
a.concat(4, 5) // return new array [1, 2, 3, 4, 5]
a.concat([4,5]); // return new array [1, 2, 3, 4, 5]
a.concat(4, [5,[6,7]]) // return new array [1, 2, 3, 4, 5, [6,7]]
slice():
var a = [1,2,3,4,5];
a.slice(0,3); // Returns [1,2,3]
a.slice(3); // Returns [4,5]
a.slice(1,-1); // Returns [2,3,4]
a.slice(-3,-2); // Returns [3]
splice():
/* Unlike slice() and concat() , splice() modifies the array on which it is invoked.
Unlike concat() , splice() inserts arrays themselves, not the elements of them.
*/
var a = [1,2,3,4,5,6,7,8];
a.splice(4); // Returns [5,6,7,8]; a is [1,2,3,4]
a.splice(1,2); // Returns [2,3]; a is [1,4]
var a = [1,2,3,4,5];
a.splice(2,0,'a','b'); // Returns []; a is [1,2,'a','b',3,4,5]
a.splice(2,2,[1,2],3); // Returns ['a','b']; a is [1,2,[1,2],3,3,4,5]
push()/pop():
/* The push() method appends one or more new elements to the end of an array and
returns the new length of the array. The pop() method ddeletes the last element
of an array, decrements the array length, and returns the value that it removed.
Note that both methods modify the array in place rather than produce a new array.
*/
var stack = [];
stack.push("one", "two"); // stack = ["one", "two"], return 2
stack.pop(); // stack = ["one"], return "two"
stack.push([2, 3]); // stack = ["one", [2, 3]], return 2
stack.pop(); // stack = ["one"], return [2, 3]
unshift()/shift():
/* The unshift() and shift() methods behave much like push() and pop() , except that
they insert and remove elements from the beginning of an array.
*/
var a = [1, 2];
a.unshift(0); // a = [0, 1, 2], return 3
a.shift() // a = [1, 2], return 0
a.unshift(3, [4, 5]) // a = [3, [4, 5], 1, 2], return 4
toString()/toLocalString():
[1,2,3].toString() // => '1,2,3'
["a", "b", "c"].toString() // => 'a,b,c'
[1, [2,'c']].toString() // => '1,2,c'
Note
The following methods are defined in ECMAScript 5
forEach():
/* function arguments: value[, index, array-itself]
*/
var data = [1,2,3,4,5];
var sum = 0;
data.forEach(function(value) { sum += value; });
sum // => 15
data.forEach(function(v, i, a) { a[i] = v + 1; });
data // => [2,3,4,5,6]
map():
/* Returns an new array containing the values returned by specified function.
The same function arguments as forEach(), neet return in function.
*/
a = [1, 2, 3];
b = a.map(function(x) { return x*x; }); // b = [1, 4, 9]
filter():
/* Unlike map(), filter method will skips missing elements in sparse arrays
*/
a = [5, 4, 3, 2, 1];
b = a.filter(function(x) { return x < 3 }); // b = [2, 1]
var dense = sparse.filter(function() { return true; }); // return dense array
reduce()/reduceRight():
/* //first time with initial value
function(firstElement, initialValue[, index=0, arrayitself])
// first time without initial value
function(firstElement, secondElement[, index=1, arrayitself])
// other times when reducing
function(lastReturnValue, nextElement[, index, arrayitself])
reduceRight() works just like reduce() , except that it processes the array from
highest index to lowest (right-to-left), rather than from lowest to highest.
*/
var a = [1,2,3,4,5]
var sum = a.reduce(function(x,y) { return x+y }, 0); // Sum of values
var product = a.reduce(function(x,y) { return x*y }, 1); // Product of values
var max = a.reduce(function(x,y) { return (x>y)?x:y; }); // Largest value
function extend(o, p) {
for (prop in p) {
o[prop] = p[prop];
}
return o;
}
var a = [{x:1, a:1}, {y:2, a:2}, {z:3, a:3}];
var unionLeft = {}
var unionRight = {} // Please note the different value to a
a.reduce(extend, unionLeft) // unionLeft = {x:1, a:3, y:2, z:3}
a.reduceRight(extend, unionRight) // unionRight = {z:3, a:1, y:2, x:1}
every()/some():
/* Both every() and some() stop iterating array elements as soon as they know
what value to return.
every() returns true and some returns false when invoked on an empty array.
*/
a = [1,2,3,4,5];
a.every(function(x) { return x < 10; }) // => true: all values < 10.
a.every(function(x) { return x % 2 === 0; }) // => false: not all values even.
a.some(function(x) { return x % 2 === 0; }) // => true: a has some even numbers.
indexOf()/lastIndexOf():
/* indexOf() and lastIndexOf() search an array for an element with a specified value,
and return the index of the first such element found, or –1 if none is found.
*/
a = [0,1,2,1,0];
a.indexOf(1) // => 1: a[1] is 1
a.lastIndexOf(1) // => 3: a[3] is 1
a.indexOf(3) // => -1: no element has value 3
a.indexOf(1,1) // => 3: a[3] is 1, begin the search at 1
JavaScript Function¶
Overview¶
Function names are often verbs or phrases that begin with verbs. It is a common con- vention to begin function names with a lowercase letter. When a name includes multiple words, one convention is to separate words with underscores like_this() ; another convention is to begin all words after the first with an uppercase letter likeThis() .
Function declaration statements are “hoisted” to the top of the enclosing script or the enclosing function, so that functions declared may be invoked from code that appears before they are defined. However, this is not true for functions defined as expressions, variable declarations are hoisted, variables are not hoisted, so functions defined with expressions cannot be invoked before they are defined.
If the return statement does not have an associated expression, it returns the undefined value. If a function does not contain a return statement, it return the undefined value to the caller.
Function declaration statements are not true statements, and the ECMAScript specification only allows them as top-level statements. They can appear in global code, or within other functions, but they cannot appear inside of loops, conditionals, or try/catch/finally or with statements. Note that this restriction applies only to functions declared as statements. Function definition expressions may appear anywhere in your JavaScript code. (But firefox allows define function in if statement)
In JavaScript, however, functions are not only syntax but also values, which means they can be assigned to variables, stored in the properties of objects or the elements of arrays, passed as arguments to functions, and so on.
Defining/Invoking Functions¶
declaration statement/definiton expression:
function functionName(arg1, arg2[, argN, ...]) {
//statements
return [value] //optional
}
// Function expressions can include names, which is useful for recursion.
var f = function fact(x) { if (x <= 1) return 1; else return x*fact(x-1); };
// Function expressions are sometimes defined and immediately invoked:
var tensquared = (function(x) {return x*x;}(10));
Invoked ways:
/* JavaScript functions can be invoked in four ways:
- as functions,
- as methods,
- as constructors,
- indirectly through their call() and apply() methods.
*/
var total = distance(0,0,2,1) + distance(2,1,3,5);
o.m(x, y); // m is method of object o
var o = new Object();
var o = new Object; //only when the constructor function has no args.
f.call(o, x, y) // f is not method of o, but want use o as invocation context.
method chaining:
// If object methods always return 'this' (the object itself)
shape.setX(100).setY(100).setSize(50).setOutline("red").setFill("blue").draw();
Constructor Invocation¶
If a function or method invocation is preceded by the keyword new , then it is a constructor invocation. Constructor invocations differ from regular function and method invocations in their handling of arguments, invocation context, and return value.
You can always omit a pair of empty parentheses in a constructor invocation. If a constructor has no parameters, then JavaScript constructor invocation syntax allows the argument list and parentheses to be omitted entirely.
A constructor invocation creates a new, empty object that inherits from the prototype property of the constructor. Constructor functions are intended to initialize objects and this newly created object is used as the invocation context, so the constructor function can refer to it with the this keyword.
Constructor functions do not normally use the return keyword. They typically initialize the new object and then return it. If a constructor explicitly used the return statement to return an object, then that object becomes the value of the invocation expression. But if the constructor uses return with no value, or if it returns a primitive value, that return value is ignored and the new object is used as the value of the invocation.
Function Arguments and Parameters¶
JavaScript function definitions do not specify an expected type for the function pa- rameters, and function invocations do not do any type checking on the argument values you pass. In fact, JavaScript function invocations do not even check the number of arguments being passed.
Optional Parameters
When a function is invoked with fewer arguments than declared parameters, the additional parameters are set to the undefined value:
function getPropertyNames(o, /* optional */ a) {
a = a || []; // If undefined, use a new array
for(var property in o) a.push(property);
return a;
}
Note
When designing functions with optional arguments, you should be sure to put the optional ones at the end of the argument list. The comment “optional” in the function definition to emphasize the fact that the parameter is optional.
Variable-Length Argument Lists
When a function is invoked with more argument values than declared parameter, names, the additional arguments cannot be read by names, but the Arguments object “arguments” refers to all the arguments. “arguments” is not a array, but it happens to have number indexes.
function max(initialValue, /* numbers ... */) {
var max = initialValue; //or arguments[0]
// Loop through the arguments, looking for other arguments.
for(var i = 1; i < arguments.length; i++) {
if (arguments[i] > max) max = arguments[i];
}
return max;
}
There are also “callee” and “caller” properties in Arguments object: - arguments.callee refers to this currently running function. - arguments.caller refers to the function that called this one.
Using Object Properties As Arguments
When a function has more than three parameters, it becomes difficult for the programmer who invokes the function to remember the correct order in which to pass arguments.
function easycopy(args) {
if (isArray(args.from) && isArray(args.to)) {
arraycopy(args.from,
args.from_start || 0, // Note default value provided
args.to,
args.to_start || 0,
args.length);
}
else throw new Error("argument from and to must be array!");
}
// Here is how you might invoke easycopy():
var a = [1,2,3,4], b = [];
easycopy({from: a, to: b, length: 4});
Properties of Function¶
length Function itself also has length, you can access it like this: “functionName.length” or “arguments.callee.length”
prototype Every function(except the one return by bind) will have this property. It’s used when invoked as constructor, and class inheritance.
“prototype” will auto have a “constructor” property. F.prototype.constructor===F is true for any function.
Defining Your Own Function Properties Function is a specialized kind of object, which means that functions can have properties. Sometimes, it is better to store the information in a property of the Function object instead of cluttering up the name space by defining a global variable.
// Compute factorials and cache results as properties of the function itself.
function factorial(n) {
if (isFinite(n) && n>0 && n==Math.round(n)) { // Finite, positive ints only
if (!(n in factorial)) // If no cached result
factorial[n] = n * factorial(n-1); // Compute and cache it
return factorial[n]; // Return the cached result
}
else return NaN; // If input was bad
}
factorial[1] = 1; // Initialize the cache to hold this base case.
Methods of Function¶
call()/apply():
functionName.call(object, arg1, arg2, ...argN)
functionName.apply(object, [arg1, arg2, ...argN]) # arguments list
bind():
var g = f.bind(o[,args]); // g(args...) is equal to o.f(args...)
function f(y,z) { return this.x + y + z };
var g1 = f.bind({x:1}); // Bind object as this only
var g2 = f.bind({x:1}, 2); // Bind this and y
g1(2, 3) // => 6: this.x is bound to 1, y is 2 and z is 3
g2(3) // => 6: this.x is bound to 1, y is bound to 2 and z is 3
toString():
f.toString(); // => '[native code]'
Function():
var f = new Function("x", "y", "return x*y;");
Note
There are a few points that are important to understand about the Function() constructor:
- The Function() constructor allows JavaScript functions to be dynamically created and compiled at runtime.
- The Function() constructor parses the function body and creates a new function object each time it is called. If the call to the constructor appears within a loop or within a frequently called function, this process can be inefficient. By contrast, nested functions and function definition expressions that appear within loops are not recompiled each time they are encountered.
- The Function() constructor creates functions which do not use lexical scoping; instead, they are always compiled as if they were top-level functions, use global object as invoked context.
Functions As Namespaces¶
Sometimes, it’s useful to define a function simply to act as a temporary namespace in which you can define variables without polluting the global namespace.
var some = (function() { // function as module or namespace
// Module code goes here.
return somethings
}()); // end the function literal and invoke it now.
this¶
Note that this is a keyword, not a variable or property name. JavaScript syntax does not allow you to assign a value to this .
For function invocation in ECMAScript 3 and nonstrict ECMAScript 5, the invocation context (the this value) is the global object. In strict mode, however, the invocation context is undefined .
Functions written to be invoked as functions do not typically use the this keyword at all. It can be used, however, to determine whether strict mode is in effect:
// Define and invoke a function to determine if we're in strict mode.
var strict = (function() { return !this; }());
When a function is invoked on or through an object, that object is the invocation context or this value for the function.
Unlike variables, the this keyword does not have a scope, and nested functions do not inherit the this value of their caller. If a nested function is invoked as a method, its this value is the object it was invoked on. If a nested function is invoked as a function then its this value will be either the global object (non-strict mode) or undefined (strict mode). It is a common mistake to assume that a nested function invoked as a function can use this to obtain the invocation context of the outer function. If you want to access the this value of the outer function, you need to store that value into a variable that is in scope for the inner function. It is common to use the variable self for this purpose. For example:
var o = {
m: function() {
var self = this; // Save the this value in a variable.
console.log(this === o); // true, if called by o.m()
function f() { // A nested function f
console.log(this === o); // false, this is global object or undefined.
console.log(self === o); // true, self is the this of outer function.
}
f();
}
};
o.m();
Note that the new object is always set as the invocation context even if the constructor invocation looks like a method invocation. For example:
// o.m() is used as constructor, o is not used as the invocation context, it's p.
p = new o.m()
closure¶
Like most modern programming languages, JavaScript uses lexical scoping. This means that functions are executed using the variable scope that was in effect when they were defined, not the variable scope that is in effect when they are invoked. In order to implement lexical scoping, the internal state of a JavaScript function object must include not only the code of the function but also a reference to the current scope chain. This is an old term that refers to the fact that the function’s variables have bindings in the scope chain and that therefore the function is “closed over” its variables.
This combination of a function object and a scope (a set of variable bindings) in which the function’s variables are resolved is called a closure in the computer science literature.
Technically, all JavaScript functions are closures: they are objects, and they have a scope chain associated with them.
Closures become interesting when they are invoked under a different scope chain than the one that was in effect when they were defined. This happens most commonly when a nested function object is returned from the function within which it was defined. common in JavaScript programming.
In C language, function’s local variables are defined on a CPU stack, then they would indeed cease to exist when the function returned. But JavaScript described it as a list of objects.
Each time a JavaScript function is invoked, a new object is created to hold the local variables for that invocation, and that object is added to the scope chain. When the function returns, that variable binding object is removed from the scope chain. If there were no nested functions, there are no more references to the binding object and it gets garbage collected. If there were nested functions defined, then each of those functions has a reference to the scope chain, and that scope chain refers to the variable binding object.
If those nested functions objects remained within their outer function, however, then they themselves will be garbage collected, along with the variable binding object they referred to. But if the function defines a nested function and returns it or stores it into a property somewhere, then there will be an external reference to the nested function. It won’t be garbage collected, and the variable binding object it refers to won’t be garbage collected either.
Example 1:
var scope = "global scope"; // A global variable
function checkscope() {
var scope = "local scope"; // A local variable
function f() { return scope; }
return f;
}
/* Remember the fundamental rule of lexical scoping: JavaScript functions are executed
using the scope chain that was in effect when they were defined. */
checkscope()(); // It should return "local scope" here.
Example 2:
// Function declarations are hoisted so we can do this assignment here.
uniqueInteger.counter = 0;
// This function returns a different integer each time it is called.
function uniqueInteger() {
return uniqueInteger.counter++; // Increment and return counter property
}
/* The above version has one problem, uniqueInteger.counter can be changed out of function.
So, rewrite the uniqueInteger() function using closures.
*/
var uniqueInteger = (function() { // Define and invoke immediately
var counter = 0; // Private state of function below
return function() { return counter++; };
}()); // no way to access counter after function return
Example 3:
/* Two or more nested functions to be defined within the same outer function
will share the same scope chain.
*/
function counter(n) {
return {
count: function() { return n++; },
reset: function() { n = 0; }
};
}
var c = counter(0), d = counter(0); // two counters with two copies of scope chains
c.count() // => 0
d.count() // => 0: they count independently
c.reset() // reset() and count() methods share state
c.count() // => 0: because we reset c
d.count() // => 1: d was not reset
Note
As what mentioned above, nested functions in closure function will not share ‘this’ and ‘arguments’ of the outer function, except that we save them by other variables.
JavaScript Classes¶
Define Class by Constructors/Prototype¶
The prototype object is fundamental to the identity of a class: two objects are instances of the same class if and only if they inherit from the same prototype object.
The constructor function that initializes the state of a new object is not fundamental: two constructor functions may have prototype properties that point to the same prototype object. Then both constructors can be used to create instances of the same class.
function Range(from, to) {
// Store the start and end points (state) of this new range object.
// These are noninherited properties that are unique to this object.
this.from = from;
this.to = to;
}
// Extend the predefined Range.prototype object so we don't overwrite
// the automatically created Range.prototype.constructor property.
// All Range objects inherit from this object.
// Note that the property name must be "prototype" for this to work.
Range.prototype.includes = function(x) {
return this.from<=x && x<=this.to;
};
Range.prototype.foreach = function(f) {
for(var x = Math.ceil(this.from); x <= this.to; x++) f(x);
};
Range.prototype.toString = function() {
return "(" + this.from + "..." + this.to + ")";
};
// Here are example uses of a range object
var r = new Range(1,3); // Create a range object
r.includes(2); // => true: 2 is in the range
r.foreach(console.log); // => "1 2 3"
r.toString(); // => "(1...3)"
Java-Style Classes in JavaScript¶
Constructor object: The constructor function (an object) defines a name for a JavaScript class.
Properties you add to this constructor object serve as class fields and class methods (depending on whether the property values are functions or not). But these properties of constructor are not inheritable.
Prototype object: The properties of this object are inherited by all instances of the class, and properties whose values are functions behave like instance methods of the class.
Instance object: Each instance of a class is an object in its own right, and properties defined directly on an instance are not shared by any other instances. Nonfunction properties defined on instances behave as the instance fields of the class.
Augmenting Classes¶
JavaScript’s prototype-based inheritance mechanism is dynamic: an object inherits properties from its prototype, even if the prototype changes after the object is created. This means that we can augment JavaScript classes simply by adding new methods to their prototype objects.
JavaScript Regular Expressions¶
Just as string literals are specified as characters within quotation marks, regular expression literals are specified as characters within a pair of slash ( / ) characters.
var pattern = /s$/;
var pattern = new RegExp("s$");
var pattern = new RegExp("\\d{5}", "g");
// RegExp can create regular expression dynamically in runtime
var something; // Maybe some thing from user.
var pattern = new RegExp(something)
Regular-expression literal characters¶
=============== ================================================================
Character Matches
=============== ================================================================
Alphanumeric character Itself
\0 The NUL character (\u0000)
\t Tab (\u0009 )
\n Newline ( \u000A )
\v Vertical tab ( \u000B )
\f Form feed ( \u000C )
\r Carriage return ( \u000D )
\xnn The hexadecimal number; \x0A == \n
\uxxxx The hexadecimal number Unicode; \u0009 == \t
\cX The control character ^X ; \cJ == \n
=============== ================================================================
Regular expression character classes¶
=============== ================================================================
Character Matches
=============== ================================================================
[...] Any one character between the brackets.
[^...] Any one character not between the brackets.
. Any character except newline or another Unicode line terminator.
\w Any ASCII word character.
\W Any character that is not an ASCII word character.
\s Any Unicode whitespace character.
\S Any character that is not Unicode whitespace.
\d Any ASCII digit. Equivalent to [0-9] .
\D Any character other than an ASCII digit. Equivalent to [^0-9] .
[\b] A literal backspace (special case).
=============== ================================================================
Regular expression repetition characters¶
=============== ================================================================
Character Meaning
=============== ================================================================
{n, m} Match previous item at least n times but no more than m times.
{n,} Match the previous item n or more times.
{n} Match exactly n occurrences of the previous item.
? Match zero or one occurrences of the previous item. {0,1}
+ Match one or more occurrences of the previous item. {1,}
* Match zero or more occurrences of the previous item. {0,}
=============== ================================================================
Note
The repetition characters listed above table match as many times as possible while still allowing any following parts of the regular expression to match. We say that this repetition is “greedy.”
It is also possible to specify that repetition should be done in a nongreedy way. Simply follow the repetition character or characters with a question mark: ?? , +? or even {1,5}? .
Match of /a+?b/ in ‘aaab’ still return ‘aaab’ because start at first ‘a’.
Regular expression alternation, grouping, and reference characters¶
=============== ================================================================
Character Meaning
=============== ================================================================
| Alternation. Match either the subexpression to the left or
the subexpression to the right.
(...) Grouping. Group items into a single unit that can be used with
* , + , ? , | , and so on. Also remember the characters
that match this group for use with later references.
(?:...) Grouping only. Group items into a single unit, but do not
remember the characters that match this group.
\n Match the same characters that were matched when group number n
was first matched.
Groups are subexpressions within (possibly nested) parentheses.
Group numbers are assigned by counting left parentheses from
left to right. Groups formed with (?: are not numbered.
=============== ================================================================
Examples:
/ab|cd|ef/ // matches “ab” or “cd” or “ef”.
/\d{3}|[a-z]{4}/ // matches either three digits or four lowercase letters.
/java(script)?/ // matches “java” followed by the optional “script”.
/(ab|cd)+|ef/ // matches either the string “ef” or one or more
// repetitions of either of the strings “ab” or “cd”.
/([Jj]ava([Ss]cript)?)\sis\s(fun\w*)/ // ([Ss]cript) is referred to as \2
/(['"][^'"]*\1) // matches '.*' or ".*"
Regular-expression anchor characters¶
=============== ================================================================
Character Meaning
=============== ================================================================
^ Match beginning of string. (or line in multiline searches)
$ Match the end of string. (end of a line in multiline searches)
\b Match a word boundary. (Note: [\b] matches backspace.)
\B Match a position that is not a word boundary.
(?=p) A positive lookahead assertion.
Require that the following characters match the pattern p,
but do not include those characters in the match.
(?!p) A negative lookahead assertion.
Require that the following characters don't match the pattern p.
=============== ================================================================
Examples:
/\B[Ss]cript/ // Matches "JavaScript" and "postscript", but not "script".
/[Jj]ava([Ss]cript)?(?=\:)/ // Matches the word "JavaScript" in
"JavaScript: The Definitive Guide", but it does not match
"Java" in "Java in a Nutshell".
/Java(?!Script)/ // Matches "Java" in "JaveBeans" but null in "JavaScript"
Regular-expression flags¶
=============== ================================================================
Character Meaning
=============== ================================================================
i Perform case-insensitive matching.
g Perform a global match—that is, find all matches.
m Multiline mode. ^/$ matche beginning/end of line or string.
=============== ================================================================
Examples:
/java$/im // Matches “java” as well as “Java\nis fun”
RegExp Properties¶
source: Read-only string that contains the text of the regular expression.
global: Read-only boolean value that specifies whether the regular expression has the g flag.
ignoreCase: Read-only boolean value that specifies whether the regular expression has the i flag.
multiline: Read-only boolean value that specifies whether the regular expression has the m flag.
lastIndex: Read/write integer. For patterns with the g flag, it stores the position in the string at which the next search is to begin. It is used by the exec() and test() methods
RegExp Methods¶
exec():
var pattern = /Java/g;
var text = "JavaScript is more fun than Java!";
var result;
while((result = pattern.exec(text)) != null) {
alert("Matched '" + result[0] + "'" +
" at position " + result.index +
"; next search begins at " + pattern.lastIndex);
}
test():
var pattern = /java/i;
pattern.test("JavaScript"); // Returns true
Shell Variables¶
Internal Variables:
$SHELL Current shell name, such as "/bin/bash"
$PPID Parent PID
$UID User ID, specially, root UID is zero
$RANDOM Return a random number
$IFS shell separator character, default value is "\ "
$PATH System path
$PWD Current directory
$HOME Home directory
$SECONDS time from it begin to run
$LINENO Current line number
$FUNCNAME Current function name
$BASH_LINENO Line number where this function is called
$$ Current PID
$! PID of the last process which running in the background
$# Number of parameters
$? Command or script return value
$0 Script or function name which is called.
$1 Ther first parameter, $N means the Nth parameter
$* all parameters
“$*” "$1 $2 $3 ..."
“$@” "$1" "$2" "$3" ...
Wildcard Variables:
[[:alnum:]] all number characters
[[:digit:]] all number characters
[[:xdigit:]] all hex number characters
[[:alpha:]] all letter characters
[[:lower:]] all lower letter characters
[[:upper:]] all upper letter characters
[[:punct:]] all punctuation characters
[[:graph:]] all nonblank characters
[[:space:]] all space characters
[[:blank:]] all blank characters, such as whitespace and tab
[[:cntrl:]] all control characters
[[:print:]] all printable characters
Wildcard Variables:
$variable variable value
${variable} variable value, separate name from the following letters
${variable:n:m} part of variable string, start at n, lenght is m
${variable:n} part of variable string, from the n to the end
${#variable} length of the variable string
${variable:?message} If the variable exists and is not NULL, returen it,
otherwise output the message and exit the script.
${variable:=word} If the variable exists and is not NULL, returen it,
otherwise set the variable to word and return the word.
${variable:-word} If the variable exists and is not NULL, returen it,
otherwise return ther word, variable is not changed.
${variable:+word} If the variable exists and is not NULL, return the word,
otherwise return NULL.
${variable%.*} From right to left, match ".*", then remove this part.
e.g. if variable="file-v1.0.txt", then return "file-v1.0"
${variable%%.*} From right to left, match ".*" as long as possible, then
remove this part. e.g. if variable="file-v1.0.txt",
then return "file-v1"
${variable#*.} From left to right, match "*.", then remove this part,
e.g. if variable="file-v1.0.txt", then return "0.txt"
${variable##*.} From left to right, match "*." as long as possible, then
remove this part. e.g. if variable="file-v1.0.txt",
then return ".txt"
${variable/patt/str} Replace pattern matching part with string.
${variable//patt/str} Replace pattern matching part (as long as possible) with string.
Variables expression:
let result=no1+no2
let i++
result=$[ $no1 + $no2 ]
result=$(($no1 + $no2))
result=`expr 3+4`
result=$(expr $no1 + 5)
Shell Test And Compoare¶
File test:
-b File is block special device (for files like /dev/hda1)
-c File is character special (for files like /dev/tty)
-d File is a directory
-e File exists
-f File is a regular file
-g File has its set-group-ID bit set
-h File is a symbolic link (same as -L)
-G File is owned by the effective group ID
-k File has its sticky bit set
-L File is a symbolic link (same as -h)
-O File is owned by the effective user ID
-p File is a named pipe
-r File is readable
-s File has a size greater than zero
-S File is a socket
-u File has its set-user-ID bit set
-w File is writable
-x File is executable
Variable Test:
-lt < Less than
-le <= Less than or equal to
-gt > Greater than
-ge >= Greater than or equal to
-eq =, == Equal to
-ne != Not equal to
-n "$var" Variable is not empty
-z "$var" Varibale is empty
Examples:
if [ -r $FILE -a -w $FILE ]
if [ -z "$V1" -o -z "${V2:=YIKES}" ]
if [[ "${MYFILENAME}" == *.jpg ]]
if [[ "$FN" == *.@(jpg|jpeg) ]]
if [[ "$CDTRACK" =~ "([[:alpha:][:blank:]]*)- ([[:digit:]]*) - (.*)$" ]]
Shell Terminal¶
terminal color:
ESC="\e" or "\033"
ESC[0m # reset all (colors and brightness)
ESC[1m # bright
ESC[2m # dim (looks same as normal brightness)
ESC[22m # normal brightness
# FOREGROUND:
ESC[30m # black
ESC[31m # red
ESC[32m # green
ESC[33m # yellow
ESC[34m # blue
ESC[35m # magenta
ESC[36m # cyan
ESC[37m # white
ESC[39m # reset
# BACKGROUND
ESC[40m # black
ESC[41m # red
ESC[42m # green
ESC[43m # yellow
ESC[44m # blue
ESC[45m # magenta
ESC[46m # cyan
ESC[47m # white
ESC[49m # reset
Other Control code:
ESC[4m # under line
ESC[5m # blink
ESC[7m # enable echo
ESC[8m # disable echo
ESC[nA # move up <n> lines
ESC[nB # move up <n> lines
ESC[nC # move right <n> colums
ESC[nD # move left <n> colums
ESC[y;xH # set the cursor position
ESC[2J # clear full screen
ESC[K # clear from current to end of the line
ESC[s # save the cursor position
ESC[u # restore the cursor position
ESC[?25l # hidden the cursor
ESC[?25h # display the cursor
References
Linux Shell Sed¶
Exampes¶
# & means the matched pattern 'aaa'.
sed -e 's/aaa/+&+/g input.txt
# Multi command be be executed together.
sed -e '2d;s/aaa/eee/g' input.txt #or
sed -e '2d' -e 's/aaa/eee/g' input.txt
#If in script file with -f option, {} is used to
execute multi line commands.
#If not last line, append to one line
#Otherwise for the last line, replace \n to ''.
:loop
N
$!b loop
s/\n//g
sed [OPTION]… {script-only-if-no-other-script} [input-file]…
OPTIONS¶
Popular Options:
-n, --quiet, --silent
suppress automatic printing of pattern space
-e script, --expression=script
-f script-file, --file=script-file
-i[SUFFIX], --in-place[=SUFFIX]
edit files in place (makes backup if SUFFIX supplied)
-l N, --line-length=N
specify the desired line-wrap length for the `l' command
--posix
disable all GNU extensions.
-r, --regexp-extended
use extended regular expressions in the script.
Addresses¶
Sed commands can be given with no address, one address or two addresses as range.
- With no addresses, the command will be executed for all input lines;
- With one address, the command will only be executed for input lines which match that address;
- With two addresses, the command will be executed for all input lines which match the inclusive range of lines starting from the first address and continuing to the second address.
Three things to note about address ranges:
- The syntax is addr1,addr2 (i.e., the addresses are separated by a comma);
- The line which addr1 matched will always be accepted, even if addr2 selects an earlier line;
- If addr2 is a regexp, it will not be tested against the line that addr1 matched.
After the address (or address-range), and before the command, a ! may be inserted, which specifies that the command shall only be executed if the address (or address-range) does not match.
The following address types are supported:
number Match only the specified line number (which increments cumulatively across files, unless the -s option is specified on the
command line).
first~step
Match every step'th line starting with line first. For example, ``sed -n 1~2p'' will print all the odd-numbered lines in
the input stream, and the address 2~5 will match every fifth line, starting with the second.
$ Match the last line.
/regexp/
Match lines matching the regular expression regexp.
0,addr2
Start out in "matched first address" state, until addr2 is found. This is similar to 1,addr2, except that if addr2
matches the very first line of input the 0,addr2 form will be at the end of its range, whereas the 1,addr2 form will still
be at the beginning of its range. This works only when addr2 is a regular expression.
addr1,+N
Will match addr1 and the N lines following addr1.
addr1,~N
Will match addr1 and the lines following addr1 until the next line whose input line number is a multiple of N.
COMMAND¶
Zero-address commands:
: label Label for b and t commands. #comment The comment extends until the next newline } The closing bracket of a { } block. Zero- or One- address commands:: = Print the current line number. a text Append text, which has each embedded newline preceded by a backslash. i text Insert text, which has each embedded newline preceded by a backslash. q [exit-code] Immediately quit the sed script without processing any more input, except that if auto-print is not disabled the current pattern space will be printed. Q [exit-code] Immediately quit the sed script without processing any more input. r filename Append text read from filename. R filename Append a line read from filename. Each invocation of the command reads a line from the file. This is a GNU extension.Commands which accept address ranges:
{ Begin a block of commands (end with a }). b label Branch to label; if label is omitted, branch to end of script. c text Replace the selected lines with text, which has each embedded newline preceded by a backslash. d Delete pattern space. Start next cycle. D If pattern space contains no newline, start a normal new cycle as if the d command was issued. Otherwise, delete text in the pattern space up to the first newline, and restart cycle with the resultant pattern space, without reading a new line of input. h H Copy/append pattern space to hold space. g G Copy/append hold space to pattern space. l List out the current line in a ``visually unambiguous'' form. l width List out the current line in a ``visually unambiguous'' form, breaking it at width characters. This is a GNU extension. n N Read/append the next line of input into the pattern space. p Print the current pattern space. P Print up to the first embedded newline of the current pattern space. s/regexp/replacement/ Attempt to match regexp against the pattern space. If successful, replace that portion matched with replacement. The replacement may contain the special character & to refer to that portion of the pattern space which matched, and the special escapes \1 through \9 to refer to the corresponding matching sub-expressions in the regexp. y/source/dest/ Transliterate the characters in the pattern space which appear in source to the corresponding character in dest. t label If a s/// has done a successful substitution since the last input line was read and since the last t or T command, then branch to label; if label is omitted, branch to end of script. T label If no s/// has done a successful substitution since the last input line was read and since the last t or T command, then branch to label; if label is omitted, branch to end of script. w filename Write the current pattern space to filename. W filename Write the first line of the current pattern space to filename. x Exchange the contents of the hold and pattern spaces.
Shell Magic Commands¶
Daily Commands¶
Run some thing in the background screen session
screen -d -m -S session_name "comman"See also
screen --helportmux --helpRun the last command by root account
sudo !!Go to the last cd history directory.
cd -Replace in last command and then run.
^old^newShow ASCII Table
man asciifind and rm
find ./ -name "xxx.xx" -exec rm -rf {} \;ssh remote command, no need to copy script to server
ssh user@server bash </path/to/script.sh
Network Commands¶
port monitor
netstat -tlnpGet the external IP of this host
curl ifconfig.me
Shell Project¶
Logo¶
Generate logo by figlet:
git clone https://github.com/cmatsuoka/figlet
cd figlet
make figlet
./showfigfonts -d ./fonts
./figlet MyApp -f ./fonts/standard.flf
Self Decompress¶
Example:
#! /bin/bash
UNZIP_DIR=`mktemp -d /tmp/mytool.$$.XXXX`
PAYLOAD_OFFSET=`grep -an -m1 "^ARCHIVE_MARKER:" $0 | cut -d: -f1`
tail -n +$((PAYLOAD_OFFSET+1)) $0 | tar zxf - --ignore-command-error -C ${UNZIP_DIR} 2>/dev/null
exec ${UNZIP_DIR}/mytool.sh "$@"
exit $?
ARCHIVE_MARKER:
...#The tar file here
Shell Switch Root¶
chroot¶
Example:
cd $ROOT
mount -t proc proc proc
mount -t sysfs sys sys
mount -t devtmpfs devtmpfs dev
chroot $ROOT /bin/systemd-tmpfiles --create --remove --boot >/dev/null 2>&1
chroot $ROOT <command-and-args-doing-something>
RET=$?
chroot $ROOT /bin/systemd-tmpfiles --clean >/dev/null 2>&1
umount -l dev
umount -l sys
umount -l proc
exit $RET
switch_root¶
Usually, it’s used in initramfs to switch to the real root filesystem. switch_root moves already mounted /proc, /dev, /sys and /run to newroot and makes newroot the new root filesystem and starts init process.
Example:
exec /sbin/switch_root "$ROOTMNT" /sbin/init $@
Python Parallel¶
Example 1:
#!/bin/bash
SUB_PROCESS_NUM=8
TMPFILE="/tmp/$$.fifo"
# Open w/r FIFO on file handler 9
mkfifo $TMPFILE
exec 9<>$TMPFILE
for ((i=0; i<$SUB_PROCESS_NUM; i++)) ; do
echo ""
done >&9
for ((i=0; i<20; i++)) ; do
read -u 9
{ # do something here
echo "${i} start------------------------------"
sleep 1
echo "${i} exit-------------------------------"
# must add new line to enable another sub-process
echo "" >&9
} &
done
# wait for all background process done and exit.
wait
exec 9>&-
rm -f $TMPFILE
Example 2:
task_start() {
local name="$1"
touch $TMPDIR/${name}.monitor_running
while [ -e "$TMPDIR/${name}.monitor_running" ] ; do
...
# add_task $other_task_name $action
done
}
task_stop() {
local name="$1"
rm -rf $TMPDIR/${name}.monitor_running
}
add_task() {
local name=$1
local action=$2
grep -o "$name.$action" $JOBFILE >&- 2>&- || {
echo "$name.$action" >> $JOBFILE 2>&-
}
}
# Process each task in the $jobfile in FIFO order
do_tasks() {
local line queued_task
if [ -f $JOBFILE ] ; then
mv $JOBFILE ${JOBFILE}.work
while read line; do
execute_task() {
case $2 in
"action1") task_start $action2_task_name &;;
"action2") task_start $action1_task_name &;;
*) DBG_ERROR "## Unknown task command: $2 ##";;
esac
}
queued_task=$(echo $line | awk -F "." '{print $1,$2}')
execute_task $queued_task
done < ${JOBFILE}.work
rm -rf ${JOBFILE}.work
fi
}
task_loop() {
task_start $first_task_name &
while [ "$task_daemon_running" == 1 ] ; do
do_tasks
sleep 1
done
}
Shell Opensource¶
Awesome-shell: https://github.com/alebcay/awesome-shell
Git Quickstart¶
Installation¶
For example, in Ubuntu:
sudo apt-get install git
sudo apt-get install git-doc git-svn git-email git-gui gitk
sudo apt-get install gitg
sudo apt-get install qgit
sudo apt-get install kdiff3
Get help¶
git help [command]
Configuration¶
git config --global user.name "myname"
git config --global user.email "my@example.com"
git config --global core.editor vim
git config --global color.ui true
git config --global merge.tool kdiff3
git config --global pull.rebase true
git config --global commit.template "~/.gitmessage"
# For current user only
git config --global alias.st status
git config --global alias.ci commit
git config --global alias.co checkout
git config --global alias.br branch
git config --global alias.lg log --color --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)<%an>%Creset' --abbrev-commit
# For all user
sudo git config --system alias.st status
sudo git config --system alias.ci commit
sudo git config --system alias.co checkout
sudo git config --system alias.br branch
# Get current repo settings
git config --get remote.origin.url
git config -f path-to-repo/.git/config --get remote.origin.url
git ls-remote <rempte-origin-url> refs/heads/master #remote head id
git ls-remote ./ refs/heads/master #local head id
git-extras¶
Install: sudo apt-get install git-extras
Usage: https://github.com/tj/git-extras/blob/master/Commands.md
Git Commands¶
Complete the command by ‘tab’
git --version #Prints the Git suite version
git config -e #edit .git/config
git config -e --global #edit ~/.gitconfig
git config -e --system #edit /etc/gitconfig
git config a.b #get a.b in .git/config
git config --global a.b something #set a.b in ~/.gitconfig
git config --unset a.b #remove a.b
git add hello.c #Add a single file to stage
git add doc/\*.txt #Add all *.txt file
git add -A #Add all the modified, deleted and new files
git add -i #Add modified contents in the working tree interactively to the index.
git add -p [filename] #equal to -i and then select the patch command
stage this hunk [y,n,q,a,d,/,s,e?]
y - stage this hunk
n - do not stage this hunk
q - quit; do not stage this hunk nor any of the remaining ones
a - stage this hunk and all later hunks in the file
d - do not stage this hunk nor any of the later hunks in the file
g - select a hunk to go to
/ - search for a hunk matching the given regex
j - leave this hunk undecided, see next undecided hunk
J - leave this hunk undecided, see next hunk
k - leave this hunk undecided, see previous undecided hunk
K - leave this hunk undecided, see previous hunk
s - split the current hunk into smaller hunks
e - manually edit the current hunk
? - print help
git clean -fd #Clear all unmanaged files
git clean -fX #Clear all the files defined in .gitignore
git checkout . #Clear all changes to the managed files
git commit -s #Add signoff line automatically, and need to add log message in editor.
git commit -m "log message" #Commit with log message without pop-up editor.
git commit --amend #Update the log message in last commit
git commit --amend --allow-empty --reset-author #Update the author in last commit
git checkout master #checkout master branch
git checkout . #clear all the changes to the managed file in workspace
git checkout -- filename #checkout single file
git checkout branchname -- filename #checkout file in other branch, both stage and workspace will be changed.
git checkout -b branchname #create and switch to the new branch
git reset --soft <commit> #reset HEAD refernce, but not the content
git reset --soft HEAD^ #revert the git commit
git reset --hard HEAD^ #reset both reference and content changed.
git reset --hard master@{2}
git reset [HEAD] #revert git add, reset what are not committed.
git reset HEAD filename #reset single file in stage
git clone
git push
git pull
git pull --rebase #rebase
git merge <commit> # merge current HEAD and commit
git cherry-pick <commit> #pick commit in any branch and put it after current HEAD
git revert HEAD #revert itself is a commit
git rebase --autostash #Stash changes before rebasing
git grep --line-number "something" #grep in tracked files
git status #option -s for short version
git status --ignored #show the ignored files list at the same time
git config user.name [value] #no value means get value, otherwise set the value
git config --luser.name
git log --pretty=fuller
git log --pretty=oneline #equal to --oneline
git log --pretty=raw
git log --graph --oneline
git log -p HEAD #-p is to show the diff patch in each commit
git log --stat HEAD #show what files are changed
git log --oneline --decorate #Show all tags and other references
git log -3
git log ^HEAD~3 HEAD
git log HEAD~3..HEAD
git log --pretty=short --decorate #Show commit log together with ref names, such as tags.
git log --format="%h | %s | %d" #Show log in "Hash | Subject | Ref-names" format
git log Branch1 ^Branch2 #Show commit which in Branch1 but not in Branch2
git log --all --grep 'something' #Show commit log which match the special text.
git log -- filename #Show commit log related to special file.
git shortlog #Show summary only, but grouped by author.
git cherry -v master #Show all commits which are not merged to master yet.
git describe --tags --abbrev=0 #Show the last tag
git show HEAD --stat
git show-ref
git show <branch_name>:<file_name> #Show file in special branch
git diff #workspace to stage
git diff --cached #stage to remote
git diff HEAD #workspace to remote
git diff HEAD^ HEAD
git diff HEAD^ HEAD -- filename #the diff of single file
git diff --word-diff #Show inline changes in detail
git blame filename
git blame -L 6 +5 filename #only show 5 lines begin from line 6
git format-patch -1
git stash #store all current changes
git stash list #list all stash
git stash pop #apply and remove the newest stash
git stash apply #only apply but do not remove
git stash drop [stash] #remove the stash, the newest by default
git stash clear #clear all stash
git branch #show local branches
git branch -r/-a #show remote/all branches
git branch branchname <commit> #create branch based on <commit>, HEAD by default
git branch -d branchname #remove local branch， see also -D
git branch -m oldbranch newbranch #rename branch，see also -M
git push <url> :remote-branch #remove remote branch, need to sync back by 'git fetch -p'
git tag -m "log message" tagname #create tag
git tag #show local tags
git tag -l V3_* #show local tags which match the pattern
git tag -d tagname # delete the tag
git push origin tagname #push the tag in local
git describe --tags #desciption about the last tags
git remote -v #show remote URL
git remote add new-remote path #add new remote URL
git remote rename old-remote new-remote
git remote rm remotename #remove remote
git remote show origin #show info about current origin
git remote update
git rev-parse --git-dir #show the path of .git folder of current repository
git rev-parse --show-toplevel #show the top direcotry of current repository
git rev-parse --show-prefix #how to go from top directory to current directory
git rev-parse --show-cdup #how to back to top directory (e.g. '../../../')
git rev-parse HEAD #show the commit ID of HEAD
git rev-parse HEAD^
git rev-parse --symbolic --branches #show local branches
git rev-parse --symbolic --tags #show local tags
git rev-parse --abbrev-ref HEAD #Show the name of current branch
git rev-list --oneline A #show the version relationship
git ls-files -s #show file tree and last commit ID
git ls-tree -l HEAD
git ls-files --others -i --exclude-standard # Show all files which are ignored.
git cat-file -t <ID> #type of the ID (commit or tag ...)
git cat-file -p <ID> #content of the ID
git reflog show master #show the log on master branch
git reflog -1 #show last action of HEAD
Git Work Flow¶
Create Repository¶
Initialize a empty repository:
git init --bare
- Clone a mirror repository， sync via “git remote update/git fetch” later::
- git clone –mirror
Generate git patch outside git repo¶
git init
git add -A
git commit -s
#change some things ...
git add -A
git commit -s
git show
git format-patch -1
Git flow¶
http://nvie.com/posts/a-successful-git-branching-model/
Install:
$ sudo apt-get install git-flow
Usage:
| +-- init
| |
|git flow +-- feature --+-- start --+-- name
| | | |
| +-- release --+-- finish --+
| | | |
| +-- hotfix --+-- publish --+
| | |
| --+-- pull --+
Details:
a. create develop branch
git branch develop
git push -u origin develop
b. Start new feature
git checkout -b some-feature develop
# Optionally, push branch to origin:
git push -u origin some-feature
git status
git add some-file
git commit
c. finish the feature
git pull origin develop
git checkout develop
git merge --no-ff some-feature
git push origin develop
git branch -d some-feature
# If you pushed branch to origin:
git push origin --delete some-feature
d. Start new relase
git checkout -b release-0.1.0 develop
# Optional: Bump version number, commit
# Prepare release, commit
e. Finish the release
git checkout master
git merge --no-ff release-0.1.0
git push
git checkout develop
git merge --no-ff release-0.1.0
git push
git branch -d release-0.1.0
# If you pushed branch to origin:
git push origin --delete release-0.1.0
git tag -a v0.1.0 master
git push --tags
f. Start new hotfix
git checkout -b hotfix-0.1.1 master
g. Finish the hotfix
git checkout master
git merge --no-ff hotfix-0.1.1
git push
git checkout develop
git merge --no-ff hotfix-0.1.1
git push
git branch -d hotfix-0.1.1
git tag -a v0.1.1 master
git push --tags
sphinx¶
- http://sphinx-doc.org/
- http://www.sphinx-doc.org/en/stable/contents.html
- http://www.ibm.com/developerworks/cn/opensource/os-sphinx-documentation/
Install¶
$ sudo apt-get install sphinx-common python-sphinx texlive$ sudo pip install recommonmark$ sudo pip install sphinx_rtd_theme
Quick Start¶
$ sphinx-quickstart
+ *conf.py*: sphinx `configuration file`_
+ *index.rst*: top level document, define project structure
+ *Makefile*: make html/pdf
$ make htmlor$ sphinx-build -b html ./ ./_build/html
Configuration¶
Theme¶
http://www.sphinx-doc.org/en/stable/theming.html
For example, use the “Read the Docs” theme:
$ sudo pip install sphinx_rtd_theme
$ vi conf.py
html_theme = 'sphinx_rtd_theme'
Document structure¶
Deploy to Read the Docs¶
http://avnpc.com/pages/writing-best-documentation-by-sphinx-github-readthedocs
http://docs.readthedocs.org/en/latest/getting_started.html
Note
When push to github, let “Read the Docs” updeate automatically: github repo page => settings tabpage => Webhooks & Services => Add service => ReadTheDocs => Active
Convert to blog¶
Markdown¶
Specification¶
Full Examples (zh_CN)¶
Difference between md and rst¶
Online Editor¶
Markdown to HTML¶
Download markdown-browser-0.6.0-beta1.tgz
Test it like this:
tar -xvf markdown-browser-0.6.0-beta1.tgz cd markdown-browser-0.6.0-beta1 vi test.html && firefox test.html
test.html:
<!DOCTYPE html> <html> <head> <link href="http://cdn.bootcss.com/bootstrap/3.3.1/css/bootstrap.min.css" rel="stylesheet"> </head> <body style="padding:30px"> <textarea id="text-input" oninput="this.editor.update()">Type **Markdown** here.</textarea> <div id="preview"> </div> <script src="markdown.js"></script> <script> function Editor(input, preview) { this.update = function () { preview.innerHTML = markdown.toHTML(input.value); }; input.editor = this; this.update(); } var $ = function (id) { return document.getElementById(id); }; new Editor($("text-input"), $("preview")); </script> </body> </html>
Linux syslinux¶
https://wiki.archlinux.org/index.php/syslinux http://git.kernel.org/cgit/boot/syslinux/syslinux.git/tree/doc/menu.txt http://git.kernel.org/cgit/boot/syslinux/syslinux.git/
Syslinux is a collection of boot loaders capable of booting from hard drives, CDs, and over the network via PXE.
It supports the FAT, ext2, ext3, ext4, and Btrfs file systems.
Boot process overview¶
- Stage 1 : Part 1 - Load MBR - At boot, the BIOS loads the 440 byte MBR boot code at the start of the disk (/usr/lib/syslinux/bios/mbr.bin or /usr/lib/syslinux/bios/gptmbr.bin).
- Stage 1 : Part 2 - Search active partition. The Stage 1 MBR boot code looks for the partition that is marked as active (boot flag in MBR disks). Let us assume this is the /boot partition for example.
- Stage 2 : Part 1 - Execute volume boot record - The Stage 1 MBR boot code executes the Volume Boot Record (VBR) of the /boot partition. In the case of syslinux, the VBR boot code is the starting sector of /boot/syslinux/ldlinux.sys which is created by the extlinux –install command.
- Stage 2 : Part 2 - Execute /boot/syslinux/ldlinux.sys - The VBR will load rest of /boot/syslinux/ldlinux.sys. The sector location of /boot/syslinux/ldlinux.sys should not change, otherwise syslinux will not boot.
Note
In the case of Btrfs, the above method will not work since files move around resulting in changing of the sector location of ldlinux.sys. Therefore, in BTRFS the entire ldlinux.sys code is embedded in the space following the VBR and is not installed at /boot/syslinux/ldlinux.sys unlike the case of other filesystems.
- Stage 3 - Load /boot/syslinux/ldlinux.c32 - The /boot/syslinux/ldlinux.sys will load the /boot/syslinux/ldlinux.c32 (core module) that contains the rest of core part of syslinux that could not be fit into ldlinux.sys (due to file-size constraints). The ldlinux.c32 should be present in every syslinux installation and should match the version of ldlinux.sys installed in the partition. Otherwise syslinux will fail to boot. See http://bugzilla.syslinux.org/show_bug.cgi?id=7 for more info.
- Stage 4 - Search and Load configuration file - Once Syslinux is fully loaded, it looks for /boot/syslinux/syslinux.cfg (or /boot/syslinux/extlinux.conf in some cases) and loads it if it is found. If no configuration file is found, you will be dropped to a syslinux boot: prompt. This step and rest of non-core part of syslinux (/boot/syslinux/*.c32 modules, excluding lib*.c32 and ldlinux.c32) require /boot/syslinux/lib*.c32 (library) modules to be present (http://www.syslinux.org/wiki/index.php/Common_Problems#ELF). The lib*.c32 library modules and non-core *.c32 modules should match the version of ldlinux.sys installed in the partition.
Limitations of UEFI Syslinux¶
- UEFI Syslinux application syslinux.efi cannot be signed by sbsign (from sbsigntool) for UEFI Secure Boot. Bug report - http://bugzilla.syslinux.org/show_bug.cgi?id=8
- Using TAB to edit kernel parameters in UEFI Syslinux menu lead to garbaged display (text on top of one-another). Bug report - http://bugzilla.syslinux.org/show_bug.cgi?id=9
- UEFI Syslinux does not support chainloading other EFI applications like UEFI Shell or Windows Boot Manager. Enhancement request - http://bugzilla.syslinux.org/show_bug.cgi?id=17
- In some cases, UEFI Syslinux might not boot in some Virtual Machines like QEMU/OVMF or VirtualBox or some VMware products/versions and in some UEFI emulation environments like DUET. A Syslinux contributor has confirmed no such issues present on VMware Workstation 10.0.2 and Syslinux-6.02 or later. Bug reports - http://bugzilla.syslinux.org/show_bug.cgi?id=21 and http://bugzilla.syslinux.org/show_bug.cgi?id=23
- Memdisk is not available for UEFI. Enhancement request - http://bugzilla.syslinux.org/show_bug.cgi?id=30
Bootloader Grub2¶
http://www.gnu.org/software/grub/manual/grub.html http://www.gnu.org/software/grub/grub-documentation.html
Runtime Tools¶
- grub-mkconfig:
- make the template of the grub.cfg file
- grub-reboot <idx>:
- save current entry index to ‘prev_saved_entry’, and save <idx> to ‘saved_entry’ in grubenv.
- grub-editenv [grubenv-file] <command> <var-name>:
- Set/Delete/List the variables in grubenv.
- grub-menulst2cfg:
- Convert the grub 0.97 conf file to grub2 cfg file.
Linux Systemd¶
https://www.freedesktop.org/wiki/Software/systemd/ https://en.wikipedia.org/wiki/Systemd https://github.com/systemd/systemd/blob/master/README
Why systemd¶
Tips And Tricks¶
Basic operators:
$ sudo systemctl start nginx.service
$ sudo systemctl stop nginx.service
$ sudo systemctl restart nginx.service
$ sudo systemctl reload nginx.service
$ sudo systemctl enable nginx.service
$ sudo systemctl disable nginx.service
$ sudo systemctl daemon-reload
$ systemctl status nginx.service
$ sudo systemctl poweroff
$ sudo systemctl reboot
Other tools:
hostnamectl #get/set hostname
localectl #get/set local settings, e.g. LANG
timedatectl #get/set time/date/timezone
$ sudo timedatectl set-timezone America/New_York
$ sudo timedatectl set-time YYYY-MM-DD
$ sudo timedatectl set-time HH:MM:SS
.. note::
timedatectl also show the systemd ntp(systemd-timesyncd service) status.
If you only wish to see the journal entries from the current boot:
$ journalctl -b
$ journalctl -u nginx.Service
To expand all dependent units recursively:
systemctl list-dependencies --all nginx.service
Cgroup Tree:
$ systemd-cgls
Switch the run level:
$ ln -sf /usr/lib/systemd/system/graphical.target /etc/systemd/system/default.target
$ ln -sf /usr/lib/systemd/system/multi-user.target /etc/systemd/system/default.target
Unit Dependencies:
$ systemctl show -p "Wants" multi-user.target
.. note::
Instead of "Wants" you might also try "WantedBy", "Requires", "RequiredBy",
"Conflicts", "ConflictedBy", "Before", "After" for the respective types of
dependencies and their inverse.
Show the main PID:
systemctl -p MainPID show <service-name>
Mask a service, ensure that service cannot even be started manually anymore:
$ ln -s /dev/null /etc/systemd/system/ntpd.service
$ systemctl daemon-reload
You can modify the system state to transition between targets with the isolate option. This will stop any units that are not tied to the specified target. Be sure that the target you are isolating does not stop any essential services:
sudo systemctl isolate multi-user.target
Check the boot time:
$ systemd-analyze
$ systemd-analyze nginx
$ systemd-analyze critical-chain
$ systemd-analyze critical-chain atd.service
Network Target¶
https://www.freedesktop.org/wiki/Software/systemd/NetworkTarget/ https://www.freedesktop.org/software/systemd/man/systemd-networkd.service.html
Systemd Units¶
service.service, socket.socket, device.device, mount.mount, automount.automount, swap.swap, target.target, path.path, timer.timer, slice.slice, scope.scope
systemd.unit¶
https://www.freedesktop.org/software/systemd/man/systemd.unit.html
[Unit] and [Install] section define the relationship between other units. [Unit] and [Install] sections configure the common configuration items, for example dependences, conditions, and other relationship between units. They are generic sections used in almost all type units.
Each unit may have a type-specific section, e.g. [Service] for a service unit.
Various settings are allowed to be specified more than once, often, multiple settings form a list
If systemd encounters an unknown option, it will write a warning log message but continue loading the unit.
If an option or section name is prefixed with X-, it is ignored completely by systemd. Options within an ignored section do not need the prefix. Applications may use this to include additional information in the unit files.
Boolean arguments used in unit files can be written in various formats. For positive settings the strings 1, yes, true and on are equivalent. For negative settings, the strings 0, no, false and off are equivalent.
Empty lines and lines starting with # or ; are ignored. This may be used for commenting. Lines ending in a backslash are concatenated with the following line while reading and the backslash is replaced by a space character. This may be used to wrap long lines.
If a unit file is empty (i.e. has the file size 0) or is symlinked to /dev/null, its configuration will not be loaded and it appears with a load state of “masked”, and cannot be activated. Use this as an effective way to fully disable a unit, making it impossible to start it even manually.
Unit files are loaded from a set of paths determined during compilation, described in the two tables below. Unit files found in directories listed earlier override files with the same name in directories lower in the list:
/etc/systemd/system Local configuration
/run/systemd/system Runtime units
/usr/lib/systemd/system Units of installed packages
systemd.service¶
https://www.freedesktop.org/software/systemd/man/systemd.service.html
[Service] section configure the service specific configuration options.
Unless DefaultDependencies is set to false, service units will implicitly have dependencies of type Requires= and After= on sysinit.target, a dependency of type After= on basic.target as well as dependencies of type Conflicts= and Before= on shutdown.target.
Note that notify type daemon has to support systemd’s notification protocol, else systemd will think the service has not started yet and kill it after a timeout. (systemd_notify can be used in script to tell systemd itself is ready.)
systemd.resource-control¶
https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html
The resource control configuration options are configured in the [Slice], [Scope], [Service], [Socket], [Mount], or [Swap] sections, depending on the unit type.
systemd debug¶
https://freedesktop.org/wiki/Software/systemd/Debugging/
debug shell¶
You can enable shell access very early in the startup process to fall back on and diagnose systemd related boot up issues with various systemctl commands. It also active when shutdown is not finished. Enable it using:
systemctl enable debug-shell.service
shutdown debug¶
If normal reboot or poweroff work, but take a suspiciously long time, then boot with the debug options:
systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M enforcing=0
save the following script as /usr/lib/systemd/system-shutdown/debug.sh and make it executable:
#!/bin/sh
mount -o remount,rw /
dmesg > /shutdown-log.txt
mount -o remount,ro /
reboot
File System btrfs¶
https://btrfs.wiki.kernel.org/index.php/Main_Page
sudo apt-get install btrfs-tools
Create btrfs filesystem¶
# Create the btrfs filesystem
mkfs.btrfs -L label /dev/sdc2
mount /dev/sdc2 /mnt
# Show information
btrfs filesystem show
Modify the btrfs filesystem¶
# Add/Delete
btrfs device add /dev/sdc3 /mnt #Don't need mkfs firtly.
btrfs device delete /dev/sdc3 /mnt
# Resize btrfs filesystem online
btffs filesystem resize 4G /mnt
btffs filesystem resize +1G /mnt
btffs filesystem resize -1G /mnt
Snapshot and Rollback¶
# Create snapshot
btrfs subvolume snapshot /mnt /mnt/snap1
# Delete snapshot
btrfs subvolume delete /mnt/snap1
# Rollback
btrfs subvolume list
btrfs subvolume set-default <id> <path>
umount /mnt && mount /dev/sdc2 /mnt || reboot
Ubuntu Installation¶
Maybe we should also fork/create a project to do this automatically. http://yuez.me/cong-ling-da-jian-he-pei-zhi-osxkai-fa-huan-jing/
Pre-Installation¶
Backup: .bashrc .vimrc .vim .zshrc .mozilla .thunderbird .gitconfig
Installation¶
Ubuntu16.04, there is a gxfboot.c32 error, after the “boot:” promote, fix it manually, input “tab” => “live” or “liveinstall” => press ENTER
Post-Configurations¶
Restore: .bashrc .vimrc .vim .mozilla .thunderbird .gitconfig (If you have not formated the /home partition, you don’t need to do this)
All Settings => Languages All Settings => Dispalys All Settings => Appearance => Behavior => Auto-hide the Launcher
Post-Intallations¶
First time Upgrade:
sudo apt-get update && sudo apt-get upgrade
Change default shell:
sudo dpkg-reconfigure dash --> select No for set bash as default shell
Basic packages:
sudo apt-get install git-core git-email kdiff3 gitg
sudo apt-get install vim ctags g++ tofrodos quilt tree
sudo apt-get install rar unrar p7zip p7zip-rar p7zip-full
sudo apt-get install ntfs-3g ntfs-config
sudo apt-get install python-pip
Packages for Developer:
sudo apt-get install \
autoconf autoconf2.64 autoconf-doc autoconf-archive \
glib2.0 glib2.0-data glib2.0-dev \
libdbus-1-dev libdbus-glib-1-dev \
libtool libtool-doc \
texlive texinfo libxml-parser-perl libxml-simple-perl \
diffstat texi2html gawk chrpath \
gcc-multilib g++-multilib libc6-dev-i386 libzip-dev \
libc6-i386
Installing the ia32 libs needs the ubuntu 13.04 sources, because this package is droped after this ubuntu 13.04:
sudo -i
cd /etc/apt/sources.list.d
echo "deb http://old-releases.ubuntu.com/ubuntu/ raring main restricted universe multiverse" > ia32-libs-raring.list
apt-get update
apt-get install ia32-libs
rm /etc/apt/sources.list.d/ia32-libs-raring.list
apt-get update
youtao-dict¶
To avoid gstreamer0.10-plugins-ugly dependence issue, use deepin version but not ubuntu version. http://codown.youdao.com/cidian/linux/youdao-dict_1.1.0-0-deepin_amd64.deb
sudo apt-get install python3-pyqt5 python3-xlib sudo apt-get install tesseract-ocr tesseract-ocr-chi-sim tesseract-ocr-chi-tra sudo apt-get install ttf-wqy-microhei sudo apt-get install python3-pyqt5.qtmultimedia python3-pyqt5.qtquick python3-pyqt5.qtwebkit sudo apt-get install qtdeclarative5-controls-plugin libqt5multimedia5-plugins sudo dpkg -i youdao-dict_1.1.0-0-deepin_amd64.deb
python3¶
Maybe the python3 is already installed, but you need to install pip3:
sudo apt-get install python3-pip
Then you can install python3 modules using sudo pip3 install xxx
Ubuntu Usage¶
Change the default application:
System Settings => System => Details => Default Applications
Ubuntu Services¶
Openssh¶
sudo apt-get install openssh-server
# Edit the etc/ssh/sshd_config file
UseDNS no
GSSAPIAuthentication no
Vnc¶
Set the vnc server password: vncpasswd
Start the vncserver: vncserver
Start the vncviewer: vncviewer IP/HOSTNAME:PORTNUM
Kill the vncserver: vncserver -kill :1
Ubuntu Shortcuts¶
Ubuntu Terminal(16.04)¶
Edit => Preferences => Shortcuts:
New Terminal in New Tab ctrl+shift+t
New Terminal in New Window ctrl+shift+n
Close Terminal tab ctrl+shift+w
Close all Terminal tabs ctrl+shift+q
Copy ctrl+shift+c
Paste ctrl+shift+v
Full Screen F11
Zoom In ctrl++
Zoom Out ctrl+-
Normal Size ctrl+0
Switch to Previous Tab ctrl+PageUp
Switch to Nexty Tab ctrl+PageDown
Switch to Tab <N> alt+<N>
Move Tab to the Left ctrl+shift+PageUp
Move Tab to the Right ctrl+shift+PageDown
Ubuntu Zsh Usage¶
zsh¶
https://wiki.gentoo.org/wiki/Zsh/Guide
sudo apt-get install zsh
chsh -s $(which zsh) # need to reboot the system
oh-my-zsh¶
install oh-my-zsh:
sh -c "$(curl -fsSL https://raw.github.com/robbyrussell/oh-my-zsh/master/tools/install.sh)"
themes
Preview: https://github.com/robbyrussell/oh-my-zsh/wiki/Themes
or:
autoload -U promptinit
prompinit
prompt -p
Modify the prompt in ~/.oh-my-zsh/themes/robbyrussell.zsh-theme or export prompt directly in .zshrc:
local ret_status="%(?:%{$fg_bold[green]%}➜ :%{$fg_bold[red]%}➜ )"
local suffix="%{$fg_bold[blue]%}$ %{$reset_color%}"
PROMPT='%{$fg_bold[cyan]%}%d%{$reset_color%} $(git_prompt_info)
${ret_status} %{$fg[green]%}[%* %D]%{$reset_color%} ${suffix}'
local ret_status="%(?:%{$fg_bold[green]%}➜ :%{$fg_bold[red]%}➜ )"
local suffix="%{$fg_bold[blue]%}$ %{$reset_color%}"
export PROMPT='%{$fg_bold[cyan]%}%d%{$reset_color%} $(git_prompt_info)
%F{magenta}%B%K{magenta}█▓▒░%F{white}%K{magenta}%B%* %D%b%F{magenta}%K{black}█▓▒░ ${ret_status}%b%k%f%{$reset_color%} '
Sequence Printed
%T System time (HH:MM)
%* System time (HH:MM:SS)
%D System date (YY-MM-DD)
%n Your username
%B - %b Begin - end bold print
%U - %u Begin - end underlining
%d Your current working directory
%~ Your current working directory, relative to ~
%M The computer's hostname
%m The computer's hostname (truncated before the first period)
%l Your current tty
plugins
1. git: Git is include by default. It will show git information in prompt line.
autojump:
sudo apt-get install autojump
The final plugins:
plugins=(git autojump)
FAQ¶
- Q: vi ~/.z (TAB Complete)arguments:450: _vim_files: function definition file not found.A: rm -rf ~/.zcompdump* ; then relogin the zsh
- Q: time -p ls (Command not found: -p)A: time in zsh is a shell reserved word, but not /usr/bin/time binary tool.
Ubuntu Vim Editor¶
Setup¶
https://github.com/VundleVim/Vundle.vim/wiki/Examples
Dependencies:
$ sudo apt-get install python vim exuberant-ctags git
$ sudo pip install dbgp vim-debug pep8 flake8 pyflakes isort
$ sudo npm -g install instant-markdown-d
- Install vundle::
- $ git clone https://github.com/VundleVim/Vundle.vim.git ~/.vim/bundle/Vundle.vim
Prepare for YouCompleteMe:
$ sudo apt-get install cmake build-essential libclang1 libclang-dev python-dev
$ vim vimrc:
Plugin 'Valloric/YouCompleteMe'
Install plugins:
:PluginInstall # Install all defined plugins
:PluginUpdate # Update all defined plugins
or:
$ vim +PluginInstall +qall # Install all defined plugins outside Vim.
Postinstall for YouCompleteMe:
$ cd ~/.vim/bundle/YouCompleteMe
$ git submodule update --init --recursive
$ ./install.py --clang-completer
Plugins Usage¶
YouCompleteMe https://github.com/Valloric/YouCompleteMe
ctrl+o # Return
ctrl+i # Go ahead
Plugins Shortcuts¶
nerdtree:
<CR>|o: open in prev window (auto close if NERDTreeQuitOnOpen = 1)
go: preview only, so will not change the window focuse.
gi: split in preview window
gs: vsplit in preview window
i: split current nerdtree window, so it's better useing it with NERDTreeQuitOnOpen = 1
s: open vsplist
Ubuntu Atom Editor¶
http://flight-manual.atom.io/ https://atom-china.org/t/atom/62 http://atom-china.org/ https://atom.io/docs
Install¶
https://github.com/atom/atom/blob/master/docs/build-instructions/linux.md
Requirements:
sudo apt-get install build-essential git libgnome-keyring-dev fakeroot nodejs
Get deb package here: https://atom.io/ Install it like this:
sudo dpkg -i atom-amd64.deb
Or:
sudo add-apt-repository ppa:webupd8team/atom
sudo apt-get update
sudo apt-get install atom
Configurations¶
Plugins¶
you-complete-me autocomplete autocomplete-html autocomplete-css autocomplete-json autocomplete-paths
python-tools python-indent autocomplete-python linter-pep8 (pip install pep8) atom-beautify
docblockr
file-icons
tree-view-git-status
vim-mode-plus or vim-mode + vim-mode-visual-block
terminal-plus
fuzzy-grep
Shortcuts¶
ctrl-o Open file
ctrl-shift-o Open folder
ctrl-\ Hide folder tree
ctrl-/ Trigger line comments
ctrl-, Open Settings pane
ctrl-w q close current pane
ctrl+p serach files
ctrl+g go to line
alt-<N> Switch to tab <N>
alt-\ Switch to foler-tree or return back
Ubuntu Tmux¶
shortcuts & cheatsheet¶
https://gist.github.com/MohamedAlaa/2961058
For example:
tmux new -s myname #Start new named tmux session
tmux a[ttach] -t myname #Attach named tmux session
tmux ls/list-sessions #List all tmux session
tmux kill-session -t myname #Kill named tmux session
Configuration Example¶
#--basic-------------------------------------------------------#
##set -g default-terminal "screen"
set -g default-terminal "screen-256color"
set -g display-time 5000
set -g repeat-time 1000
set -g escape-time 0
set -g history-limit 65535
set -g base-index 1
set -g pane-base-index 1
set -g history-limit 10000
set -g terminal-overrides 'xterm*:smcup@:rmcup@'
#--bindkeys----------------------------------------------------#
#Escape key
unbind C-b
set -g prefix C-a
bind a send-prefix
#windows
unbind '"'
bind h split-window -h
unbind %
bind v split-window -v
unbind i
bind C-g display-message
#pane
bind j select-pane -L
bind i select-pane -U
bind k select-pane -D
bind l select-pane -R
#quit
unbind x
unbind &
unbind q
bind q confirm-before -p "kill-pane #P? (y/n)" kill-pane
bind Q confirm-before -p "kill-window #W? (y/n)" kill-window
bind e confirm-before -p "kill-session #S? (y/n)" kill-session
#misc
bind r source-file ~/.tmux.conf \; display "Reloaded!"
#bind m command-prompt "splitw -h 'exec man %%'"
#bind @ command-prompt "splitw -h 'exec perldoc -f %%'"
#--Window------------------------------------------------------#
setw -g mode-keys vi
setw -g utf8 on
setw -g window-status-current-bg red
setw -g window-status-current-fg white
setw -g window-status-current-attr bright
setw -g window-status-attr bright
setw -g window-status-format '#[fg=cyan,dim]#I#[fg=blue] #[default]#W#[fg=grey,dim]'
setw -g window-status-current-format '#[fg=cyan,dim]#I#[fg=blue] #[default]#W#[fg=grey,dim]'
#--Statusbar---------------------------------------------------#
set -g status-keys vi
set -g status-utf8 on
set -g status-left ""
set -g status-left-length 0
set -g status-interval 2
set -g status-fg white
set -g status-bg blue
set -g status-right "#[fg=white]#S:#W:#P [#(tmux ls|wc -l) sessions]#[default]"
#set -g status-justify centre
#set -g status-right-length 10
#--Mouse-------------------------------------------------------#
setw -g mode-mouse on
set -g mouse-resize-pane on
set -g mouse-select-pane on
set -g mouse-select-window on
MongonDB QuickStart¶
Overview¶
A record in MongoDB is a document, which is a data structure composed of field and value pairs. MongoDB documents are similar to JSON objects. The values of fields may include other documents, arrays, and arrays of documents.
MongoDB stores documents in collections. Collections are analogous to tables in relational databases. Unlike a table, however, a collection does not require its documents to have the same schema.
ubuntu Setup¶
https://docs.mongodb.com/manual/tutorial/install-mongodb-on-ubuntu/
Note
MongoDB only provides packages for 64-bit LTS (long-term support) Ubuntu releases.
Install V3.2:
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927
echo "deb http://repo.mongodb.org/apt/ubuntu xenial/mongodb-org/3.2 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
sudo apt-get update
sudo apt-get install mongodb-org
Configuration:
https://docs.mongodb.com/manual/reference/configuration-options/
- Create sub folders::
- mkdir mongodb-simple && cd mongodb-simple mkdir data conf log
Create special conf file:
cp /etc/mongod.conf ./conf/
# vi dbpath and log path in the conf file
storage:
dbPath: ./data
systemLog:
path: ./log/mongod.log
processManagement:
fork: true
Start:
$ mongod -f conf/mongod.conf
Stop:
kill <mongod-pid> # kill or kill -15, don't use kill -9
Operators in JavaScript syntax compatiable Mongo shell:
$ mongo # Connect "localhost:<defaultPort>/test" by default
$ use mydb # use database, create new one if it doesn't exist.
$ db.user.insert({name:"xxx", age:18}) # create at first
$ db.user.find()
{ "_id" : ObjectId("549a6cae1cdcaf3b048887b8"), "age" : 18, "name" : "xxx" }
Python PyMongo¶
https://docs.mongodb.com/getting-started/python/
https://github.com/mongodb/mongo-python-driver
http://api.mongodb.com/python/current/
APIs:
collection.insert_one({})
collection.insert_many([{}])
collection.find_one({})
collection.find({})
collection.create_index()
collection.count()
WiFi Linux Commands¶
AP mode¶
WiFi AP mode commands:
brctl delif br-bridge wlan0
ifconfig br-bridge down
brctl delbr br-bridge
killall hostapd
#Configure correct WiFi mode
iw wlan0 del
iw phy phy0 interface add wlan0 type managed
/usr/local/bin/hostapd -B /root/examples/wifi-apmode/hostapd.conf
#Remove the wlan0 interface address and let the bridge do the work
brctl addbr br-bridge
ifconfig br-bridge 192.168.1.1
ifconfig wlan0 0.0.0.0
#Enable IPv4 forwarding
echo "1" > /proc/sys/net/ipv4/ip_forward
#Bring the bridge up
/sbin/ifconfig br-bridge up
#Use dnsmasq to provide dns and dhcp functions
killall dnsmasq
/usr/sbin/dnsmasq -Z -i br-bridge -F 192.168.1.100,192.168.1.249,255.255.255.0,720m
#Routing and NAT rules
/sbin/route add -net 239.0.0.0/8 br-bridge
/sbin/iptables -A FORWARD -i br-bridge -s 192.168.1.0/24 -j ACCEPT
/sbin/iptables -A POSTROUTING -t nat -j MASQUERADE
hostapd configuration file example:
interface=wlan0
bridge=br-bridge
driver=nl80211
logger_syslog=-1
logger_syslog_level=2
logger_stdout=-1
logger_stdout_level=2
dump_file=/tmp/hostapd.dump
ctrl_interface=/var/run/hostapd
ctrl_interface_group=0
ssid=WRIDPSDK
country_code=US
hw_mode=g
channel=11
beacon_int=100
dtim_period=2
max_num_sta=255
rts_threshold=2347
fragm_threshold=2346
macaddr_acl=0
auth_algs=3
ignore_broadcast_ssid=0
wmm_enabled=1
wmm_ac_bk_cwmin=4
wmm_ac_bk_cwmax=10
wmm_ac_bk_aifs=7
wmm_ac_bk_txop_limit=0
wmm_ac_bk_acm=0
wmm_ac_be_aifs=3
wmm_ac_be_cwmin=4
wmm_ac_be_cwmax=10
wmm_ac_be_txop_limit=0
wmm_ac_be_acm=0
wmm_ac_vi_aifs=2
wmm_ac_vi_cwmin=3
wmm_ac_vi_cwmax=4
wmm_ac_vi_txop_limit=94
wmm_ac_vi_acm=0
wmm_ac_vo_aifs=2
wmm_ac_vo_cwmin=2
wmm_ac_vo_cwmax=3
wmm_ac_vo_txop_limit=47
wmm_ac_vo_acm=0
eap_server=0
wpa=2
wpa_passphrase=mypassword
wpa_key_mgmt=WPA-PSK
wpa_pairwise=TKIP CCMP
wpa_group_rekey=600
wpa_gmk_rekey=86400
Client Mode¶
WiFi Client commands:
iw wlan0 del
iw phy phy0 interface add wlan0 type managed
iw wlan0 set channel <channel>
ifconfig wlan0 up
# if remote AP uses no security mode, don't need wpa_supplicant
wpa_supplicant -B -D wext -i wlan0 -c /etc/wpasupplicant.conf
iw wlan0 connect -w <ap_essid>
iw wlan0 link # show connection result information
#See also other commands
ifconfig wlan0 down
dhclient -r wlan0
ifconfig wlan0 up
iwconfig wlan0 essid "IDPDK-7AAA"
iwconfig wlan0 mode Managed
dhclient wlan0
dhclient -timeout 5 -pf /var/run/dhclient-wlan0.pid \
-lf /var/lib/dhclient/dhclient-wlan0.leases wlan0
#Scan APs
ifconfig wlan0 up
iw wlan0 scan | grep "SSID:" | sort | uniq
#List all APs
iwlist wlan0 scan | grep -B 5 -A 20 "7AAA" | sed '/Address/,/Cell/p;d'
wpasupplicant.conf examples
WEP:
#/etc/wpasupplicant.conf
ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=wheel
# Allow all valid ciphers
network={
ssid="ap_essid"
scan_ssid=1
key_mgmt=NONE
wep_key0=my_wep_key
}
WPA2:
#/etc/wpasupplicant.conf
ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=wheel
# Allow all valid ciphers
network={
ssid="ap_essid"
scan_ssid=1
key_mgmt=WPA-PSK
psk=my_wpa_key
}
WPA Enterprise:
ctrl_interface=/var/run/wpa_supplicant
ctrl_interface_group=wheel
network={
ssid="ap_essid"
scan_ssid=1
key_mgmt=WPA-EAP
pairwise=CCMP TKIP
group=CCMP TKIP WEP104 WEP40
eap=PEAP
identity="username"
password="password"
}
Ad-Hoc mode¶
To connect to a remote AP with no security Mode:
iw wlan0 del
iw phy phy0 interface add wlan0 type adhoc
iw wlan0 set channel <channel>
ifconfig wlan0 up
iw dev wlan0 ibss join <ssid> <frequency>
The following table shows the channels and their corresponding frequencies to be used in the joining of the ad-hoc network using the iw command:
1 - 2412
2 - 2417
3 - 2422
4 - 2427
5 - 2432
6 - 2437
7 - 2442
8 - 2447
9 - 2452
10 - 2457
11 - 2462
12 - 2467
13 - 2472
PPPD usage¶
The simple way to use pppd is to provide the peers and chat file. The following example is for 3G.
/etc/ppp/peers/test:
debug debug debug
/dev/ttyACM0
115200
logfile /var/log/ppp.log
nomultilink
defaultroute
noipdefault
ipcp-restart 7
ipcp-accept-local
ipcp-accept-remote
lcp-echo-interval 0
lcp-echo-failure 999
modem
noauth
nocrtscts
noipdefault
noaccomp
noccp
novj
usepeerdns
persist
connect '/usr/sbin/chat -v -f /etc/ppp/chat/test'
/etc/ppp/chat/test:
ABORT BUSY
ABORT 'NO CARRIER'
ABORT VOICE
ABORT 'NO DIALTONE'
ABORT 'NO DIAL TONE'
ABORT 'NO ANSWER'
ABORT DELAYED
ABORT ERROR
SAY "Initializing\n"
'' 'ATZ'
OK 'AT+CGDCONT=1,"IP","$USE_APN"'
OK 'AT+CFUN=1'
OK 'AT'
OK 'ATD*99***1#'
SAY "Calling UMTS/GPRS"
TIMEOUT 120
CONNECT ' '
Skill Tree¶
Knownsec_RD_Checklist: http://blog.knownsec.com/Knownsec_RD_Checklist/index.html
Vulnerability Scan Tools¶
Nexpose Community Edition¶
download https://www.rapid7.com/products/nexpose/compare-downloads.jsp
(Need to register with a non-personal email account, it seems 139.com is permitted. Then you will receive the activation code.)
Please download the user’s guide and installation guide document here: https://community.rapid7.com/community/nexpose?linkid=blue
install $ sudo NeXposeSetup-Linux64.bin -c (Note: You’d better to select the initialization when install it)
If your forgot to selecte initialization when install, then you can initialize it manually like this: $ cd ~/Disk2/rapid7/nexpose/nsc $ sudo ./nsc.sh
start
If your selected initialization when install, then it maybe already started. Otherwise, you can start it manually like this: $ cd ~/Disk2/rapid7/nexpose/nsc $ sudo ./nscsvc.sh start
Go to https://localhost:3780 to configure your scanner
Nessus Home¶
download http://www.tenable.com/products/nessus/select-your-operating-system
Get the free activation code of home version here: http://www.tenable.com/products/nessus/nessus-plugins/obtain-an-activation-code
install $ sudo dpkg -i Nessus-6.7.0-ubuntu1110_i386.deb
start $ /etc/init.d/nessusd start Go to https://localhost:8834/ to configure your scanner
OpenVAS¶
install http://www.openvas.org/install-packages.html
$ sudo vi /etc/apt/sources.list
(Add the following ppa information)
# For openvas
deb http://ppa.launchpad.net/mrazavi/openvas/ubuntu trusty main
deb-src http://ppa.launchpad.net/mrazavi/openvas/ubuntu trusty main
$ sudo apt-get update $ sudo apt-get install sqlite3 openvas
$ sudo openvas-nvt-sync $ sudo openvas-scapdata-sync $ sudo openvas-certdata-sync
start $ sudo service openvas-scanner restart $ sudo service openvas-manager restart $ sudo openvasmd –rebuild –progress
Login into https://localhost:443 with “admin” as username and password. In case of “Secure Connection Failed” in firefox, use chromium.
Openscap (Not Passed)¶
download https://fedorahosted.org/releases/o/p/openscap/ https://github.com/OpenSCAP/openscap https://github.com/OpenSCAP/scap-workbench
install $ sudo apt-get install gcc swig pkg-config xml2 libxml2-dev xsltproc libxslt-dev libgcrypt11-dev libpcre3-dev python-dev libcurl4-openssl-dev libbz2-dev $ sudo apt-get install libopenscap-dev
$ tar -xvf openscap-1.2.9.tar.gz $ cd openscap-1.2.9/ $ ./configure $ make $ sudo make install
start
scap-workbanch https://fedorahosted.org/released/scap-workbench/
https://github.com/OpenSCAP/scap-workbench $ git clone https://github.com/OpenSCAP/scap-workbench.git $ cd scap-workbench/ $ mkdir build; cd build $ cmake ../ $ make $ sudo make install