Welcome to Go Watch Folder’s documentation!¶
Installation¶
Install script¶
The install script only supports systemd
for Linux OS (like CentOS 7.x and Ubuntu 16.04), so if your Linux distro supports that and you can give a try.
$ curl https://cloud.swiftstack.com/v1/AUTH_gowf/gowf/instal
l-gowf.sh |bash
Build binary from source¶
1. Clone source code¶
$ mkdir -p ${GOPATH}/src/github.com/swiftstack
$ cd ${GOPATH}/src/github.com/swiftstack
$ git clone https://github.com/swiftstack/go-watch-folder
2. Install packages and run tests¶
Please install automake
package on your Mac to make sure you can use make
command.
$ make prepare
go get -u github.com/sanbornm/go-selfupdate/...
go get -u github.com/jteeuwen/go-bindata/...
go get -u github.com/golang/dep/cmd/dep
dep ensure
$ make test
### Unit tests for GoWF ###
Usage of /var/folders/6g/9slv6z496gn6q8n0vmhdlvbh0000gn/T/go-build594409299/b001/gowf.test:
.
.
.
INFO: 2018/07/03 16:14:44 watch.go:133: Checker-0 (/Users/charles/gowork/src/github.com/swiftstack/go-watch-folder/gowf/testdata/dummyswift.conf): /private/tmp/dummy (0 files are shuffled)
DEBUG: 2018/07/03 16:14:44 stats.go:57: Local: 0 files
DEBUG: 2018/07/03 16:14:44 stats.go:58: Remote: 0 objects
DEBUG: 2018/07/03 16:14:44 stats.go:59: Trigger jobs: 0 by User, 0 by GoWF
DEBUG: 2018/07/03 16:14:44 stats.go:60: Jobs in queue: 0 objects
DEBUG: 2018/07/03 16:14:44 stats.go:61: Jobs in progress: 0 SLOs, 0 objects, 0 MD5 calculation
DEBUG: 2018/07/03 16:14:44 stats.go:63: Jobs uploaded: 0 successes, 0 failures, 0 disregard
type: %!s(bool=true)
type: %!s(bool=true)
--- PASS: TestRunWatchFolderTask (3.01s)
PASS
coverage: 46.7% of statements
ok github.com/swiftstack/go-watch-folder/gowf (cached) coverage: 46.7% of statements
.
.
.
=== RUN TestCompareDeduperHasAnchor
Dedup: 2018/07/03 16:15:08 deduper_test.go:75: Alloc = 102503 KiB, TotalAlloc = 4935966 KiB, Sys = 2067266 KiB, NumGC = 36 KiB
2018/07/03 16:15:10 TestCompareDeduperNonStreaming took 1.472720069s
Dedup: 2018/07/03 16:15:10 deduper_test.go:75: Alloc = 196539 KiB, TotalAlloc = 5459877 KiB, Sys = 2067266 KiB, NumGC = 50 KiB
Dedup: 2018/07/03 16:15:10 deduper_test.go:75: Alloc = 100 KiB, TotalAlloc = 5459877 KiB, Sys = 2067266 KiB, NumGC = 51 KiB
delete random write file /tmp/100MB
--- PASS: TestCompareDeduperHasAnchor (1.96s)
PASS
coverage: 94.9% of statements
ok github.com/swiftstack/go-watch-folder/dedup (cached) coverage: 94.9% of statements
3. Build binary¶
$ make binary
go-bindata -pkg main -o cmd/templates.go tmpl/*.html
GOOS=linux GOARCH=amd64 go build -ldflags "-w -s -X main.Version=0.0.8-25-g4ea76d6 -X main.UpgradeServer=https://cloud.swiftstack.com/v1/AUTH_gowfdev/" -o build/gowf.linux cmd/gowf.go
GOOS=freebsd GOARCH=amd64 go build -ldflags "-w -s -X main.Version=0.0.8-25-g4ea76d6 -X main.UpgradeServer=https://cloud.swiftstack.com/v1/AUTH_gowfdev/" -o build/gowf.freebsd cmd/gowf.go
GOOS=windows GOARCH=amd64 go build -ldflags "-w -s -X main.Version=0.0.8-25-g4ea76d6 -X main.UpgradeServer=https://cloud.swiftstack.com/v1/AUTH_gowfdev/" -o build/gowf.exe cmd/gowf.go
GOOS=linux GOARCH=amd64 go build -ldflags "-w -s -X main.Version=0.0.8-25-g4ea76d6 -X main.UpgradeServer=https://cloud.swiftstack.com/v1/AUTH_gowfdev/" -o build/gowf-web.linux cmd/gowf-web.go cmd/templates.go
GOOS=freebsd GOARCH=amd64 go build -ldflags "-w -s -X main.Version=0.0.8-25-g4ea76d6 -X main.UpgradeServer=https://cloud.swiftstack.com/v1/AUTH_gowfdev/" -o build/gowf-web.freebsd cmd/gowf-web.go cmd/templates.go
GOOS=windows GOARCH=amd64 go build -ldflags "-w -s -X main.Version=0.0.8-25-g4ea76d6 -X main.UpgradeServer=https://cloud.swiftstack.com/v1/AUTH_gowfdev/" -o build/gowf-web.exe cmd/gowf-web.go cmd/templates.go
GOOS=darwin GOARCH=amd64 go build -ldflags "-w -s -X main.Version=0.0.8-25-g4ea76d6 -X main.UpgradeServer=https://cloud.swiftstack.com/v1/AUTH_gowfdev/" -o build/gowf.darwin cmd/gowf.go
GOOS=darwin GOARCH=amd64 go build -ldflags "-w -s -X main.Version=0.0.8-25-g4ea76d6 -X main.UpgradeServer=https://cloud.swiftstack.com/v1/AUTH_gowfdev/" -o build/gowf-web.darwin cmd/gowf-web.go cmd/templates.go
3. You can find binaries in build folder¶
$ tree ${GOPATH}/src/github.com/swiftstack/go-watch-folder/build
/Users/charles/gowork/src/github.com/swiftstack/go-watch-folder/build
├── gowf-web.darwin
├── gowf-web.exe
├── gowf-web.freebsd
├── gowf-web.linux
├── gowf.darwin
├── gowf.exe
├── gowf.freebsd
└── gowf.linux
0 directories, 8 files
Configuration¶
Configuration file¶
Global¶
[global]
user = demo
auth = http://192.168.190.21/auth/v1.0
key = demo
concurrency = 10
# default segment_size is 10MB
segment_size = 10485760
segment_container_prefix = .segment_
recursive = True
preserve_path = True
user¶
User name for obtaining an auth token.
auth¶
URL for obtaining an auth token.
key¶
Key for obtaining an auth token.
concurrency¶
It allows watch folder spawns how many threads to handle the upload jobs in the waiting queue. The default value is10
.
segment_size¶
The segments size for Static Large Object(SLO). The default value is10485760
(10MB
)
segment_container_prefix¶
The prefix string of segment container when gowf creates new segment container, the default value is.segment_
recursive¶
Watch subdirectories for file changes if setTrue
. The default value isFalse
.
preserve_path¶
Constructing an object name (include the relative path) when it is True. The default value is False
For example, if you have a file called woof.txt
, and it is under the folder /<the_folder_you_watched>/dog/staffy/`
, the relative path of the file is dog/staffy/woof.txt
, so the object name in Swift with be `dog/staffy/woof.txt
.
When False
, objects will always be named after the basename of the source file (eg, woof.txt
).
If you have some files have the same name (0ab5.db
, 53ef.db
, … etc) in different subfolders (`312/0ab5
,``435/0ab5``) in below,
└── subfolder └── 38965 ├── 312 │ ├── 0ab5 │ │ └── 0ab5.db │ ├── 53ef │ │ └── 53ef.db │ ├── 9c3a │ │ └── 9c3a.db │ └── f19ed │ └── f19ed.db └── 435 ├── 0ab5 │ └── 0ab5.db ├── 53ef │ └── 53ef.db ├── 9c3a │ └── 9c3a.db └── f19ed └── f19ed.db
When you set recurisive = True
and preserve_path = False
, watch folder tries to overwrite the target object in remote (0ab5.db
) when one of local files is updated (subfolder/38965/312/0ab5/0ab5.db
or subfolder/435/312/0ab5/0ab5.db
). So the remote object will be the one is updated recently, please be aware this use case.
Folder section¶
[/tmp/b1]
storage_policy = Standard-Replica
container = b1RR
# segment_container = b1RR+mysegment
# split with comma
file_patterns = *.txt, *.log
# expired remote object in `expired_after`
# y: year, w: week, d: day, h: hour, m: minute, s: second
expired_after = 60s
# Metadata for objects
metadata = key1:val1, key2:val2
# dedup parameters
anchor = 0
anchor_upper_bound = 536870912
anchor_lower_bound = 1048576
anchor_divider = 128
min_divider = 2
max_multiplier = 2
buffer_read_gate = 536870912
buffer_read_size = 1073741824
storage_policy¶
The option allows you to create containers and segment containers under the specific storage policy. Default value is ` ` (Empty) and it’ll use the default policy that defines in the Swift cluster.
Note:
If the container or segment container is exist and has a different storage policy, the watch folder won’t upload any objects until you correct the policy in config file or use a new container/segment container.You’ll see some errors in below,
2017-08-28 08:33:01,502 - watch-uploader - INFO - Created Container(post_container) testSWF with Standard-Replica policy 2017-08-28 08:33:01,502 - watch-uploader - INFO - Created Container(post_container) testSWF with Standard-Replica policy 2017-08-28 08:33:01,671 - watch-uploader - INFO - Get(stat_container) container (testSWF_seg), policy: Reduced-Redundancy 2017-08-28 08:33:01,671 - watch-uploader - ERROR - Current container(testSWF_seg) policy(Reduced-Redundancy) is mismatch! (Standard-Replica)
container¶
The remote container name.
segment_container¶
The remote segment container name.
file_patterns¶
Only upload files that matched the patterns, split with comma
file_patterns = *.log.gz file_patterns = *.gz,*.zip
expired_after¶
When you set this option in the folder stion, the uploaded files under this folder will expired after the value you set. If you want to keep the upload files forever, please remove this optoin under the folder secion.
# 300 seconds expired_after = 300s # 5 minutes expired_after = 5m # 24 hours expired_after = 24h # 90 days expired_after = 90d # 7 years expired_after = 7y # 2 dyas 10 hours expired_after = 2d10h
metadata¶
You can add metadata for each files that you want to upload from this folder.
metadata = key-1:value-1,key-2:value-2
Run¶
# Run with single configuration file
$ gowf -config gowf.conf
$ gowf -config gowf.conf -config gowf-2nd.conf
# run it in debug mode (turn on log-level to debug)
$ gowf -config gowf.conf -log-file log.txt -log-level DEBUG
# run it in debug mode and store statistic report in stats folder.
$ gowf -config gowf.conf \
-log-file log.txt \
-log-level DEBUG \
-stats-folder stats
Help message¶
$ ./gowf -h
Usage of ./gowf:
-config value
A config file. To run multiple config files, please try '-config <config1> -config <config2>'
-dedup
Enable dedup feature for large object
** THIS IS AN EXPERIMENTAL FEATURE, PLEASE CONTACT SWIFTSTACK FIRST **
-log-file string
Log file (default "gowf.log")
-log-level string
Log level (DEBUG or INFO) (default "INFO")
-stats-folder string
Store statistic output in the folder
-update
Update to the latest version
-version
Version
Development¶
Build documentation¶
### If you don't have sphinx installed on your Mac,
### please install that first
$ brew install sphinx
$ make doc
You should able to find this documentation under docs/build/html/
.
Unit tests¶
$ make test
### Check code coverage
$ go tool cover -html=coverage.gowf.out
$ go tool cover -html=coverage.dedup.out
Run tests with Gitlab runner and docker¶
Install gitlab runner and docker¶
Gitlab runner for Mac
Docker for Mac
Run unit and build tests¶
$ make gitlab
gitlab-runner exec docker format
WARNING: Since GitLab Runner 10.0 this command is marked as DEPRECATED and will be removed in one of upcoming releases
WARNING: You most probably have uncommitted changes.
WARNING: These changes will not be tested.
Running with gitlab-runner 10.4.0 (857480b6)
on ()
Using Docker executor with image golang:1.9.2 ...
Using docker image sha256:9427850b2591293a89c98ac80a9f8a67358998ea71d40fb2064c16ed7228c96d for predefined container...
Pulling docker image golang:1.9.2 ...
.
.
.
GOOS=linux GOARCH=amd64 go build -ldflags "-w -s -X main.Version=0.0.8-26-gd1389df -X main.UpgradeServer=https://cloud.swiftstack.com/v1/AUTH_gowfdev/" -o /builds/project-0/build/gowf.linux cmd/gowf.go
GOOS=freebsd GOARCH=amd64 go build -ldflags "-w -s -X main.Version=0.0.8-26-gd1389df -X main.UpgradeServer=https://cloud.swiftstack.com/v1/AUTH_gowfdev/" -o /builds/project-0/build/gowf.freebsd cmd/gowf.go
GOOS=windows GOARCH=amd64 go build -ldflags "-w -s -X main.Version=0.0.8-26-gd1389df -X main.UpgradeServer=https://cloud.swiftstack.com/v1/AUTH_gowfdev/" -o /builds/project-0/build/gowf.exe cmd/gowf.go
GOOS=linux GOARCH=amd64 go build -ldflags "-w -s -X main.Version=0.0.8-26-gd1389df -X main.UpgradeServer=https://cloud.swiftstack.com/v1/AUTH_gowfdev/" -o /builds/project-0/build/gowf-web.linux cmd/gowf-web.go cmd/templates.go
GOOS=freebsd GOARCH=amd64 go build -ldflags "-w -s -X main.Version=0.0.8-26-gd1389df -X main.UpgradeServer=https://cloud.swiftstack.com/v1/AUTH_gowfdev/" -o /builds/project-0/build/gowf-web.freebsd cmd/gowf-web.go cmd/templates.go
GOOS=windows GOARCH=amd64 go build -ldflags "-w -s -X main.Version=0.0.8-26-gd1389df -X main.UpgradeServer=https://cloud.swiftstack.com/v1/AUTH_gowfdev/" -o /builds/project-0/build/gowf-web.exe cmd/gowf-web.go cmd/templates.go
Running after script...
Job succeeded