Supervisord run command before stop

in linux

Supervisord requires that the programs it is configured to run don’t daemonize themselves. Instead, they should run in the foreground and respond to the stop signal (TERM by default) by properly shutting down.

Use Following shell script to wrapper your background program.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#!/bin/bash
# run something background
./run &
# get nearest running background pid
$PID=$!
# Perform program exit housekeeping
function clean_up {
  # dosomething for cleanup
  rm -rf dump_file
  # remove trap
  trap - SIGHUP SIGINT SIGTERM SIGQUIT SIGKILL
  exit $?
}
# Add trap for catch sistem signal
trap clean_up SIGHUP SIGINT SIGTERM SIGQUIT SIGKILL
wait $PID

Comment and share

prometheus筆記

in monitor

Histograms vs Summaries

Histogram與Summaries都適合拿來observe request duration or response size.
使用histogram and summary時, 除了有_count與_sum的matrics以外, 還會有buckets or quantile
可以拿來計算quantiles

Comment and share

go test tool筆記

in golang

run all test except vendor

1
go test $(go list ./... | grep -v /vendor/)

go tool cover

1
2
3
go test -coverprofile coverage.txt <package name>
go tool cover -html=coverage.txt -o coverage.html
chrome coverage.html

example script to run all package test and generate coverage file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
#!/usr/bin/env bash
set -e
out="coverage.txt"
tmp="profile.coverage.out"
except="example|testproto"
: > $out
i=0
for d in $(go list ./... | grep -v vendor); do
    # except some package
    if [[ $d =~ $except ]]; then
        continue
    fi
    
    echo -e "TESTS FOR: for \033[0;35m${d}\033[0m"
    go test -v -coverprofile=$tmp $d
    if [ -f $tmp ]; then
        # remove mode:xxx if not first package for go tool cover
        if [ "$i" != 0 ]; then
            sed -i '' -e '/^mode:/d' ./$tmp
        fi
        cat $tmp >> $out
        rm $tmp
    fi
    echo ""
    ((i++))
done

Comment and share

golang test 筆記

in golang

Table Driven Test

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
func TestAdd(t * testing.T) {
  tcs := []struct{ 
    Name string 
    A, B, Expected int
  }{
    {"foo", 1, 1, 2},
    {"bar", 1, -1, 0},
  }
  for _, tc := range tcs {
    t.Run(tc.Name, func(t *testing.T) {
      actual := tc.A + tc.B
      if actual != expected {
        t.Errorf("%d + %d = %d, exected %d", tc.A, tc.B, actual, tc.Expected)
      }
    })
  }
}

Test fixtures

go test set pwd as package directory
use relative path “test-fixtures” directory as a place to store test data(loading config, model data, binary data …etc)

1
2
3
4
func TestAdd(t *testing.T) {
  data := filepath.Join("test-fixtures", "add_data.json")
  // Do something with data
}

Golden files(Test flag)

expected output放在.golden file中
當update flag為true時, 代表需要更新expected output file
Very useful to test complex structures

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
var update = flag.Bool("update", false, "update golden files")
func TestAdd(t *testing.T) {
  // ... table (probably!)
  for _, tc := range tcs {
    actual := doSomething(tc)
    golden := filepath.Join("test-fixtures", tc.Name+".golden")
    if *update {
      ioUtil.WriteFile(golden, actual, 0644)
    }
    expected, _ := ioutil.ReadFile(golden)
    if !bytes.Equal(actual, expected) {
      //FAIL!
    }
  }
}

1
2
go test
go test -update

Global State

Do not use global state.
If you have to do, Use default, and let test can easily mock it.

1
2
3
4
5
const defaultPort = 1000
type ServerOpts {
  Port int // default it to defaultPort somewhere
}

Test Helpers

TestTempFile

Test helper接受testing.T參數, helper中的錯誤直接在helper中處理掉
helper return a function to clean

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
func testTempFile(t * testing.T) (string, func()) {
  t.Helper()
  tf, err := iotuil.TempFile("", "test")
  if err != nil {
    t.Fatalf("err: %s", err)
  }
  tf.Close()
  return tf.Name(), func() { os.Remove(tf.Name()) }
}
func TestThing(t *testing T) {
  tf, tfclose := testTempFile(t)
  defer tfclose()
  // doSomething with tf
}

TestChdir

切換dir並在function結束前切換回來
注意testChdir會回傳一function, 該function會將dir切回原本old
由於defer會在defer當下處理完參數所以在defer當下即會切換到dir
然後在TestThing結束前呼叫os.chdir(old)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
func testChdir(t *testing.T, dir string) func() {
  t.Helper()
  old, err := os.Getwd()
  if err != nil {
    t.Fatalf("err: %s", err)
  }
  if err := os.Chdir(dir); err != nil {
    t.Fatalf("err: %s", err)
  }
  return func() { os.Chdir(old) }
}
func TestThing(t *testing.T) {
  defer testChdir(t, "/other")()
}

Networking

不需要mock net.Conn

1
2
3
4
5
6
7
8
9
10
11
12
13
14
func TestConn(t *testing.T) (client, server net.Conn) {
  t.Helper()
  ln, err := net.Listen("tcp", "127.0.0.1:0")
  var server net.Conn
  go func() {
    defer ln.Close()
    server, err = ln.Acccept()
  }()
  client, err := net.Dial("tcp", ln.Addr().String())
  return client, server
}

Test Subprocessing

Check if git is installed. If not, skip test

1
2
3
4
5
6
7
8
9
10
11
12
13
14
var testHasGit bool
func init() {
  if _, err := exec.LookPath("git"); err == nil {
    testHasGit = true
  }
}
func TestGitGetter(t *testing.T) {
  if !testHasGit {
    t.Log("git not found, skipping")
    t.Skip()
  }
}

Mock Subprocess

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
func helperProcess(s ...string) *exec.Cmd {
  cs := []string{"-test.run=TestHelperProcess", "--"}
  cs = append(cs, s...)
  env := []string{
    "GO_WANT_HELPER_PROCESS=1"
  }
  cmd := exec.Command(os.Args[0], cs...)
  cmd.Env = append(env, os.Environ()...)
  return cmd
}
func TestHelperProcess(*testing.T) {
  if os.Getenv("GO_WANT_HELPER_PROCESS") != "1" {
    return
  }
  defer os.Exit(0)
  args := os.Args
  for len(args) > 0 {
    if args[0] == "--" {
      args = args[1:]
      break
    }
    args = args[1:]
  }
  ...
  cmd, args := args[0], args[1:]
  switch cmd {
    case "foo":
      // ...
  }
}

Testing as a public API

提供使用你的package的user一個測試你的package的public API(Maybe a mock or a helper)
test.go will be ignored by go, so use testing.go for Test for API
Example:

  • API Server
    • TestServer(t) (net.Addr, io.Closer) => Returns a fully started in-memory server and a closer to close it.
  • interface for downloading files
    • TestDownloader(t, Downloader) => Tests all the properties a downloader should have.
    • struct DownloaderMock[] => Implements Downloader as a mock

Timing-Dependent Test

timeMultiplier is configurable.
maybe lower at laptop and higher at server

1
2
3
4
5
6
7
8
9
func TestThing(t * testing.T) {
  timeout := 3 * time.Minute * timeMultiplier
  select {
  case <- thingHappened:
  case <- time.After(timeout):
    t.Fatal("timeout")
  }
}

Complex Structs Compare(maybe graph)

How to compare complex structs in test:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
type ComplexThing struct {/* ... */}
func (c *ComplexThing) testString() string {
  // produce human-friendly output for test comparison
}
//----------------------------------------
func TestComplexThing(t *testing.T) {
  c1, c2 := createComplexThings()
  if c1.testString() != c2.testString() {
    t.Fatalf("no match:\n\n%s\n\n%s", c1.testString(), c2.testString())
  }
}

Comment and share

glide筆記

in golang

https://github.com/Masterminds/glide

安裝

1
brew install glide

Usage

source code需要放在gopath下

1
2
3
4
5
6
7
$ glide create                            # Start a new workspace
$ open glide.yaml                         # and edit away!
$ glide get github.com/Masterminds/cookoo # Get a package and add to glide.yaml
$ glide install                           # Install packages and dependencies
# work, work, work
$ go build                                # Go tools work normally
$ glide up                                # Update to newest versions of the package

get 3rd party library and all their dependencies

1
2
glide get --all-dependencies <path>
glide up --all-dependencies

Comment and share

gvm筆記與vscode整合

in golang

https://github.com/moovweb/gvm

安裝

1
2
3
4
5
6
7
8
xcode-select --install
brew update
brew install mercurial
bash < <(curl -s -S -L https://raw.githubusercontent.com/moovweb/gvm/master/binscripts/gvm-installer)
gvm install go1.4 -B
gvm use go1.4
export GOROOT_BOOTSTRAP=$GOROOT
gvm install 1.8

指令

1
2
3
4
5
6
gvm install [version]
gvm uninstall [version]
gvm listall
gvm list
gvm use [version]
gvm implode    // uninstall gvm

使用gvm來管理workspace

1
2
3
4
5
6
7
go use [version]
mkdir -p ~/golang/  
cd ~/golang/  
gvm pkgset create --local  
gvm pkgset use --local  
mkdir src pkg bin
go env

利用pkgset將GOPATH設為自定的workspace path

vscode 整合

設定gopath與goroot為go env的值

1
2
go use [version]
go env

1
2
3
4
{
  "go.gopath":"/Users/meep007/cwz/develop/practice/golang:/Users/meep007/cwz/develop/practice/golang/.gvm_local/pkgsets/go1.8/local:/Users/meep007/.gvm/pkgsets/go1.8/global",
  "go.goroot": "/Users/meep007/.gvm/gos/go1.8"
}

Comment and share

nginx connection limit bandwith limit rate limit

in web

connection limit

每一ip最多50連線數

1
2
3
4
5
6
7
8
9
limit_conn_zone $binary_remote_addr zone=connlimit:10m;
server {
    # ...
    location / {
      limit_conn connlimit 50;
      # ...
    }
    # ...
 }

bandwith limit

當流量超過500k時限制速度為50k

1
2
3
4
5
6
7
8
9
10
limit_conn_zone $binary_remote_addr zone=connlimit:10m;
server {
    # ...
    location / {
      limit_rate 50k;
      limit_rate_after 500k;
      # ...
    }
 }

rate limit

每秒最多50 request

1
2
3
4
5
6
7
8
9
10
limit_req_zone $binary_remote_addr zone=one:10m rate=50r/s;
server {
    # ...
    location / {
      limit_req zone=one burst=5 nodelay;
      # ...
    }
    # ...
 }

Comment and share

nginx and ufw add rate limit

in web

production環境下需要設定rate limit來避免DOS攻擊
以下介紹兩種設定方式，一種是讓ap server來擋(這裡介紹nginx)，一種是讓系統防火墻來擋(這裡介紹ufw)

Continue reading

Cwza

Hello everyone.
I’m cwza.
Welcome to my blog.


Software Engineer


Taiwan/Taipei