Postgres on Btrfs

Postgres performs random writes on big database files, which can cause huge performance impact on files systems featured with COW(Copy On Write) like btrfs and zfs.

A good practice is add "nodatacow" option to database volume in /etc/fstab

Run

mount <mount-point> -o remount

to apply the edited mount options.

Ram disk on OS X

Create a 1.5G ram disk at /Volumes/ramdisk

$ diskutil erasevolume HFS+ "ramdisk" `hdiutil attach -nomount ram://3145728`

http://apple.stackexchange.com/questions/55794/why-do-mac-os-x-ramdisks-appear-to-be-limited-to-550mb-and-how-can-i-change-this

Find the original user of a process invoked by sudo

Let's say you want to find out who kicked off the most memory consuming process using root.

1) open top.
2) sort by mem by pressing >
3) write down the pid
4) sudo vim /proc/<pid>/environ , you will find the the user in: SUDO_USER='\<user\>'

There are some other ways: http://unix.stackexchange.com/questions/7334/using-top-to-see-processes-run-by-a-user-on-behalf-of-sudo

Join odd and even numbered lines

sed:

sed '$!N;s/\n/,/'


awk:

awk 'NR%2==0 {print p","$0;} NR%2 {p=$0;}'


paste:

paste - -


--------------------------
Seems  I like paste most. To join 3 consecutive lines:

paste - - -

Python Profiler

def do_cprofile(func):
    def profiled_func(*args, **kwargs):
        import cProfile
        profile = cProfile.Profile()
        try:
            profile.enable()
            result = func(*args, **kwargs)
            profile.disable()
            return result
        finally:
            profile.print_stats(sort=1)
    return profiled_func

try:
    from line_profiler import LineProfiler

    def do_profile(follow=[]):
        def inner(func):
            def profiled_func(*args, **kwargs):
                try:
                    profiler = LineProfiler()
                    profiler.add_function(func)
                    for f in follow:
                        profiler.add_function(f)
                    profiler.enable_by_count()
                    return func(*args, **kwargs)
                finally:
                    profiler.print_stats()
            return profiled_func
        return inner

except ImportError:
    def do_profile(follow=[]):
        "Helpful if you accidentally leave in production!"
        def inner(func):
            def nothing(*args, **kwargs):
                return func(*args, **kwargs)
            return nothing
        return inner


@do_profile(follow=[fun2])
def fun1():
    fun2()
    pass


@do_cprofile()
def fun2()
    pass

Java cpu profiling with hprof

Set JAVA_OPTS:

-Xrunhprof:cpu=samples,interval=10,depth=8

There will be a  file called "java.hprof.txt" in your working directory 

No syntax highlighting in vimdiff

Vim Syntax highlight often mixed with diff color, which makes text unreadable.

Add following line in ~/.vimrc to turn off syntax off in vimdiff

if &diff | syntax off | endif

IO Redirection - Swapping stdout and stderr (Advanced)

% (sh myscript.sh 3>&2 2>&1 1>&3) 2>/dev/null
I'm stderr
% (sh myscript.sh 3>&2 2>&1 1>&3) >/dev/null 
I'm stdout

python 3 line to solve 8 queen

from itertools import *
c = range(8)
print len([v for v in permutations(c) if 8==len(set(v[i]+i for i in c))==len(set(v[i]-i for i in c))])

Python overwrite printed line, e.g. text progress

\r will reset the cursor to the beginning of the line.

>>> for i in range(100):
...    time.sleep(1)
...    sys.stdout.write("\r%d%%" %i)    # or print >> sys.stdout, "\r%d%%" %i,
...    sys.stdout.flush()
... 

Benchmark CPU in SHELL

$ time echo "scale=5000; a(1)*4" | bc -l

brackets, parentheses, curly braces in BASH

In Bash, test and [ are builtins.

The double bracket enables additional functionality. For example, you can use && and || instead of-a and -o and there's a regular expression matching operator =~.

The braces, in addition to delimiting a variable name are used for parameter expansion so you can do things like:

• Truncate the contents of a variable
  $ var="abcde"; echo ${var%d*}
abc
• Make substitutions similar to sed
  $ var="abcde"; echo ${var/de/12}
abc12
• Use a default value
  $ default="hello"; unset var; echo ${var:-$default}
hello
• and several more

Vim folding, auto fold xml, fold, unfold

To allow folds based on syntax add something like the following to your .vimrc:

set foldmethod=syntax
set foldlevelstart=1

let javaScript_fold=1         " JavaScript
let perl_fold=1               " Perl
let php_folding=1             " PHP
let r_syntax_folding=1        " R
let ruby_fold=1               " Ruby
let sh_fold_enabled=1         " sh
let vimsyn_folding='af'       " Vim script
let xml_syntax_folding=1      " XML




zo -- To open a fold block

zc -- To fold a block

zr -- Fold less

zm -- Fold more

Python multiple thread processing

from multiprocessing import Pool
pool = Pool(processes=5)
pages = pool.map(visit, get_lines(file))

What every programmer should know…

What Every Computer Scientist Should Know About Floating-Point Arithmetic

What Every Programmer Should Know About Memory

What every programmer absolutely, positively needs to know about encodings and character sets to work with text

Latency Numbers Every Programmer Should Know

What every programmer should know about time

What Every Programmer Should Know About Races

What Every C Programmer Should Know About Undefined Behavior

What every data programmer needs to know about disks

Above contents are from http://wangcong.org/blog/archives/2255

List recently modified files under a directory in linux shell

In order to list files that have been modified recently, we could use the find command to retrieve the file information and sort them by modified date:

find . -type f -exec stat --format '%Y :%y %n' {} \; | sort -nr | cut -d: -f2- | head

However, this will miss some folders if the folders are symoblic links. So in this case we could specify find to follow symbolic links.

find -L . -type f -exec stat --format '%Y :%y %n' {} \; | sort -nr | cut -d: -f2- | head

Fourther more, if you just want to get files modified in last a few days, it's build in in find:

find . -mtime n

list files that modified n*24 hours ago.

The difference between soft link and hard link in linux

From: http://linuxgazette.net/105/pitcher.html

Unix files consist of two parts: the data part and the filename part.

The data part is associated with something called an 'inode'. The inode carries the map of where the data is, the file permissions, etc. for the data.

                               .---------------> ! data ! ! data ! etc
                              /                  +------+ !------+
        ! permbits, etc ! data addresses !
        +------------inode---------------+

The filename part carries a name and an associated inode number.

                         .--------------> ! permbits, etc ! addresses !
                        /                 +---------inode-------------+
        ! filename ! inode # !
        +--------------------+

More than one filename can reference the same inode number; these files are said to be 'hard linked' together.

        ! filename ! inode # !
        +--------------------+
                        \
                         >--------------> ! permbits, etc ! addresses !
                        /                 +---------inode-------------+
        ! othername ! inode # !
        +---------------------+

On the other hand, there's a special file type whose data part carries a path to another file. Since it is a special file, the OS recognizes the data as a path, and redirects opens, reads, and writes so that, instead of accessing the data within the special file, they access the data in the file named by the data in the special file. This special file is called a 'soft link' or a 'symbolic link' (aka a 'symlink').

python program behind proxy

export http_proxy="http://localhost:8118"

All about pip

I was stucked in pip. Just found out that pip is called "pip-python" in RHEL.

 What's more, pip cannot auto detect system proxy settings, manual specify:

 sudo pip sudo pip-python install virtualenv --proxy localhost:8118

command sort on last column

As you know, the linux sort command is agile, which can sort on column using -k.

However, if we gonna sort on the last column and each row has different number of columns, we can still achieve that with the help of sed.

sed 's/^\(.\+\s\+\)\([0-9]\+ *\)$/\2 \1/'