Linux : perf vs strace output

# perf trace ls
     0.171 ( 0.023 ms): ls/29642 brk(                                                                  ) = 0x55ad369bc000
     0.220 ( 0.013 ms): ls/29642 mmap(len: 4096, prot: READ|WRITE, flags: PRIVATE|ANONYMOUS, fd: -1    ) = 0x7f15d9245000
     0.245 ( 0.013 ms): ls/29642 access(filename: 0x7f15d9043e60, mode: R                              ) = -1 ENOENT No such file or directory
     0.271 ( 0.013 ms): ls/29642 open(filename: 0x7f15d90427af, flags: CLOEXEC                         ) = 3
     0.291 ( 0.010 ms): ls/29642 fstat(fd: 3</usr/lib64/libselinux.so.1>, statbuf: 0x7fffdcb20b90      ) = 0
     0.312 ( 0.011 ms): ls/29642 mmap(len: 41110, prot: READ, flags: PRIVATE, fd: 3</usr/lib64/libselinux.so.1>) = 0x7f15d923a000
     0.332 ( 0.010 ms): ls/29642 close(fd: 3</usr/lib64/libselinux.so.1>                               ) = 0
     0.367 ( 0.016 ms): ls/29642 open(filename: 0x7f15d9245640, flags: CLOEXEC                         ) = 3
     2.330 ( 0.177 ms): ls/29642 read(fd: 3</usr/lib64/libselinux.so.1>, buf: 0x7fffdcb20d20, count: 832) = 832
     2.529 ( 0.020 ms): ls/29642 fstat(fd: 3</usr/lib64/libselinux.so.1>, statbuf: 0x7fffdcb20bd0      ) = 0
     2.768 ( 0.028 ms): ls/29642 mmap(len: 2237248, prot: EXEC|READ, flags: PRIVATE|DENYWRITE, fd: 3</usr/lib64/libselinux.so.1>) = 0x7f15d8e03000
     3.129 ( 0.157 ms): ls/29642 mprotect(start: 0x7f15d8e22000, len: 2097152                          ) = 0
     3.347 ( 0.031 ms): ls/29642 mmap(addr: 0x7f15d9022000, len: 8192, prot: READ|WRITE, flags: PRIVATE|DENYWRITE|FIXED, fd: 3</usr/lib64/libselinux.so.1>, off: 126976) = 0x7f15d9022000
     3.607 ( 0.026 ms): ls/29642 mmap(addr: 0x7f15d9024000, len: 4928, prot: READ|WRITE, flags: PRIVATE|ANONYMOUS|FIXED, fd: -1) = 0x7f15d9024000
     3.838 ( 0.021 ms): ls/29642 close(fd: 3</usr/lib64/libselinux.so.1>                               ) = 0
     4.087 ( 0.028 ms): ls/29642 open(filename: 0x7f15d9245b10, flags: CLOEXEC                         ) = 3
     4.365 ( 0.078 ms): ls/29642 read(fd: 3</usr/lib64/libcap.so.2.24>, buf: 0x7fffdcb20cf0, count: 832) = 832
     4.567 ( 0.078 ms): ls/29642 fstat(fd: 3</usr/lib64/libcap.so.2.24>, statbuf: 0x7fffdcb20ba0       ) = 0
     4.756 ( 0.066 ms): ls/29642 mmap(len: 2113840, prot: EXEC|READ, flags: PRIVATE|DENYWRITE, fd: 3</usr/lib64/libcap.so.2.24>) = 0x7f15d8bfe000
     4.944 ( 0.037 ms): ls/29642 mprotect(start: 0x7f15d8c02000, len: 2093056                          ) = 0
     5.241 ( 0.107 ms): ls/29642 mmap(addr: 0x7f15d8e01000, len: 8192, prot: READ|WRITE, flags: PRIVATE|DENYWRITE|FIXED, fd: 3</usr/lib64/libcap.so.2.24>, off: 12288) = 0x7f15d8e01000
     5.456 ( 0.078 ms): ls/29642 close(fd: 3</usr/lib64/libcap.so.2.24>                                ) = 0
     5.568 ( 0.027 ms): ls/29642 open(filename: 0x7f15d9245fe0, flags: CLOEXEC                         ) = 3
     5.715 ( 0.073 ms): ls/29642 read(fd: 3</usr/lib64/libc-2.22.so>, buf: 0x7fffdcb20cc0, count: 832  ) = 832
     5.856 ( 0.028 ms): ls/29642 fstat(fd: 3</usr/lib64/libc-2.22.so>, statbuf: 0x7fffdcb20b70         ) = 0
     6.043 ( 0.085 ms): ls/29642 mmap(len: 4096, prot: READ|WRITE, flags: PRIVATE|ANONYMOUS, fd: -1    ) = 0x7f15d9239000
     6.217 ( 0.036 ms): ls/29642 mmap(len: 3934784, prot: EXEC|READ, flags: PRIVATE|DENYWRITE, fd: 3</usr/lib64/libc-2.22.so>) = 0x7f15d883d000
     6.413 ( 0.088 ms): ls/29642 mprotect(start: 0x7f15d89f4000, len: 2097152                          ) = 0
     6.562 ( 0.041 ms): ls/29642 mmap(addr: 0x7f15d8bf4000, len: 24576, prot: READ|WRITE, flags: PRIVATE|DENYWRITE|FIXED, fd: 3</usr/lib64/libc-2.22.so>, off: 1798144) = 0x7f15d8bf4000
     6.802 ( 0.113 ms): ls/29642 mmap(addr: 0x7f15d8bfa000, len: 14912, prot: READ|WRITE, flags: PRIVATE|ANONYMOUS|FIXED, fd: -1) = 0x7f15d8bfa000
     6.964 ( 0.032 ms): ls/29642 close(fd: 3</usr/lib64/libc-2.22.so>                                  ) = 0
     7.210 ( 0.129 ms): ls/29642 open(filename: 0x7f15d92394b8, flags: CLOEXEC                         ) = 3
     7.358 ( 0.045 ms): ls/29642 read(fd: 3</usr/lib64/libpcre.so.1.2.5>, buf: 0x7fffdcb20b70, count: 832) = 832
     7.462 ( 0.023 ms): ls/29642 fstat(fd: 3</usr/lib64/libpcre.so.1.2.5>, statbuf: 0x7fffdcb20a20     ) = 0
     7.628 ( 0.031 ms): ls/29642 mmap(len: 2552072, prot: EXEC|READ, flags: PRIVATE|DENYWRITE, fd: 3</usr/lib64/libpcre.so.1.2.5>) = 0x7f15d85cd000
     7.763 ( 0.030 ms): ls/29642 mprotect(start: 0x7f15d863b000, len: 2097152                          ) = 0
     7.934 ( 0.031 ms): ls/29642 mmap(addr: 0x7f15d883b000, len: 8192, prot: READ|WRITE, flags: PRIVATE|DENYWRITE|FIXED, fd: 3</usr/lib64/libpcre.so.1.2.5>, off: 450560) = 0x7f15d883b000
     8.089 ( 0.069 ms): ls/29642 close(fd: 3</usr/lib64/libpcre.so.1.2.5>                              ) = 0
     8.242 ( 0.031 ms): ls/29642 open(filename: 0x7f15d9239988, flags: CLOEXEC                         ) = 3
     8.354 ( 0.034 ms): ls/29642 read(fd: 3</usr/lib64/libdl-2.22.so>, buf: 0x7fffdcb20b40, count: 832 ) = 832
     8.501 ( 0.021 ms): ls/29642 fstat(fd: 3</usr/lib64/libdl-2.22.so>, statbuf: 0x7fffdcb209f0        ) = 0
     8.598 ( 0.026 ms): ls/29642 mmap(len: 2109712, prot: EXEC|READ, flags: PRIVATE|DENYWRITE, fd: 3</usr/lib64/libdl-2.22.so>) = 0x7f15d83c9000
     8.766 ( 0.027 ms): ls/29642 mprotect(start: 0x7f15d83cc000, len: 2093056                          ) = 0
     8.908 ( 0.070 ms): ls/29642 mmap(addr: 0x7f15d85cb000, len: 8192, prot: READ|WRITE, flags: PRIVATE|DENYWRITE|FIXED, fd: 3</usr/lib64/libdl-2.22.so>, off: 8192) = 0x7f15d85cb000
     9.020 ( 0.022 ms): ls/29642 close(fd: 3</usr/lib64/libdl-2.22.so>                                 ) = 0
     9.230 ( 0.129 ms): ls/29642 open(filename: 0x7f15d9239eb0, flags: CLOEXEC                         ) = 3
     9.424 ( 0.052 ms): ls/29642 read(fd: 3</usr/lib64/libattr.so.1.1.0>, buf: 0x7fffdcb20ae0, count: 832) = 832
     9.634 ( 0.042 ms): ls/29642 fstat(fd: 3</usr/lib64/libattr.so.1.1.0>, statbuf: 0x7fffdcb20990     ) = 0
     9.799 ( 0.026 ms): ls/29642 mmap(len: 4096, prot: READ|WRITE, flags: PRIVATE|ANONYMOUS, fd: -1    ) = 0x7f15d9238000
     9.947 ( 0.072 ms): ls/29642 mmap(len: 2117648, prot: EXEC|READ, flags: PRIVATE|DENYWRITE, fd: 3</usr/lib64/libattr.so.1.1.0>) = 0x7f15d81c3000
    10.055 ( 0.025 ms): ls/29642 mprotect(start: 0x7f15d81c7000, len: 2097152                          ) = 0
    10.206 ( 0.030 ms): ls/29642 mmap(addr: 0x7f15d83c7000, len: 4096, prot: READ|WRITE, flags: PRIVATE|DENYWRITE|FIXED, fd: 3</usr/lib64/libattr.so.1.1.0>, off: 16384) = 0x7f15d83c7000
    10.309 ( 0.024 ms): ls/29642 mmap(addr: 0x7f15d83c8000, len: 16, prot: READ|WRITE, flags: PRIVATE|ANONYMOUS|FIXED, fd: -1) = 0x7f15d83c8000
    10.455 ( 0.066 ms): ls/29642 close(fd: 3</usr/lib64/libattr.so.1.1.0>                              ) = 0
    10.568 ( 0.027 ms): ls/29642 open(filename: 0x7f15d9238518, flags: CLOEXEC                         ) = 3
    10.721 ( 0.075 ms): ls/29642 read(fd: 3</usr/lib64/libpthread-2.22.so>, buf: 0x7fffdcb20ab0, count: 832) = 832
    10.823 ( 0.020 ms): ls/29642 fstat(fd: 3</usr/lib64/libpthread-2.22.so>, statbuf: 0x7fffdcb20960   ) = 0
    10.970 ( 0.076 ms): ls/29642 mmap(len: 2213040, prot: EXEC|READ, flags: PRIVATE|DENYWRITE, fd: 3</usr/lib64/libpthread-2.22.so>) = 0x7f15d7fa6000
    11.079 ( 0.025 ms): ls/29642 mprotect(start: 0x7f15d7fbe000, len: 2093056                          ) = 0
    11.226 ( 0.030 ms): ls/29642 mmap(addr: 0x7f15d81bd000, len: 8192, prot: READ|WRITE, flags: PRIVATE|DENYWRITE|FIXED, fd: 3</usr/lib64/libpthread-2.22.so>, off: 94208) = 0x7f15d81bd000
    11.335 ( 0.026 ms): ls/29642 mmap(addr: 0x7f15d81bf000, len: 13488, prot: READ|WRITE, flags: PRIVATE|ANONYMOUS|FIXED, fd: -1) = 0x7f15d81bf000
    11.481 ( 0.066 ms): ls/29642 close(fd: 3</usr/lib64/libpthread-2.22.so>                            ) = 0
    11.596 ( 0.024 ms): ls/29642 mmap(len: 4096, prot: READ|WRITE, flags: PRIVATE|ANONYMOUS, fd: -1    ) = 0x7f15d9237000
    11.761 ( 0.065 ms): ls/29642 mmap(len: 8192, prot: READ|WRITE, flags: PRIVATE|ANONYMOUS, fd: -1    ) = 0x7f15d9235000
    11.973 ( 0.131 ms): ls/29642 arch_prctl(option: 4098, arg2: 139731814012928, arg3: 139731814019072, arg4: 34, arg5: 4294967295) = 0
    12.207 ( 0.030 ms): ls/29642 mprotect(start: 0x7f15d8bf4000, len: 16384, prot: READ                ) = 0
    12.355 ( 0.027 ms): ls/29642 mprotect(start: 0x7f15d81bd000, len: 4096, prot: READ                 ) = 0
    12.515 ( 0.066 ms): ls/29642 mprotect(start: 0x7f15d83c7000, len: 4096, prot: READ                 ) = 0
    12.626 ( 0.025 ms): ls/29642 mprotect(start: 0x7f15d85cb000, len: 4096, prot: READ                 ) = 0
    12.806 ( 0.069 ms): ls/29642 mprotect(start: 0x7f15d883b000, len: 4096, prot: READ                 ) = 0
    12.924 ( 0.027 ms): ls/29642 mprotect(start: 0x7f15d8e01000, len: 4096, prot: READ                 ) = 0
    13.142 ( 0.100 ms): ls/29642 mprotect(start: 0x7f15d9022000, len: 4096, prot: READ                 ) = 0
    13.283 ( 0.026 ms): ls/29642 mprotect(start: 0x55ad3550b000, len: 8192, prot: READ                 ) = 0
    13.434 ( 0.070 ms): ls/29642 mprotect(start: 0x7f15d9246000, len: 4096, prot: READ                 ) = 0
    13.621 ( 0.108 ms): ls/29642 munmap(addr: 0x7f15d923a000, len: 41110                               ) = 0
    13.830 ( 0.113 ms): ls/29642 set_tid_address(tidptr: 0x7f15d9235ad0                                ) = 29642
    13.979 ( 0.066 ms): ls/29642 set_robust_list(head: 0x7f15d9235ae0, len: 24                         ) = 0
    14.187 ( 0.081 ms): ls/29642 rt_sigaction(sig: 0x20, act: 0x7fffdcb21270, sigsetsize: 8            ) = 0
    14.322 ( 0.021 ms): ls/29642 rt_sigaction(sig: 0x21, act: 0x7fffdcb21270, sigsetsize: 8            ) = 0
    14.514 ( 0.117 ms): ls/29642 rt_sigprocmask(how: UNBLOCK, nset: 0x7fffdcb213e8, sigsetsize: 8      ) = 0
    14.684 ( 0.066 ms): ls/29642 getrlimit(resource: STACK, rlim: 0x7fffdcb213d0                       ) = 0
    14.916 ( 0.077 ms): ls/29642 statfs(pathname: 0x7f15d8e1c762, buf: 0x7fffdcb213a0                  ) = 0
    15.025 ( 0.024 ms): ls/29642 statfs(pathname: 0x7f15d8e1c762, buf: 0x7fffdcb21290                  ) = 0
    15.228 ( 0.101 ms): ls/29642 brk(                                                                  ) = 0x55ad369bc000
    15.330 ( 0.022 ms): ls/29642 brk(brk: 0x55ad369dd000                                               ) = 0x55ad369dd000
    15.564 ( 0.158 ms): ls/29642 access(filename: 0x7f15d8e1c249                                       ) = 0
    15.690 ( 0.027 ms): ls/29642 open(filename: 0x7f15d89c8f80, flags: CLOEXEC                         ) = 3
    15.856 ( 0.093 ms): ls/29642 fstat(fd: 3</usr/lib/locale/locale-archive>, statbuf: 0x7f15d8bf99a0  ) = 0
    15.963 ( 0.026 ms): ls/29642 mmap(len: 110439344, prot: READ, flags: PRIVATE, fd: 3</usr/lib/locale/locale-archive>) = 0x7f15d1653000
    16.118 ( 0.078 ms): ls/29642 close(fd: 3</usr/lib/locale/locale-archive>                           ) = 0
    16.348 ( 0.093 ms): ls/29642 ioctl(fd: 1</dev/pts/2>, cmd: TCGETS, arg: 0x7fffdcb20fe0             ) = 0
    16.504 ( 0.071 ms): ls/29642 ioctl(fd: 1</dev/pts/2>, cmd: TIOCGWINSZ, arg: 0x7fffdcb210c0         ) = 0
    16.679 ( 0.080 ms): ls/29642 stat(filename: 0x55ad369c1c20, statbuf: 0x7fffdcb20fc0                ) = 0
    16.829 ( 0.069 ms): ls/29642 open(filename: 0x55ad369c1c20, flags: CLOEXEC|DIRECTORY|NONBLOCK      ) = 3
    16.938 ( 0.029 ms): ls/29642 fstat(fd: 3</sys/devices/pci0000:00/0000:00:03.0/net/enp0s3/queues/rx-0>, statbuf: 0x7fffdcb20fc0) = 0
    17.133 ( 0.118 ms): ls/29642 getdents(fd: 3</sys/devices/pci0000:00/0000:00:03.0/net/enp0s3/queues/rx-0>, dirent: 0x55ad369c1c70, count: 32768) = 112
    17.274 ( 0.023 ms): ls/29642 getdents(fd: 3</sys/devices/pci0000:00/0000:00:03.0/net/enp0s3/queues/rx-0>, dirent: 0x55ad369c1c70, count: 32768) = 0
    17.414 ( 0.064 ms): ls/29642 close(fd: 3</sys/devices/pci0000:00/0000:00:03.0/net/enp0s3/queues/rx-0>) = 0
    17.531 ( 0.022 ms): ls/29642 fstat(fd: 1</dev/pts/2>, statbuf: 0x7fffdcb1eb20                      ) = 0
    17.670 ( 0.066 ms): ls/29642 mmap(len: 4096, prot: READ|WRITE, flags: PRIVATE|ANONYMOUS, fd: -1    ) = 0x7f15d9244000
rps_cpus  rps_flow_cnt
    18.008 ( 0.254 ms): ls/29642 write(fd: 1</dev/pts/2>, buf: 0x7f15d9244000, count: 23               ) = 23
    18.178 ( 0.088 ms): ls/29642 close(fd: 1</dev/pts/2>                                               ) = 0
    18.282 ( 0.025 ms): ls/29642 munmap(addr: 0x7f15d9244000, len: 4096                                ) = 0
    18.428 ( 0.067 ms): ls/29642 close(fd: 2                                                           ) = 0
    18.544 ( 0.000 ms): ls/29642 exit_group(                                  

compare with strace

# strace ls

execve("/usr/bin/ls", ["ls"], [/* 26 vars */]) = 0

brk(NULL)                               = 0x556341408000

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1316109000

access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)

open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3

fstat(3, {st_mode=S_IFREG|0644, st_size=41110, ...}) = 0

mmap(NULL, 41110, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f13160fe000

close(3)                                = 0

open("/lib64/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3

read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260b\0\0\0\0\0\0"..., 832) = 832

fstat(3, {st_mode=S_IFREG|0755, st_size=136600, ...}) = 0

mmap(NULL, 2237248, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f1315cc7000

mprotect(0x7f1315ce6000, 2097152, PROT_NONE) = 0

mmap(0x7f1315ee6000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1f000) = 0x7f1315ee6000

mmap(0x7f1315ee8000, 4928, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f1315ee8000

close(3)                                = 0

open("/lib64/libcap.so.2", O_RDONLY|O_CLOEXEC) = 3

read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0p\26\0\0\0\0\0\0"..., 832) = 832

fstat(3, {st_mode=S_IFREG|0755, st_size=18608, ...}) = 0

mmap(NULL, 2113840, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f1315ac2000

mprotect(0x7f1315ac6000, 2093056, PROT_NONE) = 0

mmap(0x7f1315cc5000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3000) = 0x7f1315cc5000

close(3)                                = 0

open("/lib64/libc.so.6", O_RDONLY|O_CLOEXEC) = 3

read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\6\2\0\0\0\0\0"..., 832) = 832

fstat(3, {st_mode=S_IFREG|0755, st_size=2102288, ...}) = 0

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f13160fd000

mmap(NULL, 3934784, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f1315701000

mprotect(0x7f13158b8000, 2097152, PROT_NONE) = 0

mmap(0x7f1315ab8000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1b7000) = 0x7f1315ab8000

mmap(0x7f1315abe000, 14912, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f1315abe000

close(3)                                = 0

open("/lib64/libpcre.so.1", O_RDONLY|O_CLOEXEC) = 3

read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\360\26\0\0\0\0\0\0"..., 832) = 832

fstat(3, {st_mode=S_IFREG|0755, st_size=459240, ...}) = 0

mmap(NULL, 2552072, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f1315491000

mprotect(0x7f13154ff000, 2097152, PROT_NONE) = 0

mmap(0x7f13156ff000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6e000) = 0x7f13156ff000

close(3)                                = 0

open("/lib64/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3

read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0`\16\0\0\0\0\0\0"..., 832) = 832

fstat(3, {st_mode=S_IFREG|0755, st_size=19344, ...}) = 0

mmap(NULL, 2109712, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f131528d000

mprotect(0x7f1315290000, 2093056, PROT_NONE) = 0

mmap(0x7f131548f000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x2000) = 0x7f131548f000

close(3)                                = 0

open("/lib64/libattr.so.1", O_RDONLY|O_CLOEXEC) = 3

read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\320\23\0\0\0\0\0\0"..., 832) = 832

fstat(3, {st_mode=S_IFREG|0755, st_size=23320, ...}) = 0

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f13160fc000

mmap(NULL, 2117648, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f1315087000

mprotect(0x7f131508b000, 2097152, PROT_NONE) = 0

mmap(0x7f131528b000, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x4000) = 0x7f131528b000

mmap(0x7f131528c000, 16, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f131528c000

close(3)                                = 0

open("/lib64/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3

read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\260`\0\0\0\0\0\0"..., 832) = 832

fstat(3, {st_mode=S_IFREG|0755, st_size=148912, ...}) = 0

mmap(NULL, 2213040, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f1314e6a000

mprotect(0x7f1314e82000, 2093056, PROT_NONE) = 0

mmap(0x7f1315081000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x17000) = 0x7f1315081000

mmap(0x7f1315083000, 13488, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f1315083000

close(3)                                = 0

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f13160fb000

mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f13160f9000

arch_prctl(ARCH_SET_FS, 0x7f13160f9800) = 0

mprotect(0x7f1315ab8000, 16384, PROT_READ) = 0

mprotect(0x7f1315081000, 4096, PROT_READ) = 0

mprotect(0x7f131528b000, 4096, PROT_READ) = 0

mprotect(0x7f131548f000, 4096, PROT_READ) = 0

mprotect(0x7f13156ff000, 4096, PROT_READ) = 0

mprotect(0x7f1315cc5000, 4096, PROT_READ) = 0

mprotect(0x7f1315ee6000, 4096, PROT_READ) = 0

mprotect(0x5563407d0000, 8192, PROT_READ) = 0

mprotect(0x7f131610a000, 4096, PROT_READ) = 0

munmap(0x7f13160fe000, 41110)           = 0

set_tid_address(0x7f13160f9ad0)         = 29849

set_robust_list(0x7f13160f9ae0, 24)     = 0

rt_sigaction(SIGRTMIN, {0x7f1314e6fb50, [], SA_RESTORER|SA_SIGINFO, 0x7f1314e7a9f0}, NULL, 8) = 0

rt_sigaction(SIGRT_1, {0x7f1314e6fbe0, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7f1314e7a9f0}, NULL, 8) = 0

rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0

getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0

statfs("/sys/fs/selinux", {f_type="SELINUX_MAGIC", f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0

statfs("/sys/fs/selinux", {f_type="SELINUX_MAGIC", f_bsize=4096, f_blocks=0, f_bfree=0, f_bavail=0, f_files=0, f_ffree=0, f_fsid={0, 0}, f_namelen=255, f_frsize=4096, f_flags=4128}) = 0

brk(NULL)                               = 0x556341408000

brk(0x556341429000)                     = 0x556341429000

access("/etc/selinux/config", F_OK)     = 0

open("/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3

fstat(3, {st_mode=S_IFREG|0644, st_size=110439344, ...}) = 0

mmap(NULL, 110439344, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f130e517000

close(3)                                = 0

ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0

ioctl(1, TIOCGWINSZ, {ws_row=24, ws_col=80, ws_xpixel=0, ws_ypixel=0}) = 0

stat(".", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0

open(".", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3

fstat(3, {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0

getdents(3, /* 4 entries */, 32768)     = 112

getdents(3, /* 0 entries */, 32768)     = 0

close(3)                                = 0

fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 2), ...}) = 0

mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f1316108000

write(1, "rps_cpus  rps_flow_cnt\n", 23rps_cpus  rps_flow_cnt

) = 23

close(1)                                = 0

munmap(0x7f1316108000, 4096)            = 0

close(2)                                = 0

exit_group(0)                           = ?

+++ exited with 0 +++

Linux : Performance monitoring tool

Collect Statistics

* Sysstat package : mpstat, vmstat, pidstat, iostat
* dstat, nicstat
* sar , atop
* Percona Cacti template

Watch online
* top
* iotop
* iftop

Tracing
* perf_events
* ftrace
* perf-toold
* Flamegraph
http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html


* uptime
If load average > # of CPUs, may mean CPU saturation

* top or htop
Can miss short-lived processes (atop won't)
can consume noticeable CPU to read /proc

* ps

* vmstat
cpu section , us means user or application, sy means kernel / system call usage

* iostat
ex. iostat -xmdz 1 or iostat -x 1
very useful set of stats

* mpstat
ex mpstat -P ALL 1 (look for unbalanced workloads, hot CPUs)

* free
main memory usage
ex. free -m
buffers: block device I/O cache
cached: virtual page cache

* sar
ex. sar -n DEV 1 (network device)

Observability Tools : Intermediate
* strace (system call tracer)
ex. strace -tttT -p (#PID)
-ttt : time (us) since epoch
-T: syscall time(s)
Translate syscall args, very helpful for solving system usage issues
Currently has massive overhead (ptrace based)
Can slow the target by > 100x , so use with caution
ex. strike -Tttvfs 1024 -o /tmp/strace.out <command>

* tcpdump
Sniff network packets for post analysis
Study packet sequences with timestamps (us)
CPU overhead optimized (socket ring buffers), but can be significant. Use caution.
ex: tcpdump -s 0 -n -i ethX -w /tmp/tcpdump.pcap &

* netstat
-s : various network protocol statistics
-i : interface stats
-r : route table
-p : shows process details
-c : per second interval

* pidstat
very useful process stats. by thread, disk I/O
ex. pidstat -t 1
ex. pidstat -d 1

* swapon
show swap device usage: swapon -s

* lsof
more a debug tool, lsof shows file descriptor usage
ex. lsof -iTCP -sTCP:ESTABLISHED

* sar (and collectl, dstat, etc. )
Archive or live mode (interval [count])
logical groups: TCP, ETCP, DEV, EDEV
ex. sar -n TCP,ETCP,DEV 1

Report network statistics (sar -n)

This reports various network statistics. For example: number of packets received (transmitted) through the network card, statistics of packet failure etc.,. “1 3″ reports for every 1 seconds a total of 3 times.

sar -n KEYWORD

KEYWORD can be one of the following:

• DEV – Displays network devices vital statistics for eth0, eth1, etc.,
• EDEV – Display network device failure statistics
• NFS – Displays NFS client activities
• NFSD – Displays NFS server activities
• SOCK – Displays sockets in use for IPv4
• IP – Displays IPv4 network traffic
• EIP – Displays IPv4 network errors
• ICMP – Displays ICMPv4 network traffic
• EICMP – Displays ICMPv4 network errors
• TCP – Displays TCPv4 network traffic
• ETCP – Displays TCPv4 network errors
• UDP – Displays UDPv4 network traffic
• SOCK6, IP6, EIP6, ICMP6, UDP6 are for IPv6
• ALL – This displays all of the above information. The output will be very long.

* strace
trace system call of a running process
ex. strace -tp `pgrep [process name, eg. java] ` 2>&1 | head -100

Advanced Observability tools
Misc: ltrace, ss, iptraf, ethtool, snmpget, lldptool, iotop, blktrace, slabtop, /proc, pcstat
CPU Performance Counters: perf_events, tiptop, rdmsr
Advanced Tracers: perf_events, ftrace, eBPF, SystermTap, ktap, LTTng, dtrace4linux, sysdig

* ss
socket statistics
ex. ss -mop
ex. ss -i

* iptraf
provide histograms of different packet sizes

* iotop
block device I/O disk by process
Need kernel support enabled : CONFIG_TASK_IO_ACCOUNTING

* slabtop
Kernel slab allocator memory usage

* pcstat
Show page cache residency by file
Uses the mincore syscall. Useful for database performance analysis.

* perf_events
Provides the perf command
In Linux source code: tools/perf
Usually pkg added by linux-tools-common
Multi-tool : CPU profiling, PMC profiling, Static & dynamic tracing

* tiptop
IPC by process
Still cannot use it in clouds yet. needs PMCs enabled

* rdmsr
Model Specific Registers (MSRs), unlike PMCs, can be read by default in Xen guests

Benchmarking Tools
Multi: UnixBench, lmbench, sysbench, perf bench
FS/disk: dd, hdparm, fio
App/lib: ab, wrk, jmeter, openssl
Networking: ping, hping3, iperf, ttcp, traceroute, mtr, pchar

Disk benchmark: run iostat...
CPU benchmark: run pidstat, perf, flame graph....

Tuning Tools
Generic interfaces: sysctl, /sys
CPU/scheduler : nice, renice, taskset, ulimit, chcpu
Storage I/O: tune2fs, ionice, hdparm, blockdev
Network: ethtool, tc, ip, route
Dynamic patching: stap, kpatch

Tuning Methods :
Scientific Method:
Question
Hypothesis
Prediction
Test
Analysis

* ip route get [IP address]

ETC
system messages: dmesg
Network interface config: ifconfig -a , ip link
File system capacity: df -h
Volume config : mdadm --misc -D /dev/md0
Storage device info : smartctl
NUMA config: numactl -s , numactl -H
PCI info : lspci
Installed kernel modules: lsmod
Root crontab onfig : crontab -l
Services: service --status-all


Reference :
http://www.thegeekstuff.com/2011/03/sar-examples/
http://www.thegeekstuff.com/2011/03/linux-performance-monitoring-intro/
http://www.brendangregg.com/index.html