Tuesday, December 8, 2015

Python : Build PyKerberos Package

Build PyKerberos package from https://pypi.python.org/pypi/kerberos

# yum install python-kerberos gcc krb5-devel python-devel
# wget https://pypi.python.org/packages/source/k/kerberos/kerberos-1.2.2.tar.gz
# tar -xzf kerberos-1.2.2.tar.gz
# cd kerberos-1.2.2
# python setup.py build

# python setup.py install
running install
running bdist_egg
running egg_info
writing kerberos.egg-info/PKG-INFO
writing top-level names to kerberos.egg-info/top_level.txt
writing dependency_links to kerberos.egg-info/dependency_links.txt
reading manifest file 'kerberos.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'kerberos.egg-info/SOURCES.txt'
installing library code to build/bdist.linux-x86_64/egg
running install_lib
running build_ext
creating build/bdist.linux-x86_64
creating build/bdist.linux-x86_64/egg
copying build/lib.linux-x86_64-2.7/kerberos.so -> build/bdist.linux-x86_64/egg
creating stub loader for kerberos.so
byte-compiling build/bdist.linux-x86_64/egg/kerberos.py to kerberos.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying kerberos.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying kerberos.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying kerberos.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying kerberos.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
writing build/bdist.linux-x86_64/egg/EGG-INFO/native_libs.txt
zip_safe flag not set; analyzing archive contents...
creating dist
creating 'dist/kerberos-1.2.2-py2.7-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
Processing kerberos-1.2.2-py2.7-linux-x86_64.egg
Copying kerberos-1.2.2-py2.7-linux-x86_64.egg to /usr/lib64/python2.7/site-packages
Adding kerberos 1.2.2 to easy-install.pth file

Installed /usr/lib64/python2.7/site-packages/kerberos-1.2.2-py2.7-linux-x86_64.egg
Processing dependencies for kerberos==1.2.2
Finished processing dependencies for kerberos==1.2.2

Linux : Fedora : Install and Configure Keberos KDC and Client

#yum install krb5-server krb5-workstation pam_krb5

Kerboros Server Configuration:
#cd /var/kerberos/krb5kdc

#ls -l
total 8
-rw------- 1 root root  22 Nov  5 12:36 kadm5.acl
-rw------- 1 root root 451 Nov  5 12:36 kdc.conf

#vi kadm5.acl

*/admin@EXAMPLE.COM *

#vi kdc.conf

[realms]
EXAMPLE.COM = {

Kerberos Client configuration
#cd /etc
#ls -ld krb5.conf
-rw-r--r-- 1 root root 525 Nov  5 12:36 krb5.conf

#vi krb5.conf

[libdefaults]
 default_realm = EXAMPLE.COM

[realms]
 EXAMPLE.COM = {
  kdc = kerberos.example.com
  admin_server = kerberos.example.com
 }

[domain_realm]
 .example.com = EXAMPLE.COM
 example.com = EXAMPLE.COM

Initialize database
# kdb5_util create -s -r EXAMPLE.COM
Loading random data
Initializing database '/var/kerberos/krb5kdc/principal' for realm 'EXAMPLE.COM',
master key name 'K/M@EXAMPLE.COM'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
Re-enter KDC database master key to verify:

# cd /var/kerberos/krb5kdc

# ls -al
total 36
drwxr-xr-x  2 root root 4096 Dec  7 12:34 .
drwxr-xr-x. 4 root root   31 Dec  7 12:15 ..
-rw-------  1 root root   68 Dec  7 12:34 .k5.EXAMPLE.COM
-rw-------  1 root root   18 Dec  7 12:19 kadm5.acl
-rw-------  1 root root  447 Dec  7 12:23 kdc.conf
-rw-------  1 root root 8192 Dec  7 12:34 principal
-rw-------  1 root root 8192 Dec  7 12:34 principal.kadm5
-rw-------  1 root root    0 Dec  7 12:34 principal.kadm5.lock
-rw-------  1 root root    0 Dec  7 12:34 principal.ok

# systemctl start krb5kdc kadmin
# systemctl enable krb5kdc kadmin
Created symlink from /etc/systemd/system/multi-user.target.wants/krb5kdc.service to /usr/lib/systemd/system/krb5kdc.service.
Created symlink from /etc/systemd/system/multi-user.target.wants/kadmin.service to /usr/lib/systemd/system/kadmin.service.

Add entries
# kadmin.local
Authenticating as principal root/admin@EXAMPLE.COM with password.
kadmin.local:  listprincs
K/M@EXAMPLE.COM
kadmin/admin@EXAMPLE.COM
kadmin/changepw@EXAMPLE.COM
kadmin/p-68@EXAMPLE.COM
kiprop/p-68@EXAMPLE.COM
krbtgt/EXAMPLE.COM@EXAMPLE.COM
kadmin.local:  addprinc root/admin
WARNING: no policy specified for root/admin@EXAMPLE.COM; defaulting to no policy
Enter password for principal "root/admin@EXAMPLE.COM":
Re-enter password for principal "root/admin@EXAMPLE.COM":
Principal "root/admin@EXAMPLE.COM" created.
kadmin.local:  listprincs
:
:
:
root/admin@EXAMPLE.COM

kadmin.local:  addprinc test  <- add user
kadmin.local:  addprinc -randkey host/p-68.example.com <- add host

kadmin.local:  ktadd host/p-68.example.com
Entry for principal host/p-68.example.com with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/p-68.example.com with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/p-68.example.com with kvno 2, encryption type des3-cbc-sha1 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/p-68.example.com with kvno 2, encryption type arcfour-hmac added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/p-68.example.com with kvno 2, encryption type camellia256-cts-cmac added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/p-68.example.com with kvno 2, encryption type camellia128-cts-cmac added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/p-68.example.com with kvno 2, encryption type des-hmac-sha1 added to keytab FILE:/etc/krb5.keytab.
Entry for principal host/p-68.example.com with kvno 2, encryption type des-cbc-md5 added to keytab FILE:/etc/krb5.keytab.

kadmin.local:  quit

Set up SSH client to accept Kerberos
# cd /etc/ssh
# vi ssh_config

   GSSAPIAuthentication yes
   GSSAPIDelegateCredentials yes

# authconfig --enablekrb5 --update
# systemctl reload sshd.service

Now login the test user account
$ id
uid=1000(test) gid=1000(test) groups=1000(test)
$ klist
klist: Credentials cache keyring 'persistent:1000:1000' not found

$ kinit
Password for test@EXAMPLE.COM:

$ klist (now we have Kerberos token assigned for 24 hours)
Ticket cache: KEYRING:persistent:1000:1000
Default principal: test@EXAMPLE.COM

Valid starting       Expires              Service principal
12/07/2015 13:33:41  12/08/2015 13:33:41  krbtgt/EXAMPLE.COM@EXAMPLE.COM

Monday, December 7, 2015

Linux : kinit: Cannot find KDC for realm while getting initial credentials

Problem:
$ id
uid=1000(test) gid=1000(test) groups=1000(test)
$ klist
klist: Credentials cache keyring 'persistent:1000:1000' not found

$ kinit
kinit: Cannot find KDC for realm "EXAMPLE.COM" while getting initial credentials

Solution:
Check client's /etc/krb5.conf and make sure [realms] section is correct

Linux : Performance monitoring tool

Collect Statistics

* Sysstat package : mpstat, vmstat, pidstat, iostat
* dstat, nicstat
* sar , atop
* Percona Cacti template

Watch online
* top
* iotop
* iftop

Tracing
* perf_events
* ftrace
* perf-toold
* Flamegraph
http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html


* uptime
If load average > # of CPUs, may mean CPU saturation

* top or htop
Can miss short-lived processes (atop won't)
can consume noticeable CPU to read /proc

* ps

* vmstat
cpu section , us means user or application, sy means kernel / system call usage

* iostat
ex. iostat -xmdz 1 or iostat -x 1
very useful set of stats

* mpstat
ex mpstat -P ALL 1 (look for unbalanced workloads, hot CPUs)

* free
main memory usage
ex. free -m
buffers: block device I/O cache
cached: virtual page cache

* sar
ex. sar -n DEV 1 (network device)

Observability Tools : Intermediate
* strace (system call tracer)
ex. strace -tttT -p (#PID)
-ttt : time (us) since epoch
-T: syscall time(s)
Translate syscall args, very helpful for solving system usage issues
Currently has massive overhead (ptrace based)
Can slow the target by > 100x , so use with caution
ex. strike -Tttvfs 1024 -o /tmp/strace.out <command>

* tcpdump
Sniff network packets for post analysis
Study packet sequences with timestamps (us)
CPU overhead optimized (socket ring buffers), but can be significant. Use caution.
ex: tcpdump -s 0 -n -i ethX -w /tmp/tcpdump.pcap &

* netstat
-s : various network protocol statistics
-i : interface stats
-r : route table
-p : shows process details
-c : per second interval

* pidstat
very useful process stats. by thread, disk I/O
ex. pidstat -t 1
ex. pidstat -d 1

* swapon
show swap device usage: swapon -s

* lsof
more a debug tool, lsof shows file descriptor usage
ex. lsof -iTCP -sTCP:ESTABLISHED

* sar (and collectl, dstat, etc. )
Archive or live mode (interval [count])
logical groups: TCP, ETCP, DEV, EDEV
ex. sar -n TCP,ETCP,DEV 1

Report network statistics (sar -n)

This reports various network statistics. For example: number of packets received (transmitted) through the network card, statistics of packet failure etc.,. “1 3″ reports for every 1 seconds a total of 3 times.
sar -n KEYWORD
KEYWORD can be one of the following:
  • DEV – Displays network devices vital statistics for eth0, eth1, etc.,
  • EDEV – Display network device failure statistics
  • NFS – Displays NFS client activities
  • NFSD – Displays NFS server activities
  • SOCK – Displays sockets in use for IPv4
  • IP – Displays IPv4 network traffic
  • EIP – Displays IPv4 network errors
  • ICMP – Displays ICMPv4 network traffic
  • EICMP – Displays ICMPv4 network errors
  • TCP – Displays TCPv4 network traffic
  • ETCP – Displays TCPv4 network errors
  • UDP – Displays UDPv4 network traffic
  • SOCK6, IP6, EIP6, ICMP6, UDP6 are for IPv6
  • ALL – This displays all of the above information. The output will be very long.
* strace
trace system call of a running process
ex. strace -tp `pgrep [process name, eg. java] ` 2>&1 | head -100

Advanced Observability tools
Misc: ltrace, ss, iptraf, ethtool, snmpget, lldptool, iotop, blktrace, slabtop, /proc, pcstat
CPU Performance Counters: perf_events, tiptop, rdmsr
Advanced Tracers: perf_events, ftrace, eBPF, SystermTap, ktap, LTTng, dtrace4linux, sysdig

* ss
socket statistics
ex. ss -mop
ex. ss -i

* iptraf
provide histograms of different packet sizes

* iotop
block device I/O disk by process
Need kernel support enabled : CONFIG_TASK_IO_ACCOUNTING

* slabtop
Kernel slab allocator memory usage

* pcstat
Show page cache residency by file
Uses the mincore syscall. Useful for database performance analysis.

* perf_events
Provides the perf command
In Linux source code: tools/perf
Usually pkg added by linux-tools-common
Multi-tool : CPU profiling, PMC profiling, Static & dynamic tracing

* tiptop
IPC by process
Still cannot use it in clouds yet. needs PMCs enabled

* rdmsr
Model Specific Registers (MSRs), unlike PMCs, can be read by default in Xen guests

Benchmarking Tools
Multi: UnixBench, lmbench, sysbench, perf bench
FS/disk: dd, hdparm, fio
App/lib: ab, wrk, jmeter, openssl
Networking: ping, hping3, iperf, ttcp, traceroute, mtr, pchar

Disk benchmark: run iostat...
CPU benchmark: run pidstat, perf, flame graph....

Tuning Tools
Generic interfaces: sysctl, /sys
CPU/scheduler : nice, renice, taskset, ulimit, chcpu
Storage I/O: tune2fs, ionice, hdparm, blockdev
Network: ethtool, tc, ip, route
Dynamic patching: stap, kpatch

Tuning Methods :
Scientific Method:
Question
Hypothesis
Prediction
Test
Analysis

* ip route get [IP address]

ETC
system messages: dmesg
Network interface config: ifconfig -a , ip link
File system capacity: df -h
Volume config : mdadm --misc -D /dev/md0
Storage device info : smartctl
NUMA config: numactl -s , numactl -H
PCI info : lspci
Installed kernel modules: lsmod
Root crontab onfig : crontab -l
Services: service --status-all


Reference :
http://www.thegeekstuff.com/2011/03/sar-examples/
http://www.thegeekstuff.com/2011/03/linux-performance-monitoring-intro/
http://www.brendangregg.com/index.html

Friday, December 4, 2015

Data : Notes

Types of data
1Categorical (factor in R, level = category)
 A Nominal ( = !=  distinctness)
 B Ordinal ( < > order + distinctness)

2Numeric (numeric in R)
 A Interval ( + - addition + order + distinctness )
 B Ratio ( * / multiplication + addition + order + distinctness )

Thursday, December 3, 2015

Linux : Red Hat view kernel parameter

sysctl -a
cat /proc/cmdline

Linux : Red Hat 7 boot to single user mode

1 Append rd.break  to kernel cmdline and Ctrl-x to boot to single user mode

kernel cmdline is the line contains (linux16 /vmlinuz-3.10.0-2........)

After boot up in single mode, run
mount -o remount,rw /sysroot
chroot /sysroot
Then we can modify necessary files

Or

2 Append init=/bin/bash to kernel cmdline and Ctrl-x to boot to single user mode
After boot up in single mode, run
mount -o remount,rw /
Then we can modify necessary files