Big Data and Cloud

Saturday, December 31, 2016

Mask r-cnn Faster r-cnn

https://blog.athelas.com/a-brief-history-of-cnns-in-image-segmentation-from-r-cnn-to-mask-r-cnn-34ea83205de4

Mask R-CNN
https://arxiv.org/abs/1703.06870
http://kaiminghe.com
https://www.zhihu.com/question/57403701

Faster R-CNN
https://github.com/rbgirshick/py-faster-rcnn

https://devtalk.nvidia.com/default/topic/974063/caffe-failed-with-py-faster-rcnn-demo-py-on-tx1/

Monday, December 19, 2016

https://vision.princeton.edu/courses/COS598/2015sp/slides/Caffe/caffe_tutorial.pdf

https://docs.google.com/presentation/d/1UeKXVgRvvxg9OUdh_UiC5G71UMscNPlvArsWER41PsU/edit#slide=id.gc2fcdcce7_216_339

http://caffe.berkeleyvision.org/

http://caffe.berkeleyvision.org/gathered/examples/mnist.html

Friday, December 16, 2016

Ubuntu 16.04 + GTX1080 + Caffe

USB install Ubuntu 16.04

sudo apt-get install ssh

sudo service ssh restart

sudo add-apt-repository ppa:graphics-drivers/ppa

sudo apt-get update

sudo apt-get install nvidia-367 (no need to upgrade to 370 as CUDA 8 use 367)

download 

linux-headers-4.7.0-040700_4.7.0-040700.201608021801_all.deb

linux-headers-4.7.0-040700-generic_4.7.0-040700.201608021801_amd64.deb

linux-image-4.7.0-040700-generic_4.7.0-040700.201608021801_amd64.deb

sudo dpkg -i *.deb

upgrade to kernel 4.7

fix Intel Dual BandWireless-AC 3168

sudo dpkg -i linux-firmware_1.161_all.deb

CUDA

wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_8.0.44-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1604_8.0.44-1_amd64.deb
sudo apt-get update
sudo apt-get install cuda

https://yangcha.github.io/GTX-1080/  

cuDNN 5.1 

tar xvzf cudnn-8.0-linux-x64-v5.1.tgz
sudo cp cuda/include/cudnn.h /usr/local/cuda/include
sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64
sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn*
 
Install Caffe 

https://github.com/BVLC/caffe/wiki/Ubuntu-16.04-or-15.10-Installation-Guide 

sudo apt-get update

sudo apt-get -y upgrade

sudo apt-get install -y build-essential cmake git pkg-config

sudo apt-get install -y libprotobuf-dev libleveldb-dev libsnappy-dev libhdf5-serial-dev protobuf-compiler

sudo apt-get install -y libatlas-base-dev 

sudo apt-get install -y --no-install-recommends libboost-all-dev

sudo apt-get install -y libgflags-dev libgoogle-glog-dev liblmdb-dev

# (Python general)
sudo apt-get install -y python-pip

# (Python 2.7 development files)
sudo apt-get install -y python-dev
sudo apt-get install -y python-numpy python-scipy

# (or, Python 3.5 development files)
sudo apt-get install -y python3-dev
sudo apt-get install -y python3-numpy python3-scipy

# (OpenCV 2.4)
sudo apt-get install -y libopencv-dev
 
git clone https://github.com/BVLC/caffe.git

cd caffe
 
cp Makefile.config.example Makefile.config

vi Makefile.config
 
cd python

for req in $(cat requirements.txt); do pip install $req; done
 
cd ..

vi Makefile

make all
make test
make runtest
make pycaffe  
make distribute
 
The binary models can be download with the following script. In caffe-master directory, 

cd scripts
./download_model_binary.py ../models/bvlc_alexnet/
./download_model_binary.py ../models/bvlc_googlenet/
./download_model_binary.py ../models/bvlc_reference_caffenet/
./download_model_binary.py ../models/bvlc_reference_rcnn_ilsvrc13/
./download_model_binary.py ../models/finetune_flickr_style/
 
Python
sudo apt-get install python-tk

Saturday, December 10, 2016

Ansible : lineinfile

https://groups.google.com/forum/#!msg/ansible-project/JvHfchsgRaU/Vw_CzBbvadgJ

-- code extract ----

# Regexp implementation note:

# - the line to match follows the following pattern:

# - <startofline>: matched by ^

# - A number of white spaces/tabs: matched by \s+

# - the "kernel" string: matched by kernel

# - A list of: <spaces|tab><string> -> this is all the options passed to the kernel

# - possible spaces and tabs at the end : matched by the latest \s*

# - the <end of line>: matched by $

# - the list of: <spaces|tab><string1> described above

# e.g: /vmlinuz-2.6.32-358.el6.x86_64 ro root=/dev/mapper/VolGroup00-root_vol … rd_LVM_LV=VolGroup00/root_vol SYSFONT=latarcyrheb-sun16 rd_NO_DM

# is matched by: (\s+(?!pcie_aspm=off)[\w=/\-\.]+)*

# The (?!pcie_aspm=off) is to direct the regexp engine to check that the string 'pcie_aspm=off'

is not present

# anywhere in the in the <string1> part. This is called negative lookahead.

# This is just a check, no characters are "consumed". The following [\w=/\-\.]+

# is ther to consume aka match any strings made of alphanumeric characters,the '/', the '-', the '.'

# - if the 'pcie_aspm=off' string is present, there is no match

# - the capturing group 1 referred as to \1 in the line= statement

# will capture the whole line but the trailing <withe spaces> and <end of line>

- name: Activate a lacking kernel option 'pcie_aspm=off' in {{ targetgrub }}

lineinfile: dest={{ targetgrub }}

backup=True

backrefs=True

state=present

regexp='(^\s+kernel(\s+(?!pcie_aspm=off)[\w=/\-\.]+)*)\s*$'

line='\1 pcie_aspm=off'

-- end of code extract ----

http://daveops.co.uk/2014/05/using-ansible-to-append-a-string-to-the-end-of-a-line/

http://www.handverdrahtet.org/2016/01/ansible-using-numbered-backreference.html

Friday, December 9, 2016

Ubuntu 16.10 + GTX 1080

USB install Ubuntu 16.10

$ uname -r

4.8.0-30-generic

Ctrl+Alt+T to start terminal as mouse not working

sudo apt-get install ssh

sudo service ssh restart

Fix mouse not working problem

sudo add-apt-repository ppa:graphics-drivers/ppa

sudo apt-get update

sudo apt-get install nvidia-367

https://yangcha.github.io/GTX-1080/

wget http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_8.0.44-1_amd64.deb
sudo dpkg -i cuda-repo-ubuntu1604_8.0.44-1_amd64.deb
sudo apt-get update
sudo apt-get install cuda

$ nvidia-smi

Fri Dec  9 23:21:17 2016       

+-----------------------------------------------------------------------------+

| NVIDIA-SMI 367.57                 Driver Version: 367.57                    |

|-------------------------------+----------------------+----------------------+

| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |

| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |

|===============================+======================+======================|

|   0  GeForce GTX 1080    Off  | 0000:01:00.0      On |                  N/A |

| 27%   25C    P8     6W / 180W |    284MiB /  8111MiB |      0%      Default |

+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+

| Processes:                                                       GPU Memory |

|  GPU       PID  Type  Process name                               Usage      |

|=============================================================================|

|    0      1078    G   /usr/lib/xorg/Xorg                             146MiB |

|    0      2170    G   /usr/bin/compiz                                136MiB |

+-----------------------------------------------------------------------------+

cudnn 5.1

https://developer.nvidia.com/rdp/cudnn-download

Download cuDNN v5.1 (August 10, 2016)

cuDNN v5.1 Library for Linux

tar xvzf cudnn-8.0-linux-x64-v5.1.tgz

sudo cp cuda/include/cudnn.h /usr/local/cuda/include

sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64

sudo chmod a+r /usr/local/cuda/include/cudnn.h /usr/local/cuda/lib64/libcudnn*

Install Caffe

https://github.com/BVLC/caffe/wiki/Ubuntu-16.04-or-15.10-Installation-Guide

CXX src/caffe/util/db.cpp
CXX src/caffe/util/db_leveldb.cpp
CXX src/caffe/util/db_lmdb.cpp
CXX src/caffe/util/hdf5.cpp
CXX src/caffe/util/im2col.cpp
CXX src/caffe/util/insert_splits.cpp
CXX src/caffe/util/io.cpp
CXX src/caffe/util/math_functions.cpp
CXX src/caffe/util/signal_handler.cpp
CXX src/caffe/util/upgrade_proto.cpp
NVCC src/caffe/layers/absval_layer.cu
nvcc warning : The 'compute_20', 'sm_20', and 'sm_21' architectures are deprecated, and may be removed in a future release (Use -Wno-deprecated-gpu-targets to suppress warning).
In file included from /usr/local/cuda-8.0/include/cuda_runtime.h:78:0,
                 from <command-line>:0:
/usr/local/cuda-8.0/include/host_config.h:119:2: error: #error -- unsupported GNU version! gcc versions later than 5 are not supported!
 #error -- unsupported GNU version! gcc versions later than 5 are not supported!
  ^~~~~
Makefile:589: recipe for target '.build_release/cuda/src/caffe/layers/absval_layer.o' failed
make: *** [.build_release/cuda/src/caffe/layers/absval_layer.o] Error 1
 
Fix: https://github.com/BVLC/caffe/wiki/GeForce-GTX-1080,---CUDA-8.0,---Ubuntu-16.04,---Caffe

#if __GNUC__ > 5

//#error -- unsupported GNU version! gcc versions later than 5 are not supported!

#endif /* __GNUC__ > 5 */

Ubuntu 16.10 gcc version is 6.2 

Use this step to change gcc version to 5.4 

https://gist.github.com/beci/2a2091f282042ed20cda

sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get install gcc-5 g++-5
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 60 --slave /usr/bin/g++ g++ /usr/bin/g++-5

sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-5 1

$ gcc --version
gcc (Ubuntu 5.4.1-2ubuntu2) 5.4.1 20160929

$ g++ --version
g++ (Ubuntu 5.4.1-2ubuntu2) 5.4.1 20160929

Problem when running caffe runtest

[----------] 5 tests from ImageDataLayerTest/2, where TypeParam = caffe::GPUDevice<float>
[ RUN      ] ImageDataLayerTest/2.TestSpace
[       OK ] ImageDataLayerTest/2.TestSpace (32 ms)
[ RUN      ] ImageDataLayerTest/2.TestShuffle
[       OK ] ImageDataLayerTest/2.TestShuffle (104 ms)
[ RUN      ] ImageDataLayerTest/2.TestRead
[       OK ] ImageDataLayerTest/2.TestRead (104 ms)
[ RUN      ] ImageDataLayerTest/2.TestResize
*** Aborted at 1481571270 (unix time) try "date -d @1481571270" if you are using GNU date ***
PC: @     0x7fa785a370ba (unknown)
*** SIGSEGV (@0xfffffffffffffff7) received by PID 20554 (TID 0x7fa75d032700) from PID 18446744073709551607; stack trace: ***
    @     0x7fa786f1d630 (unknown)
    @     0x7fa785a370ba (unknown)
    @     0x7fa785a3718b (unknown)
    @     0x7fa785a38ce8 (unknown)
    @     0x7fa785a37692 (unknown)
    @     0x7fa785a32020 (unknown)
    @     0x7fa785a30165 tbb::internal::allocate_root_with_context_proxy::allocate()
    @     0x7fa78937ce22 cv::parallel_for_()
    @     0x7fa788c63b2a (unknown)
    @     0x7fa788c60bb7 cv::resize()
    @     0x7fa787b3a5e1 caffe::ReadImageToCVMat()
    @     0x7fa787a58550 caffe::ImageDataLayer<>::load_batch()
    @     0x7fa787a10019 caffe::BasePrefetchingDataLayer<>::InternalThreadEntry()
    @     0x7fa7879e8dbe caffe::InternalThread::entry()
    @     0x7fa7879ebd0f boost::_mfi::mf5<>::operator()()
    @     0x7fa7879ebbe7 boost::_bi::list6<>::operator()<>()
    @     0x7fa7879ebadc boost::_bi::bind_t<>::operator()()
    @     0x7fa7879eba8e boost::detail::thread_data<>::run()
    @     0x7fa788923596 (unknown)
    @     0x7fa786f136ca start_thread
    @     0x7fa786c4d0af clone
    @                0x0 (unknown)
Makefile:527: recipe for target 'runtest' failed
make: *** [runtest] Segmentation fault (core dumped)

Tried recompile protobuf verion 2.5 : https://github.com/BVLC/caffe/wiki/GeForce-GTX-1080,---CUDA-8.0,---Ubuntu-16.04,---Caffe

But it does not solve the problem.

Ubuntu 16.04 WiFi is not detected for Intel Dual BandWireless-AC 3168

Ubuntu 16.04 WiFi is not detected for Intel Dual BandWireless-AC 3168

https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1627114

http://askubuntu.com/questions/827795/how-do-i-get-an-intel-wireless-3168-802-11ac-wireless-card-to-work/828614#828614

nvidia
http://askubuntu.com/questions/760934/graphics-issues-after-while-installing-ubuntu-16-04-16-10-with-nvidia-graphics

Saturday, December 3, 2016

Ansible : sphere_guest

https://github.com/ansible/ansible-modules-core/issues/2006

ansible use /Resources/<RESOURCE_NAME>

http://stackoverflow.com/questions/38736888/trouble-with-pysphere-ansible

- hosts: localhost
  tasks:
    - vsphere_guest:
         ...