Birkhoff's Blog

Thoughts and technonogies.

在 macOS Sierra 10.12.2 上編譯 Caffe 並使用 DIGITS

Birkhoff Lee's Avatar January 8th 2017

Update: 目前推薦使用 Docker 環境的 DIGITS

最近在玩 Machine learning,編譯 caffe 跟啟動 DIGITS 的時候遇到一些瓶頸,在這裡記錄下解決方案並分享給需要的人。過程中發生錯誤都可以在下面留言詢問喔!

這邊因為我 training 的量比較少,我就沒有用 GPU,因此這篇不會教怎麼裝 n 卡的驅動程式,不過這邊有教學,在設定 Makefile.config 的時候也記得去掉 USE_CUDNN := 1 最前面的 #,這樣應該就可以了,其他部分跟本教學通用。

homebrew install

如果已經安裝過 homebrew 則可以跳過此步驟。

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

dependency install

首先要確認透過 homebrew 安裝的 Python 已經刪除:

brew uninstall python

接著執行:

brew install -vd snappy leveldb gflags glog szip lmdb
brew install hdf5 opencv
brew upgrade libpng
brew tap homebrew/versions

執行 brew edit opencv,然後更改下面兩行:

-DPYTHON_LIBRARY=#{py_prefix}/lib/libpython2.7.dylib
-DPYTHON_INCLUDE_DIR=#{py_prefix}/include/python2.7

編譯並安裝 protobuf 2.6.1:

cd ~/Documents
curl -LO https://github.com/google/protobuf/releases/download/v2.6.1/protobuf-2.6.1.tar.bz2
tar xvjf protobuf-2.6.1.tar.bz2
rm protobuf-2.6.1.tar.bz2
cd protobuf-2.6.1
./configure
make
make check
sudo make install
cd python
sudo python2.7 setup.py build
sudo python2.7 setup.py install

然後編譯並安裝 boost:

brew install --build-from-source -vd boost159 boost-python159

接下來執行:

brew link --force boost159

download caffe & prepare deps

cd ~/Documents
git clone https://github.com/BVLC/caffe.git
cd caffe
git checkout 20feab5771ae5cbb257cfec85e0b98da06269068
cp Makefile.config.example Makefile.config

然後我們要換一個 compiler,請登入 https://developer.apple.com/downloads/ ,然後下載 Xcode Command Line Tools 7.3,接著安裝它,然後執行:

sudo xcode-select --switch /Library/Developer/CommandLineTools

Configure Makefile.config

打開 Makefile.config,將 BLAS_INCLUDE 那行變更為:

BLAS_INCLUDE := /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/Headers

BLAS_LIB 那行變更為:

BLAS_LIB := /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A

PYTHON_INCLUDE 那行變更為:

PYTHON_INCLUDE := /usr/include/python2.7 \
/Library/Python/2.7/site-packages/numpy/core/include

PYTHON_LIB 那行變更為:

PYTHON_LIB := /usr/lib

# WITH_PYTHON_LAYER := 1 變更為:WITH_PYTHON_LAYER := 1

# USE_LEVELDB := 0 變更為:USE_LEVELDB := 0

build caffe

開始編譯!

make -j8 all

最後裝好一些 python dependency,然後就開始跑測試,測試過程中可能會有很多 Warning,不用理他。

sudo -H easy_install pip
pip install --user -r python/requirements.txt
sudo -H pip install protobuf
make -j8 pytest
make -j8 test
make -j8 runtest
make -j8 pycaffe
make -j8 pytest

測試跑完之後,執行下列指令:

make -j8 distribute

make 好之後,執行以下指令讓你可以在 python 內 import caffe

cp -r distribute/python/caffe ~/Library/Python/2.7/lib/python/site-packages/
cp distribute/lib/libcaffe.so.1.0.0-rc3 ~/Library/Python/2.7/lib/python/site-packages/caffe/libcaffe.so.1.0.0-rc3
install_name_tool -change @rpath/libcaffe.so.1.0.0-rc3 ~/Library/Python/2.7/lib/python/site-packages/caffe/libcaffe.so.1.0.0-rc3 ~/Library/Python/2.7/lib/python/site-packages/caffe/_caffe.so

verify build

最後跑下面這行試試看,如果沒有輸出就代表安裝成功了!

python -c 'import caffe'

install & launch DIGITS

將下面這兩行塞到你的 .bashrc.zshrc 等類似東西裡面去:

export DIGITS_ROOT="~/Documents/DIGITS"
export CAFFE_ROOT='~/Documents/caffe'

重啟 shell,接著執行:

git clone https://github.com/NVIDIA/caffe.git $CAFFE_ROOT
cd $DIGITS_ROOT
sudo pip install -r $CAFFE_ROOT/python/requirements.txt

接下來 vim ./digits-devserver 並把最後一行的開頭 python 改為 python2.7,然後跑下面這行:

./digits-devserver

看到下面這樣就是啟動成功了!如果出現錯誤,先重複前面編譯並安裝 protobuf 2.6.1 的步驟試試,這邊一定要使用 2.6.1 版本,大於 3.0.0 的版本目前有 bug 無法搭配 DIGITS 使用。

接著開啟 http://localhost:5000/ 就會進入 DIGITS 首頁囉,大概長這樣:

remember…

最後再安裝回原本的 Command-Line-Tools:下載並安裝 Command_Line_Tools_macOS_10.12_for_Xcode_8.2.dmg

FAQ

1. 碰到下列狀況就是 numpy 版本不對或者衝突了:
$ make -j8 pytest
cd python; python -m unittest discover -s caffe/test
dyld: warning, LC_RPATH @executable_path/ in /usr/local/cuda/lib/libcublas.8.0.dylib being ignored in restricted program because of @executable_path
dyld: warning, LC_RPATH @executable_path/ in /usr/local/cuda/lib/libcurand.8.0.dylib being ignored in restricted program because of @executable_path
python(71396,0x7fffefcc13c0) malloc: *** malloc_zone_unregister() failed for 0x7fffefcb7000
RuntimeError: module compiled against API version 0xa but this version of numpy is 0x9
/bin/sh: line 1: 71396 Illegal instruction: 4 python -m unittest discover -s caffe/test
make: *** [pytest] Error 132

這時先重新安裝 numpy,然後看一下它裝去哪了:

sudo -H pip uninstall numpy
sudo -H pip install 'numpy==1.11.3'
pip show numpy

在 Location 那一行就是它的 lib 位置,這時打開 Makefile.config 找到 PYTHON_INCLUDE,把它的第二行 path 改成這邊顯示的位置。

2. 出現 ImportError: No module named protobuf, fatal error 'numpy/arrayobject.h' file not found

執行下面兩行即可解決

sudo -H pip uninstall protobuf
sudo -H pip install 'protobuf==3.1.0.post1'

references