Deep Learning - Installing Caffe on Ubuntu



Caffe

Recently, I want to try another deep learning framework instead of tensorflow.
I find Caffe (Convolutional Architecture for Fast Feature Embedding), which is another deep learning framework, originally developed at UC Berkeley, is easy to handle. There are still a bunch of troubles, the biggest one is the environment of installation.

Many developers are complaining about the complex dependence of Caffe environment, I try it today, it really runs into many problems.
In this tutorial, I mainly talke about how to build Caffe environment on 64-bit Ubuntu 16.04 Linux.

In Caffe official home page, we can find the installation tutorial: http://caffe.berkeleyvision.org/install_apt.html,
which still remaining a stack of problems for different system environments.
In order to install Caffe, we need to prepare some dependencies softwares including: libgflags, libatlas, libgoogle-glog, libmdb, python-dev, while for GPU support, we need to extra install Nvidia driver, cuDNN, opencv3+.
After successfully installing those packages, we also need to modify the Makefile.config to tell the compiler where to find it.


1.General dependencies

First of all, check our ubuntue apt-get source list is updated:

sudo apt-get update -y

Next, following installation tutorial of home page, we may install the dependencies:

sudo apt-get install python-dev libprotobuf-dev libleveldb-dev libsnappy-dev libopencv-dev libhdf5-serial-dev protobuf-compiler  -y
sudo apt-get install --no-install-recommends libboost-all-dev  -y

2.Install Caffe package

Secondly, clone Caffe repository, and make a copy of file Makefile.config.example:

git clone https://github.com/BVLC/caffe
cd caffe
cp Makefile.config.example Makefile.config

Install basic python dependencies:

sudo pip install numpy scikit-image protobuf
cd python
for req in $(cat requirements.txt); do sudo pip install $req; done
cd ../

To this step, we have installed all dependencies for Caffe, we shall tell compiler where to find the package next step.
Edit Makefile.config just for cpu support here, if you want to make it supporting gpu, please switch to Step-3:

➜  $: vim Makefile.config
...
# cuDNN acceleration switch (uncomment to build with cuDNN).
USE_CUDNN := 0
# CPU-only switch (uncomment to build without GPU support).
CPU_ONLY := 1
...
# Change INCLUDE_DIRS to add hdf5 files, if you have problem with hdf5.h package, please see Step-4.3.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial
...
#: Change python include package path.
PYTHON_INCLUDE := /usr/include/python2.7 \
    /usr/local/lib/python2.7/dist-packages/numpy/core/include
...

After check everything is installed, we begin to compile Caffe, for any fatal error, please see Step-4:

make all

If no error occur, just to test:

make test && make runtest

We shall see the successful interface like this:
Successful installed Caffe


3.Setup for GPU supporting

To make our Caffe supporting GPU, we shall install Nvidia development toolkit at first, tutorial of installation please see: http://homeway.me/2017/08/08/setup-gpu-for-tensorflow/
Next is opencv3+: https://www.learnopencv.com/install-opencv3-on-ubuntu/

After a series of installation steps, just edit Makefile.config for compiling:

➜  $: vim Makefile.config
...
# cuDNN acceleration switch (uncomment to build with cuDNN).
USE_CUDNN := 1
# CPU-only switch (uncomment to build without GPU support).
CPU_ONLY := 0
...
# CUDA directory contains bin/ and lib/ directories that we need.
CUDA_DIR := /usr/local/cuda
...
# Uncomment if you're using OpenCV 3
OPENCV_VERSION := 3
...
# Change INCLUDE_DIRS to add hdf5 files, if you have problem with hdf5.h package, please see Step-4.3.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial
...
#: Change python include package path.
PYTHON_INCLUDE := /usr/include/python2.7 \
    /usr/local/lib/python2.7/dist-packages/numpy/core/include
...

Everything is ok, just wait for successful testing in screen:

make all
make test && make runtest

4.Fix bugs

Make a summary, here lists some common mistakes and solutions, mostly due to missing packages or packages not found by compiler, just install it and tell compiler.

4.1 gflags not found

Error description:

PROTOC src/caffe/proto/caffe.proto
CXX .build_release/src/caffe/proto/caffe.pb.cc
CXX src/caffe/layer.cpp
In file included from ./include/caffe/blob.hpp:8:0,
                 from ./include/caffe/layer.hpp:8,
                 from src/caffe/layer.cpp:1:
./include/caffe/common.hpp:5:27: fatal error: gflags/gflags.h: ?????????
compilation terminated.
Makefile:581: recipe for target '.build_release/src/caffe/layer.o' failed
make: *** [.build_release/src/caffe/layer.o] Error 1

Just install it, and try again:

sudo apt-get install libgflags-dev -y

4.2 glog not found

Error description:

CXX src/caffe/layer.cpp
In file included from ./include/caffe/blob.hpp:8:0,
                 from ./include/caffe/layer.hpp:8,
                 from src/caffe/layer.cpp:1:
./include/caffe/common.hpp:6:26: fatal error: glog/logging.h: ?????????
compilation terminated.
Makefile:581: recipe for target '.build_release/src/caffe/layer.o' failed
make: *** [.build_release/src/caffe/layer.o] Error 1

Install it by apt-get package:

sudo apt-get install libgoogle-glog-dev liblmdb-dev -y

Or install it from source package, installation wiki from Github: Installing Glog on Ubuntu 14.04

#: remove the old gflags 
sudo apt-get purge libgflags-dev
#: download glog from github
git clone https://github.com/google/glog && cd glog
# compiling
mkdir build && cd build
cmake -DGFLAGS_NAMESPACE=google -DCMAKE_CXX_FLAGS=-fPIC ..
make && sudo make install

4.3 hdf5.h

Error description:

src/caffe/layers/hdf5_data_layer.cpp:13:18: fatal error: hdf5.h: No such file or directory
compilation terminated.
Makefile:581: recipe for target '.build_release/src/caffe/layers/hdf5_data_layer.o' failed
make: *** [.build_release/src/caffe/layers/hdf5_data_layer.o] Error 1

In general, we have installed hdf5.h package in Step-1, the problem is that compiler does not know where to find header file hdf5.h.
First, let’s check it exists, enter the following command, we should see like this:

➜  $: ls /usr/lib/x86_64-linux-gnu/ | grep libhdf5_serial
...
libhdf5_serial.so.10.1.0
libhdf5_serialhl_fortran.so.10.0.2
...

We can find two files like above, please don’t care about version, what we need to just make a soft link for it:

sudo ln -s /usr/lib/x86_64-linux-gnu/libhdf5_serial.so.10.1.0 /usr/lib/x86_64-linux-gnu/libhdf5.so
sudo ln -s /usr/lib/x86_64-linux-gnu/libhdf5_serial_hl.so.10.0.2 /usr/lib/x86_64-linux-gnu/libhdf5_hl.so

We need to edit Makefile.conf to tell compiler where to find hdf5.h

vim Makefile.config

Change lines 95~96 like this:

# Whatever else you find you need goes here.
INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include /usr/include/hdf5/serial
LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib /usr/lib/x86_64-linux-gnu/hdf5/serial

And ok now, try to compile again.

4.4 liblmdb not found

Error description:

In file included from src/caffe/util/db.cpp:3:0:
./include/caffe/util/db_lmdb.hpp:8:18: fatal error: lmdb.h: No such file or directory
compilation terminated.
Makefile:581: recipe for target '.build_release/src/caffe/util/db.o' failed
make: *** [.build_release/src/caffe/util/db.o] Error 1

Fix it:

sudo apt install liblmdb-dev -y

4.5 opencv3 not found

Error description:

/usr/bin/ld: cannot find -lopencv_imgcodecs
/usr/bin/ld: cannot find -lcblas
/usr/bin/ld: cannot find -latlas
collect2: error: ld returned 1 exit status
Makefile:572: recipe for target '.build_release/lib/libcaffe.so.1.0.0' failed
make: *** [.build_release/lib/libcaffe.so.1.0.0] Error 1

Please see opencv 3 installation tutorial:
https://www.learnopencv.com/install-opencv3-on-ubuntu/

4.6 numpy not found

Error description:

CXX/LD -o python/caffe/_caffe.so python/caffe/_caffe.cpp
python/caffe/_caffe.cpp:10:31: fatal error: numpy/arrayobject.h: No such file or directory
compilation terminated.
Makefile:501: recipe for target 'python/caffe/_caffe.so' failed
make: *** [python/caffe/_caffe.so] Error 1

First, check if you have installed numpy:

sudo pip install numpy

If still error, this seems to be a configuration file problem, we need to edit Makefile.config from

PYTHON_INCLUDE := /usr/include/python2.7 \
    /usr/lib/python2.7/dist-packages/numpy/core/include

to this:

PYTHON_INCLUDE := /usr/include/python2.7 \
    /usr/local/lib/python2.7/dist-packages/numpy/core/include

5.Reference


本文出自 夏日小草,转载请注明出处:http://homeway.me/2018/01/25/setup-caffe-for-deep-learning/

by grasses 2018.01.25

Fork me on GitHub