How to debug Linux Init Scripts (in Ubuntu) which fail at bootime

How to debug Linux Init Scripts (Ubuntu).

Often you have the problem that you hacked an init script or have to modify an existing one which will work whenever you manually start it but the process seems not be there after rebooting.

The problem is that your system is not in normal “user mode” during bootime but uses a slim version of your system. For example in most cases for me it is always that my script relies on environment variables such those I usally set in /etc/environment which are loaded after executing the init scripts.

But here is my actual troubleshooting for init scripts which dont work at bootime (manually executing often works):

1) you havent set up any runlevels for your script (solution: update-rc.d)

2) you havent made them executable (solution: chmod +x)

3) your script relies on environment variables which are not set at this stage of bootime

4) your script also does not work at manually running – this meens you have general problems with it – script ERRORS -use bashdb tool

So what to do if it does not run at bootime but runs manually?

Answer: simulate a bootime environment. You can do this by using the env command [1]



cd /

env -i LANG="$LANG" PATH="$PATH" TERM="$TERM" /etc/init.d/daemon start



this simulates the absolut exact situation which are at boottime!!!!

But often if you run modern scripts it does not output anything also you run it in a bootime-like environment.

So what I usally do I use the bashdebugger bashdb tool [2]. In Ubuntu it is easy to install because it is in the repository under “bashdb”.

So now I run the full thing like



cd /

env -i LANG="$LANG" PATH="$PATH" TERM="$TERM" bashdb /etc/init.d/daemon start



and it works like a charme to see whats going on.

Often now I can trace down errors but in some cases it was not possible to get the errornous output. Often this is because the error lies in lines which contain the “start-stop-daemon” program which often modern init scripts rely on. But a solution is near [3]. Just single step to the line in bashdb which contains the start-stop-daemon execution and print it out in the shell e.g. the line is called:



"start-stop-daemon -S -p/var/run/jetty.pid -cjetty -d/path/to/solr -b -m -a /usr/bin/java -- -Dsolr.solr.home=/path/to/solr/solr -Djetty.logs=/path/to/solr/logs -Djetty.home=/path/to/solr -Djava.io.tmpdir=/tmp -jar /path/to/solr/start.jar --daemon"



so in bashdb i hop to the line, hold on at there and use the command “x” :



x start-stop-daemon -S -p"$JETTY_PID" $CH_USER -d"$JETTY_HOME" -b -m -a "$JAVA" -- "${RUN_ARGS[@]}" --daemon



and it will print me out the translated string code:



sudo start-stop-daemon -S -p/var/run/jetty.pid -cjetty -d/path/to/solr -v -b -m -a /usr/bin/java -- -Dsolr.solr.home=/path/to/solr/solr -Djetty.logs=/path/to/solr/logs -Djetty.home=/path/to/solr -Djava.io.tmpdir=/tmp -jar /path/to/solr/start.jar --daemon



so I usally just execut this line then in the shell but often I cannot see any output again:-(. But using this blogpost here [3] I see that I have to get rid of the -b parameter in order to see the output.



sudo start-stop-daemon -S -p/var/run/jetty.pid -cjetty -d/path/to/solr -v -m -a /usr/bin/java -- -Dsolr.solr.home=/path/to/solr/solr -Djetty.logs=/path/to/solr/logs -Djetty.home=/path/to/solr -Djava.io.tmpdir=/tmp -jar /path/to/solr/start.jar --daemon



So in my case the JAVA_HOME was set in /etc/environment but could not be seen by this script so it did not know where JAVA_HOME was. So I putted the JAVA_HOME path in the top of the init script and everything worked fine then.

[1] http://stackoverflow.com/questions/1421607/erlang-daemon-init-d-script-fails-to-start

[2] http://bashdb.sourceforge.net/

[3] http://greenash.net.au/thoughts/2011/02/solr-jetty-and-daemons-debugging-jettysh/

Installing jetty hightide 7.4.5 under Ubuntu 10.04.3 LTS (Lucid Lynx) with port 80 support (setuid)

Installing jetty hightide 7.4.5 under Ubuntu 10.04.3 LTS (Lucid Lynx) with port 80 support (setuid)

Since I needed an javax.servlet container on our brand new vServer setup and tomcat needs too much resources in my opinion
I wanted to install jetty on this clean ubuntu LTS system.
So I installed jetty6 from package with “apt-get install jetty”. After a bit of configuring the /etc/default/jetty it was
working like a charme but I couldn’t get it run on port 80 because my vServer kernel (openvz) did not support NAT out of the box
for port forwarding and also I did not want to install apache2 for port forwarding because apache2 would also be needing some amount
of memory and since our resources are limited and I need the full power/memory for all the other server applications (mysql etc.) I searched
for a alternative. So jetty hightide to the rescue!
Jetty 7 hightide has a new feature called setuid so one can run the webserver with normal user under port 80!!!
Since I did not find any useful FULL description on how to get it to run I decided to write my own. My basic idea comes from website [1] and
since I needed to find out why things were not working I found this beautiful blog entry [2]
So here are my details to install the latest jetty 7 under Lucid Lynx server:


#login with your root account on server
#first lets make an update
apt-get update
#ok my openvz system did not even had these "standard" programs :-D...
#install java
# open /etc/apt/sources.list and put in:
deb http://archive.canonical.com/ubuntu lucid partner
deb-src http://archive.canonical.com/ubuntu lucid partner
#so we can install java
apt-get install less wget unzip sun-java6-jdk
#make your shell aware of java: open /etc/environment and alter line:PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games"
#put in:
PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/lib/jvm/java-6-sun"
#add the line:
JAVA_HOME=/usr/lib/jvm/java-6-sun

#download latest jetty hightide from codehaus

wget http://dist.codehaus.org/jetty/jetty-hightide-7.4.5/jetty-hightide-7.4.5.v20110725.zip
unzip jetty-hightide-7.4.5.v20110725.zip
#copy the start script to the init scripts so it will later run on server startup
cp jetty-hightide-7.4.5.v20110725/bin/jetty.sh /etc/init.d/jetty
#move the jetty to system dir
mkdir /opt/jetty
mv jetty-hightide-7.4.5.v20110725/* /opt/jetty
#add a user for security reasons one should not run as root
useradd jetty
#change permissions
chown -R jetty /opt/jetty
chmod -R ugo+rw /opt/jetty
mkdir -p /var/log/jetty
chown jetty /var/log/jetty -R
#create an jetty config file
touch /etc/default/jetty
vi /etc/default/jetty

#set in this file
DEBUG=1
JAVA=$JAVA_HOME/bin/java
JETTY_HOME=/opt/jetty
JETTY_LOGS=/var/log/jetty
JETTY_PORT=8080 #we will change this later
JETTY_USER=jetty #we will change this later

#add t server startup init levels
update-rc.d jetty defaults

#ok after this jetty is ready to start “/etc/init.d/jetty start” so test it under port 8080 before proceeding

Now to the second part: enable port 80 setuid feature
#move the seuid jar to a different location since it is at the wrong one in the hightide package (see [5] for a why)
cp /opt/jetty/lib/setuid/jetty-setuid-java-7.4.5.v20110725.jar /opt/jetty/lib/ext/


#open /etc/default/jetty again and add the following line:
JETTY_SETUID=/opt/jetty/etc/jetty-setuid.xml
#change the following line:
JETTY_PORT=8080 => JETTY_PORT=80
#this is f**** crucial and took me a long time: you have to start jetty now as "root" user but it will actually set
#the jetty user as owner of the jetty server...without setting to root the setuid feature will not work
JETTY_USER=root

#now open /etc/init.d/jetty
#after those lines:

#####################################################
# Add jetty properties to Java VM options.
#####################################################
JAVA_OPTIONS+=("-Djetty.home=$JETTY_HOME" "-Djava.io.tmpdir=$TMPDIR")

[ -f "$JETTY_HOME/etc/start.config" ] && JAVA_OPTIONS=("-DSTART=$JETTY_HOME/etc/start.config" "${JAVA_OPTIONS[@]}")

#insert the following lines:

##################################################
# enable setuid
##################################################
if [ -z "$JETTY_SETUID" ]
then
JETTY_SETUID=
fi

#change the line: RUN_ARGS=(${JAVA_OPTIONS[@]} -jar “$JETTY_START” $JETTY_ARGS $”${CONFIGS[@]}”) to

RUN_ARGS=(${JAVA_OPTIONS[@]} -jar "$JETTY_START" $JETTY_SETUID $JETTY_ARGS $"${CONFIGS[@]}")

#now go back to the shell
id jetty
#write out for uid for example mine was: uid=1000(jetty) gid=1000(jetty) groups=1000(jetty)
vi /opt/jetty/etc/jetty-setuid.xml
#insert following code (replace USERID with uid from the command above)
#actually the original description on jetty homepage [3]does no longer work for hightide (was written for jetty6)
#one has to change the XML attribute id from tag Configure from “Server” to “org.eclipse.jetty.server.Server”
#otherwise there will be errors (thanks the post on [4])


<Configure id="org.eclipse.jetty.server.Server" class="org.mortbay.setuid.SetUIDServer">
<Set name="uid">UID</Set>
</Configure>

#now change the following lines using your actual uid of user jetty with the lines above:

<Configure id="org.eclipse.jetty.server.Server" class="org.mortbay.setuid.SetUIDServer">
<Set name="uid">1000</Set>
</Configure>

chown jetty /opt/jetty/etc/jetty-setuid.xml

restart jetty and then port80 will work (/etc/init.d/jetty restart)
You can make proof of this by going to the stderr logfile less /var/log/jetty/20xx_xx_xx.stderrout.log after you restarted the server and there you will see:

2011-08-03 10:26:18.478:INFO::Started SelectChannelConnector@0.0.0.0:80 STARTING
2011-08-03 10:26:18.479:INFO::Setting UID=1000
2011-08-03 10:26:18.487:INFO::jetty-7.4.5.v20110725

Note:
In any case there are errors always “tail” the “/var/log/jetty/” start.out and error log files. Also for some errors produced by startup script
its best to debug the jetty init script, see [2] for a very good guideline to do so!

Sources:
[1]http://webcache.googleusercontent.com/search?q=cache:-YW2xPMZYk0J:ubuntuforums.org/showthread.php%3Ft%3D466143+jetty+init+script&cd=1&hl=de&ct=clnk&gl=de&client=firefox-a&source=www.google.de
[2]http://greenash.net.au/thoughts/2011/02/solr-jetty-and-daemons-debugging-jettysh/
[3]http://docs.codehaus.org/display/JETTY/port80
[4]http://dev.eclipse.org/mhonarc/lists/jetty-users/msg00787.html
[5]http://groups.google.com/group/cometd-users/browse_thread/thread/8ae7677f60c5100b