You may want to try resetting the ports on any slow servers you have. This can be done by using the commands:
ethtool -s eth0 autoneg on
mii-tool -r
ethtool -s eth0 speed 100 duplex full
-----------------------
To set the interface speed, duplex or auto negotiation on Linux system boot up (make settings permanent), you need edit /etc/sysconfig/network-scripts/ifcfg-eth0 file for eth0 interface. This file used by Red Hat enterprise Linux, Fedora core, Cent Os etc.
Open the file:
# vi /etc/sysconfig/network-scripts/ifcfg-eth0
Append following line or add the following line
ETHTOOL_OPTS=”speed 100 duplex full autoneg off”
Save and close the file.
Finally restart the network service
# /etc/init.d/network restart
You can check the current network interface speed using the below command
# /usr/sbin/ethtool eth0 |grep Speed
----------
Linux Force or restart network card auto-negotiation with ethtool
by Vivek Gite · 0 comments
ethtool or mii-tool utility checks or sets the status of a network interface’s Media Independent Interface (MII) unit. Most fast ethernet adapters use an MII to autonegotiate link speed and duplex setting.
Due to software or hardware (network switch or Ethernet card ) error it may be possible that you need to restarts auto-negotiation on the specified ethernet device.
Restart autonegotiation
You don’t have to reboot Linux box, all you have to do is type the following command :
# ethtool -r eth0
Or you can use mii-tool (outdated, use ethtool only)
# mii-tool -r eth0
Output:
# tail -f /var/log/messages
Jul 16 09:34:25 smtp1 kernel: e1000: eth0: e1000_watchdog_task: NIC Link is Up 10 Mbps Full Duplex
Jul 16 09:34:25 smtp1 kernel: e1000: eth0: e1000_watchdog_task: 10/100 speed: disabling
Monday, April 19, 2010
Monday, February 15, 2010
kill command usage
Howto: Linux kill and logout users
by Vivek Gite · 18 comments
Yet another newbie question that suggests people love to kill and show their power to rest of the world ;)
There is a package called procps. It includes various useful (read as nifty) utilities. One of such utility is skill which is responsible to send a signal to users and process such as:
* Halt user terminal
* Kill user and logout
Also note that these are utilities to browse the /proc filesystem, which is not a real file system but a way for the kernel to provide information about the status of entries in its process table.
Task: How to halt/stop user called didi
Type the skill command as follows:
# skill -STOP -u didi
You muse be root to stop other users.
Task: How to resume already halted user called didi
Send CONT single to user didi, type the following command:
# skill -CONT -u didi
Task: How to kill and logout user called didi
You can send KILL single, type the following command:
# skill -KILL -u didi
Task: Kill and logout all users
The ultimate command to kill and logout all users, type the following command:
# skill -KILL -v /dev/pts/*
Please note that send users warning message or logout notice using wall command to avoid data loss.
Reference from
http://www.cyberciti.biz/tips/howto-linux-kill-and-logout-users.html
by Vivek Gite · 18 comments
Yet another newbie question that suggests people love to kill and show their power to rest of the world ;)
There is a package called procps. It includes various useful (read as nifty) utilities. One of such utility is skill which is responsible to send a signal to users and process such as:
* Halt user terminal
* Kill user and logout
Also note that these are utilities to browse the /proc filesystem, which is not a real file system but a way for the kernel to provide information about the status of entries in its process table.
Task: How to halt/stop user called didi
Type the skill command as follows:
# skill -STOP -u didi
You muse be root to stop other users.
Task: How to resume already halted user called didi
Send CONT single to user didi, type the following command:
# skill -CONT -u didi
Task: How to kill and logout user called didi
You can send KILL single, type the following command:
# skill -KILL -u didi
Task: Kill and logout all users
The ultimate command to kill and logout all users, type the following command:
# skill -KILL -v /dev/pts/*
Please note that send users warning message or logout notice using wall command to avoid data loss.
Reference from
http://www.cyberciti.biz/tips/howto-linux-kill-and-logout-users.html
Thursday, January 21, 2010
Thursday, January 14, 2010
LOG details.
Here are some basic log files, their locations & a brief description which can be found in Linux distros [mostly in cPanel ]:
General:
Quote:
cPanel/WHM Initial Installation Errors:
Location : /var/log/cpanel*install*
Description : These log files contain cPanel installation logs & should be referenced first for any issues resulting from new cPanel installations..
Quote:
Cpanel/WHM Service Status Logs:
Location : /var/log/chkservd.log
Description :The service monitoring demon (chkservd) logs all service checks here. Failed service are represented with a [-] and active services are represented by [+].
Quote:
Cpanel/WHM Accounting Logs:
Location : /var/cpanel/accounting.log
Description : Contains a list of accounting functions performed through WHM, including account removal and creation..
cPanel/WHM Specific Requests and Errors:
Quote:
cPanel error logs:
Location : /usr/local/cpanel/logs/error_log
Description : cPanel logs any error it incurs here. This should be checked when you encounter errors or strange behavior in cPanel/WHM…
Quote:
cPanel License Error Logs:
Location : /usr/local/cpanel/logs/license_log
Description : All license update attempts are logged here. If you run into any errors related to license when logging in, check here.
Quote:
Stats Daemon Logs:
Location : /usr/local/cpanel/logs/stats_log
Description : The stats daemon (cpanellogd) logs the output from all stats generators (Awstats, Webalizer, Analog) here.
Quote:
Client Information, Requested URL Logs:
Location : /usr/local/cpanel/logs/access_log
Description : General information related to access cPanel requests is logged here.
Quote:
cPanel/WHM Update Logs:
Location : /var/cpanel/updatelogs/update-[TIMESTAMP].log
Description : Contains all output from each cPanel update [upcp]. It’s named with the timestamp at which the upcp process was initiated..
Quote:
Bandwidth Logs:
Location : /var/cpanel/bandwidth
Description : Files contain a list of the bandwidth history for each account. Each named after their respective user.
Quote:
Tailwatchd [New]:
Location : /usr/local/cpanel/logs/tailwatchd_log
Description : Logs for daemon configuired under tailwatchd ie. cPBandwd, Eximstats, Antirelayd.
Apache Logs:
Quote:
General Error and Auditing Logs:
Location : /usr/local/apache/logs/error_log
Description : All exceptions caught by httpd along with standard error output from CGI applications are logged here..
The first place you should look when httpd crashes or you incur errors when accessing website.
Quote:
Apache SuExec Logs:
Location : /usr/local/apache/logs/suexec_log
Description : Auditing information reported by suexec each time a CGI application is executed. Useful for debugging internal server errors, with no relevant information being reported to the Apache error_log, check here for potential suexec policy violations…
Quote:
Domain Access Logs:
Location : /usr/local/apache/domlogs/domain.com
Description : General access log file for each domain configured with cPanel.
Quote:
Apache Access Logs:
Location : /usr/local/apache/logs/access_log
Description : Complete web server access log records all requests processed by the server.
Exim :
Quote:
Message Reception and Delivery:
Location : /var/log/exim_mainlog or /var/log/exim/mainlog
Description : Receives an entry every time a message is received or delivered.
Quote:
Exim ACLs/Policies based RejectLog :
Location : /var/log/exim_rejectlog
Description : An entry is written to this log every time a message is rejected based on either ACLs or other policies eg: aliases configured to :fail
Quote:
Unexpected or Fatal Errors:
Location : /var/log/exim_paniclog
Description : Logs any entries exim doesn’t know how to handle. It’s generally a really bad thing when log entries are being written here, and they should be properly investigated..
Quote:
IMAP/POP/SpamAssassin General Logging and Errors:
Location : /var/log/maillog & /var/log/messages
Description : The IMAP, POP, and SpamAssassin services all log here. This includes all general logging information (login attempts, transactions, spam scoring), along with fatal errors.
FTP:
Quote:
FTP Logins and General Errors:
Location : /var/log/messages
Description : General information and login attempts are logged here..
Quote:
FTP Transactions logging:
Location : /var/log/xferlog or /var/log/messages
Description : Is a symbolic link in most cases to /usr/local/apache/domlogs/ftpxferlog, which contains a history of the transactions made by FTP users…
MySQL:
Quote:
MySQL General Information and Errors :
Location : /var/lib/mysql/$(hostname).err
Description : This path could vary, but is generally located in /var/lib/mysql. Could also be located at /var/log/mysqld.log
Security:
Quote:
Authentication attempts:
Location : /var/log/secure
Description : Logs all daemons which requires PAM Authentication.
Quote:
Tracking all Bad Logins and Logouts:
Location : /var/log/btmp
Description : Log of all attempted bad logins to the system. Accessed via the lastb command..
Quote:
Tracking all Logins and Logouts:
Location : /var/log/wtmp
Description : The wtmp file records all logins and logouts.
Quote:
Last Logins:
Location : /var/log/lastlog
Description : Database times of previous user logins. The lastlog file is a database which contains info on the last login of each user.
Quote:
WebDav or WebDisk Log :
Location : /usr/local/cpanel/logs/cpdavd_error_log
Description : The cpdavd daemon is “WebDav” (better known as “WebDisk”) which was introduced in cPanel 11 to allow users to mount their home directory on their personal computer, always having access to the files and content…
Quote:
Cphulkd Logs:
Location : /usr/local/cpanel/logs/cphulkd_errors.log
Description : cPHulk Brute Force Protection prevents malicious forces from trying to access your server’s services by guessing the login password for that service….
It blacklists IPs that it thinks are trying to run a brute force attack.
Quote:
Failure Logging:
Location : /var/log/faillog
Description : Faillog formats the contents of the failure log from /var/log/faillog database. It also can be used for maintains failure counters and limits. Run faillog without arguments display only list of user faillog records who have ever had a login failure.
Quote:
Startup/Boot, Kernel & Hardware error messages :
Location : /var/log/dmesg
Description : dmesg is a “window” into the kernels ring-buffer. It’s a message buffer of the kernel. The content of this file is referred to by the dmesg command. It shows bootlog and the hardware errors..
General:
Quote:
cPanel/WHM Initial Installation Errors:
Location : /var/log/cpanel*install*
Description : These log files contain cPanel installation logs & should be referenced first for any issues resulting from new cPanel installations..
Quote:
Cpanel/WHM Service Status Logs:
Location : /var/log/chkservd.log
Description :The service monitoring demon (chkservd) logs all service checks here. Failed service are represented with a [-] and active services are represented by [+].
Quote:
Cpanel/WHM Accounting Logs:
Location : /var/cpanel/accounting.log
Description : Contains a list of accounting functions performed through WHM, including account removal and creation..
cPanel/WHM Specific Requests and Errors:
Quote:
cPanel error logs:
Location : /usr/local/cpanel/logs/error_log
Description : cPanel logs any error it incurs here. This should be checked when you encounter errors or strange behavior in cPanel/WHM…
Quote:
cPanel License Error Logs:
Location : /usr/local/cpanel/logs/license_log
Description : All license update attempts are logged here. If you run into any errors related to license when logging in, check here.
Quote:
Stats Daemon Logs:
Location : /usr/local/cpanel/logs/stats_log
Description : The stats daemon (cpanellogd) logs the output from all stats generators (Awstats, Webalizer, Analog) here.
Quote:
Client Information, Requested URL Logs:
Location : /usr/local/cpanel/logs/access_log
Description : General information related to access cPanel requests is logged here.
Quote:
cPanel/WHM Update Logs:
Location : /var/cpanel/updatelogs/update-[TIMESTAMP].log
Description : Contains all output from each cPanel update [upcp]. It’s named with the timestamp at which the upcp process was initiated..
Quote:
Bandwidth Logs:
Location : /var/cpanel/bandwidth
Description : Files contain a list of the bandwidth history for each account. Each named after their respective user.
Quote:
Tailwatchd [New]:
Location : /usr/local/cpanel/logs/tailwatchd_log
Description : Logs for daemon configuired under tailwatchd ie. cPBandwd, Eximstats, Antirelayd.
Apache Logs:
Quote:
General Error and Auditing Logs:
Location : /usr/local/apache/logs/error_log
Description : All exceptions caught by httpd along with standard error output from CGI applications are logged here..
The first place you should look when httpd crashes or you incur errors when accessing website.
Quote:
Apache SuExec Logs:
Location : /usr/local/apache/logs/suexec_log
Description : Auditing information reported by suexec each time a CGI application is executed. Useful for debugging internal server errors, with no relevant information being reported to the Apache error_log, check here for potential suexec policy violations…
Quote:
Domain Access Logs:
Location : /usr/local/apache/domlogs/domain.com
Description : General access log file for each domain configured with cPanel.
Quote:
Apache Access Logs:
Location : /usr/local/apache/logs/access_log
Description : Complete web server access log records all requests processed by the server.
Exim :
Quote:
Message Reception and Delivery:
Location : /var/log/exim_mainlog or /var/log/exim/mainlog
Description : Receives an entry every time a message is received or delivered.
Quote:
Exim ACLs/Policies based RejectLog :
Location : /var/log/exim_rejectlog
Description : An entry is written to this log every time a message is rejected based on either ACLs or other policies eg: aliases configured to :fail
Quote:
Unexpected or Fatal Errors:
Location : /var/log/exim_paniclog
Description : Logs any entries exim doesn’t know how to handle. It’s generally a really bad thing when log entries are being written here, and they should be properly investigated..
Quote:
IMAP/POP/SpamAssassin General Logging and Errors:
Location : /var/log/maillog & /var/log/messages
Description : The IMAP, POP, and SpamAssassin services all log here. This includes all general logging information (login attempts, transactions, spam scoring), along with fatal errors.
FTP:
Quote:
FTP Logins and General Errors:
Location : /var/log/messages
Description : General information and login attempts are logged here..
Quote:
FTP Transactions logging:
Location : /var/log/xferlog or /var/log/messages
Description : Is a symbolic link in most cases to /usr/local/apache/domlogs/ftpxferlog, which contains a history of the transactions made by FTP users…
MySQL:
Quote:
MySQL General Information and Errors :
Location : /var/lib/mysql/$(hostname).err
Description : This path could vary, but is generally located in /var/lib/mysql. Could also be located at /var/log/mysqld.log
Security:
Quote:
Authentication attempts:
Location : /var/log/secure
Description : Logs all daemons which requires PAM Authentication.
Quote:
Tracking all Bad Logins and Logouts:
Location : /var/log/btmp
Description : Log of all attempted bad logins to the system. Accessed via the lastb command..
Quote:
Tracking all Logins and Logouts:
Location : /var/log/wtmp
Description : The wtmp file records all logins and logouts.
Quote:
Last Logins:
Location : /var/log/lastlog
Description : Database times of previous user logins. The lastlog file is a database which contains info on the last login of each user.
Quote:
WebDav or WebDisk Log :
Location : /usr/local/cpanel/logs/cpdavd_error_log
Description : The cpdavd daemon is “WebDav” (better known as “WebDisk”) which was introduced in cPanel 11 to allow users to mount their home directory on their personal computer, always having access to the files and content…
Quote:
Cphulkd Logs:
Location : /usr/local/cpanel/logs/cphulkd_errors.log
Description : cPHulk Brute Force Protection prevents malicious forces from trying to access your server’s services by guessing the login password for that service….
It blacklists IPs that it thinks are trying to run a brute force attack.
Quote:
Failure Logging:
Location : /var/log/faillog
Description : Faillog formats the contents of the failure log from /var/log/faillog database. It also can be used for maintains failure counters and limits. Run faillog without arguments display only list of user faillog records who have ever had a login failure.
Quote:
Startup/Boot, Kernel & Hardware error messages :
Location : /var/log/dmesg
Description : dmesg is a “window” into the kernels ring-buffer. It’s a message buffer of the kernel. The content of this file is referred to by the dmesg command. It shows bootlog and the hardware errors..
Sunday, January 10, 2010
kernel compile details
http://www.linuxchix.org/content/courses/kernel_hacking/lesson9
http://www.linuxdocs.org/HOWTOs/Kernel-HOWTO-7.html
refer it from 1 page
http://www.linuxdocs.org/HOWTOs/Kernel-HOWTO-7.html
refer it from 1 page
Saturday, January 2, 2010
web server +site speed up
http://www.catswhocode.com/blog/speed-up-your-website-quick
http://www.askapache.com/htaccess/speed-up-sites-with-htaccess-caching.html
http://www.serverwatch.com/tutorials/article.php/3436911/Optimizing-Apache-Server-Performance.htm
https://calomel.org/save_web_bandwidth.html
http://www.webpagetest.org/
http://www.askapache.com/htaccess/speed-up-sites-with-htaccess-caching.html
http://www.serverwatch.com/tutorials/article.php/3436911/Optimizing-Apache-Server-Performance.htm
https://calomel.org/save_web_bandwidth.html
http://www.webpagetest.org/
kernel compile
http://www.cyberciti.biz/tips/compiling-linux-kernel-26.html
http://www.cyberciti.biz/tips/compiling-linux-kernel-module.html
http://www.cyberciti.biz/tips/build-linux-kernel-module-against-installed-kernel-source-tree.html
http://www.sysdesign.ca/guides/linux_kernel.html
http://www.cyberciti.biz/tips/compiling-linux-kernel-module.html
http://www.cyberciti.biz/tips/build-linux-kernel-module-against-installed-kernel-source-tree.html
http://www.sysdesign.ca/guides/linux_kernel.html
Friday, January 1, 2010
lsof+strace
The Basics
This section covers briefly the terminology that you need to familiarize yourself with in order to understand the results of the operations described later in the article. Do not worry, it is not more difficult when answering to Windows Vista UAC question ;)
System Call (a.k.a. "syscall")
System Call is a mechanism for the application to make a call to the operating system. The OS usually has a limited number of the "entry points" that can be used for the basic and fundamental tasks like file I/O, memory allocation, networking etc. For example, most of the UNIX systems have only 1-3 different system calls to open a file (although various libraries offer dozens of ways to deal with a file, they all finally make the same system call). The important thing about the system call that you need to understand is that there are few of them and they are the only way for any program to interact with the OS.
Process ID (PID)
Any process in the UNIX OSes is defined by its unique numeric PID, even if the process runs several threads. From the moment you start a program till the moment it is finished it maintains the same PID. However, some processes can span other processes (usually by using one of the fork() syscalls).
File Handle
Any I/O operations in UNIX is associated with a file handle which is a positive integer number. It applies to any file on the disk, a network socket etc. A typical operation with a file on the disk, for example, is one of the following:
1. open("filename") -> returns a handle that from this moment will be associated with this open file
2. read(handle, where, how_much)
3. write(handle, from_where, how_much)
4. close(handle) -> removes the association established with open()
Handles are unique only for an individual process. There are well-known handles available by default for any application, such as 0 for the standard input, 1 for standard output and 2 for the standard error stream.
Turn out your pockets, process!
We will start from a tool that can show us a snapshot of what is happening right now with our system. There is an excellent tool that is available for many UNIX systems called "lsof" ("LiSt Open Files"). You can find more information about this tool here. Most of the Linux distributions and Mac OS X have this tool included.
Here is what this tool can do:
* Show the list of files and other resources open by a selected process or by all processes in the system
* Show the list of processes using a particular resource (like a TCP socket, for example)
You can do various types of sorting, filtering and formatting for getting exactly the results you want. The program has about 3 dozens of command line options, we will try a couple of typical and most representative examples now. Note: most of these examples require root access to the system (or you will be limited to seeing only the information that is related to your processes).
How to see the list of files opened by a running process
For example, we would like to see which resources are currently used by Apache HTTP server. First, we need to identify the process we are interested in:
[root@localhost]$ ps ax | grep httpd | grep -v grep
2383 ? Ss 0:00 /usr/sbin/httpd
2408 ? S 0:00 /usr/sbin/httpd
2409 ? S 0:00 /usr/sbin/httpd
2410 ? S 0:00 /usr/sbin/httpd
2411 ? S 0:00 /usr/sbin/httpd
2412 ? S 0:00 /usr/sbin/httpd
2414 ? S 0:00 /usr/sbin/httpd
2415 ? S 0:00 /usr/sbin/httpd
2416 ? S 0:00 /usr/sbin/httpd
Hmm...which one I am interested in? Unfortunately, there is no universal recipe to answer this question. However...lets look at the problem from another perspective. At some point I have mentioned that when you start a program, the OS creates a process. And a process can spawn other processes. So, we can assume that there may be a tree of sub-processed related to one parent. Lets see how we can get this information (this command will work on both Linux and OS X):
[root@localhost]$ ps axo pid,ppid,command | egrep '(httpd)|(PID)' |
grep -v grep
2383 1 /usr/sbin/httpd
2408 2383 /usr/sbin/httpd
2409 2383 /usr/sbin/httpd
2410 2383 /usr/sbin/httpd
2411 2383 /usr/sbin/httpd
2412 2383 /usr/sbin/httpd
2414 2383 /usr/sbin/httpd
2415 2383 /usr/sbin/httpd
2416 2383 /usr/sbin/httpd
PPID is simply Parent PID, i.e. Parent Process ID. As you can see, most of these processes have the same PPID (2383) and there is one process with the PID of 2383. This must be the parent process. If you are curious to find out who is its parent, I will let you to do this exercise yourself.
Linux has another fancy tool called "pstree". We could use it to get the same kind of information:
[root@localhost]$ pstree -p | grep httpd
|-httpd(2383)-+-httpd(2408)
| |-httpd(2409)
| |-httpd(2410)
| |-httpd(2411)
| |-httpd(2412)
| |-httpd(2414)
| |-httpd(2415)
| `-httpd(2416)
Looks better and quite self-explaining, isn't it? OK, back to the original question. Now we know that there are certain relationships with these "httpd"s. So, our target will be the process 2383.
In its simples form, "lsof" can be called like this:
[root@localhost]$ lsof -p 2383
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
httpd 2383 root cwd DIR 3,6 1024 2 /
httpd 2383 root rtd DIR 3,6 1024 2 /
httpd 2383 root txt REG 3,8 328136 261136 /usr/sbin/httpd
httpd 2383 root mem REG 3,6 130448 8167 /lib64/ld-2.5.so
httpd 2383 root DEL REG 0,8 8072 /dev/zero
httpd 2383 root mem REG 3,6 615136 79579 /lib64/libm-2.5.so
httpd 2383 root mem REG 3,6 117656 8186 /lib64/libpcre.so.0.0.1
httpd 2383 root mem REG 3,6 95480 8213 /lib64/libselinux.so.1
...
httpd 2383 root mem REG 3,8 46160 1704830 /usr/lib64/
php/modules/ldap.so
httpd 2383 root DEL REG 0,8 8120 /dev/zero
httpd 2383 root DEL REG 0,8 8122 /dev/zero
httpd 2383 root 0r CHR 1,3 1488 /dev/null
httpd 2383 root 1w CHR 1,3 1488 /dev/null
httpd 2383 root 2w REG 3,12 1077 4210 /var/log/
httpd/error_log
httpd 2383 root 3r CHR 1,9 1922 /dev/urandom
httpd 2383 root 4u IPv6 8052 TCP *:http (LISTEN)
httpd 2383 root 5u sock 0,5 8053 can't identify protocol
httpd 2383 root 6u IPv6 8057 TCP *:https (LISTEN)
httpd 2383 root 7u sock 0,5 8058 can't identify protocol
httpd 2383 root 8r FIFO 0,6 8069 pipe
httpd 2383 root 9w FIFO 0,6 8069 pipe
httpd 2383 root 10w REG 3,12 1077 4210 /var/log/
httpd/error_log
httpd 2383 root 11w REG 3,12 711 4215 /var/log/
httpd/ssl_error_log
httpd 2383 root 12w REG 3,12 0 4138 /var/log/
httpd/access_log
httpd 2383 root 13w REG 3,12 0 4151 /var/log/
httpd/ssl_access_log
httpd 2383 root 14w REG 3,12 0 4152 /var/log/
httpd/ssl_request_log
Looks quite interesting, doesn't it? Now I am going to briefly explain the meaning of each kind of entry (you can get more information from the manual page for "lsof").
COMMAND
Contains first 9 characters of the UNIX command name associated with the process
PID
The process ID. In our example it is simply the PID we have asked the information for
USER
The UNIX user owning the process
FD
File Descriptor number. Actually, it is more than the descriptor I have mentioned before. There are special descriptors identifying various resources in the UNIX system. For more details I recommend to check the man page. As you can see, everything is there: the current working directory (cwd), the process image itself (txt), memory-mapped libraries etc. The most interesting for us are the open files and sockets: you can see the numbers 0..14 followed by a suffix ("w" for writing only, "r" for reading only, "u" for both reading and writing and others). For example, you can see that Apache httpd redirects all its error output (2w) to /var/log/https/error_log. It also has a bunch of other regular files open for writing (handles 10-14, various log files). You can also see that it is listening on two TCP sockets (the port numbers shown are resolved through /etc/services file, you can always use "-n" option to disable any name resolution - useful if you, for example, know the port number and want to look for it specifically). You can see two more sockets (do not be afraid of the message "can't identify...", it is just application-specific protocol), Apache is using them to talk to its sub-processes.
TYPE
Type of the node. There is several dozens of various node types, like "REG" for the regular files, "IPv4" (or "IPv6 for IP version 6) for the Internet sockets, "CHR" for the character devices, "DIR" for the directories etc.
DEVICE
Shows the UNIX device numbers (major, minor) associated with the resource. Another interesting parameter. For example, on my system the numbers 3,12 correspond to:
[root@localhost]$ ls -l /dev/* | egrep '3,\s*12'
brw-r----- 1 root disk 3, 12 Jul 17 19:17 /dev/hda12
[root@localhost]$ mount | grep hda12
/dev/hda12 on /var type ext3 (rw)
This is true, the files are locates on one of the partitions on the first IDE disk mounted at /var.
SIZE
The size of the file or the current offset (in bytes). This number does not make sense for some of the resource types so you see blank values there.
NODE
Node number of the file, protocol identifier (for Internet sockets) etc.
Name
The name of the file or resource, if applicable. The value is quite self-explaining and probably one of the most useful.
Try using this command for other processes in your system. Probably one of the most interesting ones to look at will be your X11 server.
There is a number of filters you can use to see only the information you are interested in. For example, I want to know does my iTunes talk to someone over the network or not. Here is a command I would use something like:
[root@localhost]$ lsof -p 1559 -a -i
Which produces empty answer (exactly what I was hoping for :) ). But if I try my SSH client instead of iTunes:
[root@localhost]$ lsof -p 1586 -a -i
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ssh 1586 nik 3u IPv4 0x40e7394 0t0 TCP macbook:50648
->oberon:ssh (ESTABLISHED)
Summary: you can get a snapshot of any live process using this command. Sometimes it is simpler to use "lsof" (especially on a system you are not familiar with) to find the location of the log file, config files for a service etc.
Who is using that file?
From time to time you may become very curious about who (or at least which process :) ) is using certain resource(s) on your system. There may be a number of reasons: a server that does not start complaining about the port that is already being used, a suspicious network connection that you see when using "netstat" command, inability to unmount a network disk or a external storage unit etc.
A traditional "fuser" command is not always good enough for these purposes. For example, if I want to see all the processes that have any open files under /etc directory. Here is the command that give me the answer:
[root@localhost]$ lsof +D /etc
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
dbus-daem 2206 dbus 5r DIR 3,6 1024 59500 /etc/dbus-1/system.d
acpid 2305 root 6r REG 3,6 236 63639 /etc/acpi/
events/video.conf
avahi-dae 2541 avahi cwd DIR 3,6 1024 63468 /etc/avahi
avahi-dae 2541 avahi rtd DIR 3,6 1024 63468 /etc/avahi
prefdm 2662 root 255r REG 3,6 1465 59322 /etc/X11/prefdm
000-delay 3831 root 255r REG 3,6 577 59446 /etc/cron.
daily/000-delay.cron
When you use "+D path" option, lsof checks every single file under the specified path and shows the information about the processes using it. This command may take some time to execute if you have a lot of files there.
Another situation: a network connection. "-i" option of lsof allows you to search for a specific network connection. The format is following: lsof -i [4|6][protocol][@hostname|hostaddr][:service|port]
"4" or "6" defines the IP protocol version. If you do not use any filters and just use "-i" you will get the list of all connections in your system. Lets find all the UDP connections that we have on a OS X box:
[root@localhost]$ lsof -i UDP
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
launchd 1 root 10u IPv4 0x393ed40 0t0 UDP *:netbios-ns
mDNSRespo 39 root 7u IPv4 0x393e790 0t0 UDP *:mdns
mDNSRespo 39 root 8u IPv6 0x393e6c0 0t0 UDP *:mdns
mDNSRespo 39 root 9u IPv4 0x41556c0 0t0 UDP *:mdns
mDNSRespo 39 root 11u IPv4 0x4155ee0 0t0 UDP *:mdns
mDNSRespo 39 root 12u IPv4 0x393dd00 0t0 UDP macbook:49727
netinfod 40 root 6u IPv4 0x393ee10 0t0 UDP localhost:
netinfo-local
syslogd 41 root 16u IPv4 0x393ec70 0t0 UDP *:49175
Directory 52 root 10u IPv4 0x3fbd1a0 0t0 UDP *:*
Directory 52 root 12u IPv4 0x4155d40 0t0 UDP *:*
ntpd 141 root 5u IPv4 0x393e520 0t0 UDP *:ntp
ntpd 141 root 6u IPv6 0x393e110 0t0 UDP *:ntp
ntpd 141 root 7u IPv6 0x393ead0 0t0 UDP localhost:ntp
ntpd 141 root 8u IPv6 0x393e860 0t0 UDP [fe80:1::1]:ntp
ntpd 141 root 9u IPv4 0x393e5f0 0t0 UDP localhost:ntp
ntpd 141 root 10u IPv6 0x393e450 0t0 UDP [fe80:5::216:
cbff:febf:a08d]:ntp
ntpd 141 root 11u IPv4 0x393e380 0t0 UDP macbook:ntp
prl_dhcpd 147 root 5u IPv4 0x393db60 0t0 UDP *:*
automount 168 root 8u IPv4 0x3fbd000 0t0 UDP localhost:1023
automount 179 root 8u IPv4 0x393e930 0t0 UDP localhost:1022
nmbd 1078 root 0u IPv4 0x393ed40 0t0 UDP *:netbios-ns
nmbd 1078 root 1u IPv4 0x393ed40 0t0 UDP *:netbios-ns
nmbd 1078 root 8u IPv4 0x393ea00 0t0 UDP *:netbios-dgm
nmbd 1078 root 9u IPv4 0x3fbd4e0 0t0 UDP 10.37.129.2:
netbios-ns
nmbd 1078 root 10u IPv4 0x3fbeba0 0t0 UDP 10.37.129.2:
netbios-dgm
nmbd 1078 root 11u IPv4 0x4155380 0t0 UDP macbook:
netbios-ns
nmbd 1078 root 12u IPv4 0x4155450 0t0 UDP macbook:
netbios-dgm
cupsd 1087 root 6u IPv4 0x3fbd340 0t0 UDP *:ipp
Do not be surprised if you do not see anything on your machine - you will not see the information about the processes that do not belong to you unless you are root.
Summary: using lsof with various filters allows you to search through your system and find out the detailed information about any local or network resource used.
Watch your steps
We have learned how to get a snapshot or your UNIX system (or any individual process living in it). But sometimes it is not enough to look at the static information. Dynamic runtime information may answer a lot of other questions - why does the program crash, why does it say "cannot open resource" without providing the resource location, is it accepting the network connection or not etc.
In the fisrt section it was mentioned that the way the UNIX program talks to the operating system (i.e. deals with almost any resource on your computer) is called "system calls". Most of the UNIX kernels, including the Linux and OS X provide a way for the user to see the list of the system calls a process makes in real time, including the parameters the process passes to the kernel. There are different tools on different systems that provide this functionality. On Linux it is called "strace", on BSD (including OS X) there are two toolss working together - "ktrace" and "kdump". We will briefly cover both of them starting from the Linux one.
The really great thing about these tools is that they use the kernel tracing facility in order to work. Thus, they work with ANY program that is running in your system, it does not have to be something you have developed and you do not need the source code of the program in order to understand (in general, of course) what it is doing.
It is important to mention again that just like in case of "lsof" you cannot see anything that you are not supposed to see, i.e. you cannot see what process is doing if it is not yours (or unless you are root user).
The simplest way to run the "strace" would be as: strace command Lets see what a simple well-known like "w" (the command that shows the users logged in and their current activities) does:
[root@localhost]$ strace /usr/bin/w 2>&1 | less
execve("/usr/bin/w", ["/usr/bin/w"], [/* 30 vars */]) = 0
brk(0) = 0x603000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x2aaaaaaab000
uname({sys="Linux", node="oberon.anglab.com", ...}) = 0
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such
file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=103874, ...}) = 0
mmap(NULL, 103874, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2aaaaaaac000
close(3) = 0
open("/lib64/libproc-3.2.7.so", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\
240;`\314"..., 832) = 832
...
open("/proc/uptime", O_RDONLY) = 3
lseek(3, 0, SEEK_SET) = 0
read(3, "7267.86 7230.07\n", 1023) = 16
access("/var/run/utmpx", F_OK) = -1 ENOENT (No such file
or directory)
open("/var/run/utmp", O_RDWR) = -1 EACCES (Permission denied)
open("/var/run/utmp", O_RDONLY) = 4
fcntl(4, F_GETFD) = 0
...
open("/proc", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 3
fstat(3, {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
fcntl(3, F_SETFD, FD_CLOEXEC) = 0
getdents(3, /* 35 entries */, 1024) = 1000
getdents(3, /* 38 entries */, 1024) = 1016
stat("/proc/1", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
open("/proc/1/stat", O_RDONLY) = 4
read(4, "1 (init) S 0 1 1 0 -1 4194560 64"..., 1023) = 216
close(4) = 0
socket(PF_FILE, SOCK_STREAM, 0) = 4
...
fcntl(5, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
alarm(0) = 1
rt_sigaction(SIGALRM, {SIG_DFL}, NULL, 8) = 0
close(5) = 0
write(1, " 21:31:48 up 2:01, 2 users, l"..., 281 21:31:48 up 2:01,
2 users, load average: 0.00, 0.00, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
nick pts/0 192.168.8.18 21:12 9.00s 0.03s 0.03s -bash
nick pts/1 192.168.8.18 21:19 0.00s 0.13s 0.04s strace
/usr/bin
) = 281
exit_group(0) = ?
Process 3947 detached
Only a small part of the long output is shown here. As you can see, everything starts with the "execve" syscall. Which is correct - this is the call that executes a program in UNIX. By the way, for each of these calls you can get more information from the manual pages - just type "man execve" to learn more about the call and its parameters (as you can see the parameters are shown in the trace).And, as it is supposed to be, the execution ends with "exit_group" call with the status 0 (success).
You can see how the program being started loads the shared libraries it depends on. You can see that at some point it tries to open "var/run/utmpx" file but it does not exist (ENOENT). Then it tries to open "/var/run/utmp" file but it is not readable for this program (i.e. for this user) so it gets back (EACCES). Finally this program decides that the only way to find out who is logged in is to go through all the active processes and check where their standard input and output are attached to. In order to do it, "w" readsthe list of files and directories under "/proc" (open - getdents - close). By the way, the number returned by the open() call is the file handle. By using it, you can follow the operations with this file since all read() or write() system calls will use the handle. After getting the list of files, "w" goes over all process directories and reads the virtual files from there to get more information about the processes. And fnally it displays the results by writing them to the file descriptor 1 (standard output, stdout). Very simple and informative.
There is a number of parameters that you can use with "strace" to filter the results and get exactly the information you are looking for. For example, if you are interested only in the network operations, lets see what does "ping" command do (run it as root user):
[root@localhost]$ strace -x -e trace=network,desc ping -c 1 127.0.0.1
...
socket(PF_INET, SOCK_RAW, IPPROTO_ICMP) = 3
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
connect(4, {sa_family=AF_INET, sin_port=htons(1025), sin_addr=
inet_addr("127.0.0.1")}, 16) = 0
getsockname(4, {sa_family=AF_INET, sin_port=htons(1028), sin_addr=
inet_addr("127.0.0.1")}, [16]) = 0
close(4) = 0
setsockopt(3, SOL_RAW, ICMP_FILTER, ~(ICMP_ECHOREPLY|ICMP_DEST_UNREACH|
ICMP_SOURCE_QUENCH|ICMP_REDIRECT|ICMP_TIME_EXCEEDED|
ICMP_PARAMETERPROB),
4) = 0
setsockopt(3, SOL_IP, IP_RECVERR, [1], 4) = 0
setsockopt(3, SOL_SOCKET, SO_SNDBUF, [324], 4) = 0
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [65536], 4) = 0
getsockopt(3, SOL_SOCKET, SO_RCVBUF, [27472947287556096], [4]) = 0
fstat(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
setsockopt(3, SOL_SOCKET, SO_TIMESTAMP, [1], 4) = 0
setsockopt(3, SOL_SOCKET, SO_SNDTIMEO, "\x01\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00"..., 16) = 0
setsockopt(3, SOL_SOCKET, SO_RCVTIMEO, "\x01\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00"..., 16) = 0
ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff418a7c90) = -1
EINVAL (Invalid argument)
sendmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(0),
sin_addr=inet_addr("127.0.0.1")}, msg_iov(1)=[{"\x08\x00
\xc7\x8b\xd6\x10\x00\x01\x86\x16\xa0\x46\x00\x00"..., 64}],
msg_controllen=0, msg_flags=0}, 0) = 64
recvmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(0),
sin_addr=inet_addr("127.0.0.1")}, msg_iov(1)=[{"\x45
\x00\x00\x54\xed\xd5\x00\x00\x40\x01\x8e\xd1\x7f\x00"..., 192}],
msg_controllen=32, {cmsg_len=32, cmsg_level=SOL_SOCKET,
cmsg_type=0x1d /* SCM_??? */, ...}, msg_flags=0}, 0) = 84
write(1, "PING 127.0.0.1 (127.0.0.1) 56(84"..., 106PING
127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.321 ms
) = 106
write(1, "\n", 1
) = 1
write(1, "--- 127.0.0.1 ping statistics --"..., 144---
127.0.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.321/0.321/0.321/0.000 ms
) = 144
Process 4310 detached
"-e" option allows to specify what kind(s) of system calls you are interested in.
As you can see, this program opes 2 network sockets (with the descriptors 3 and 4). One of them is used only to do getsockname() call (do "man getsockname" to get more information on this call) and then it gets closed. The one associated with the descriptor 3 is used to send and receive raw IP traffic (this is the way to send the packets using the protocols different from UDP and TCP). It configures the socket using a number of options and then finally sends an IP packet (sendmsg) to the given destination. You can even see part of the message data ("-x" option requests strace to display the hex numbers instead of non-ASCII strings). If you want to see the complete data you may want to use "-s" option to set the maximum string length you want to be displayed. After sending the message it gets back the response (recvmsg). Then it displays the output of the program.
You can use "-o" option with the file name argument to redirect the output to a text file. Usually when analyzing more complicated examples it is easier to get the trace into a file first and then study this file using text editor or "grep" command.
Mac OS X (like all BSD systems) offers "ktrace" command. In order to get similar result when analyzing the "ping" command we would have to do something like this:
[root@localhost]$ ktrace -f ping.trace -t ci ping -c 1 127.0.0.1
[root@localhost]$ kdump -f ping.trace | less
...
684 ping CALL socket(0x2,0x3,0x1)
684 ping RET socket 3
684 ping CALL getuid
684 ping RET getuid 0
684 ping CALL setuid(0)
684 ping RET setuid 0
684 ping CALL getuid
684 ping RET getuid 0
684 ping CALL getpid
684 ping RET getpid 684/0x2ac
684 ping CALL setsockopt(0x3,0xffff,0x400,0xbffffdc8,0x4)
684 ping RET setsockopt 0
684 ping CALL setsockopt(0x3,0xffff,0x1002,0xbffffdb8,0x4)
684 ping RET setsockopt 0
684 ping CALL setsockopt(0x3,0xffff,0x1001,0xbffffdb8,0x4)
684 ping RET setsockopt 0
684 ping CALL fstat(0x1,0xbffef0c0)
684 ping RET fstat 0
684 ping CALL ioctl(0x1,FIODTYPE,0xbffef08c)
684 ping RET ioctl 0
...
684 ping CALL sendto(0x3,0x5294,0x40,0,0x5260,0x10)
684 ping GIO fd 3 wrote 64 bytes
"\b\08\^R\M-,\^B\0\0\M-T\^X\240F\M-'\M^H\r\0\b
\v\f\r\^N\^O\^P\^Q\^R\^S\^T\^U\^V\^W\^X\^Y\^Z
\^[\^\\^]\^^\^_ !"#$%&'()*+,-./01234567"
684 ping RET sendto 64/0x40
684 ping CALL select(0x4,0xbffffc3c,0,0,0xbffffd90)
684 ping RET select 1
684 ping CALL recvmsg(0x3,0xbffffd24,0)
684 ping GIO fd 3 wrote 84 bytes
"E\0@\0|\M^N\0\0@\^A\0\0\^?\0\0\^A\^?\0\0
\^A\0\0@\^R\M-,\^B\0\0\M-T\^X\240F\M-'\M^H\r\0\b
\v\f\r\^N\^O\^P\^Q\^R\^S\^T\^U\^V\^W\^X\^Y\^Z
\^[\^\\^]\^^\^_ !"#$%&'()*+,-./01234567"
684 ping RET recvmsg 84/0x54
684 ping CALL write(0x1,0x18000,0x39)
684 ping GIO fd 1 wrote 57 bytes
"64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.069 ms
"
684 ping RET write 57/0x39
684 ping CALL sigaction(0x2,0xbffef8c0,0xbffef8f8)
684 ping RET sigaction 0
684 ping CALL sigaction(0xe,0xbffef8c0,0xbffef8f8)
684 ping RET sigaction 0
684 ping CALL write(0x1,0x18000,0x1)
684 ping GIO fd 1 wrote 1 byte
"
"
684 ping RET write 1
684 ping CALL write(0x1,0x18000,0x22)
684 ping GIO fd 1 wrote 34 bytes
"--- 127.0.0.1 ping statistics ---
...
"ktrace" writes the tracing information into a file ("ktrace.out" if the "-f" option is not specified) and the "kdump" tool is used to decode the file into more human-readable format. As you can see the behavior of the ping program on OS X is similar to what we saw on Linux. ktrace/kdump also have some basic filtering capabilities.
Important thing to know about strace is that by default it does not follow the child processes. I.e. if you try to trace a program than spawns other processes, you will not see the operations performed by the child processes. You will see only the fork() calls. In order to "follow" the execution of all child processes you need to use "-f" option.
Tapping the wire
Another nice feature offered by strace (and ktrace on BSD systems) is the ability to attach to any process in the system for a limited time and capture its activities. Sometimes it may help to troubleshoot the network server processes a lot. In order to "attach" to a process you need to use "-p process_ID" option offered by strace. When you are done capturing the information it is enough just to terminate strace by pressing Ctrl+C. Lets try to see what happens when Sendmail accepts an incoming connection (in the given example the number 233 is the process ID of the sendmail process).
[root@localhost]$ strace -f -p 233
And on the second console do:
[root@localhost]$ telnet localhost 25
Trying 127.0.0.1...
Connected to oberon.
Escape character is '^]'.
220 oberon ESMTP Sendmail 8.11.6p2/8.11.6-20030304; Sat, 21 Jul 2007
06:00:02 +0400
quit
221 2.0.0 oberon closing connection
Connection closed by foreign host.
And the press Ctrl+C in the first console. Here is what you will see there (use "-o" option when running strace if you want to write the output to a file):
233 select(5, [4], NULL, NULL, {5, 0}) = 1 (in [4], left {1, 750000})
233 accept(4, {sin_family=AF_INET, sin_port=htons(1226), sin_addr=
inet_addr("127.0.0.1")}, [16]) = 5
233 time([1184983202]) = 1184983202
233 sigprocmask(SIG_BLOCK, [ALRM], []) = 0
233 pipe([6, 7]) = 0
233 sigprocmask(SIG_BLOCK, [CHLD], [ALRM]) = 0
233 fork() = 20953
20953 sigprocmask(SIG_UNBLOCK, [ALRM],
233 getpid(
20953 <... sigprocmask resumed> [ALRM CHLD]) = 0
233 <... getpid resumed> ) = 233
20953 sigprocmask(SIG_UNBLOCK, [CHLD],
233 sigprocmask(SIG_UNBLOCK, [CHLD],
20953 <... sigprocmask resumed> [CHLD]) = 0
233 <... sigprocmask resumed> [ALRM CHLD]) = 0
20953 sigaction(SIGCHLD, {SIG_DFL},
233 close(6
...
20953 open("/etc/hosts", O_RDONLY) = 6
20953 fstat(6, {st_mode=S_IFREG|0644, st_size=963, ...}) = 0
20953 mmap(0, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|0x20, 4294967295, 0)
= 0x11a000
20953 read(6, "#\n# hosts\t\tThis file describe"..., 4096) = 963
20953 close(6) = 0
...
20953 write(3, "<22>Jul 21 06:00:03 sendmail[209"..., 143) = -1 EPIPE
(Broken pipe)
20953 --- SIGPIPE (Broken pipe) ---
20953 close(3) = 0
20953 close(-1) = -1 EBADF (Bad file number)
20953 sigaction(SIGPIPE, {SIG_IGN}, NULL) = 0
20953 sigprocmask(SIG_UNBLOCK, [ALRM], []) = 0
20953 sigprocmask(SIG_BLOCK, NULL, []) = 0
20953 time([1184983203]) = 1184983203
20953 unlink("./dfl6L202l20953") = -1 ENOENT (No such
file or directory)
20953 unlink("./qfl6L202l20953") = -1 ENOENT (No such
file or directory)
20953 unlink("./xfl6L202l20953") = -1 ENOENT (No such
file or directory)
20953 getpid() = 20953
20953 close(7) = 0
20953 setuid(0) = 0
20953 _exit(71) = ?
233 <... select resumed> ) = ? ERESTARTNOHAND (To be restarted)
233 --- SIGCHLD (Child exited) ---
233 wait4(-1, [WIFEXITED(s) && WEXITSTATUS(s) == 71], WNOHANG, NULL)
= 20953
233 wait4(-1, 0xbfffdf60, WNOHANG, NULL) = 0
233 sigprocmask(SIG_BLOCK, [ALRM], []) = 0
233 time([1184983203]) = 1184983203
233 open("/proc/loadavg", O_RDONLY) = 5
233 fstat(5, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
233 mmap(0, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|0x20, 4294967295, 0)
= 0x118000
233 read(5, "0.16 0.09 0.29 3/53 21316\n", 1024) = 26
233 close(5) = 0
233 munmap(0x118000, 4096) = 0
233 sigprocmask(SIG_UNBLOCK, [ALRM], [ALRM]) = 0
233 getpid() = 233
233 select(5, [4], NULL, NULL, {5, 0}
As you can see, sendmail is initially sitting there listening for the incoming connections on the file descriptor 4 (you can use "lsof" to see that it is associated with a TCP socket) - see the second parameter of the select() call. Then the incoming connection comes in and sendmail spawns another process to handle this connection (fork() is not followed by an execve() call because the child process is also sendmail). Then the child process (PID=20953) handles the SMTP dialog and finally exits.
As you can see, it is very simple. Just to add a couple of important points: do not forget to use "-f" flag when troubleshooting the server processes because most of them spawn the child processes to handle the connections. You can also specify multiple "-p" options with different PIDs to monitor many processes at the same time.
I hope this tutorial was useful. The process tracing mechanisms available in the UNIX systems give most of the people confidence that they actually control their systems and there is nothing "mysterical" running on their computers.
refer:http://myhowto.org/solving-problems/7-exploring-system-internals-with-lsof-and-strace/
-----------------------------
For tracing the system calls of a program, we have a very good tool in strace. What is unique about strace is that, when it is run in conjunction with a program, it outputs all the calls made to the kernel by the program. In many cases, a program may fail because it is unable to open a file or because of insufficient memory. And tracing the output of the program will clearly show the cause of either problem.
The use of strace is quite simple and takes the following form:
$ strace
For example, I can run a trace on 'ls' as follows :
$ strace ls
And this will output a great amount of data on to the screen. If it is hard to keep track of the scrolling mass of data, then there is an option to write the output of strace to a file instead which is done using the -o option. For example,
$ strace -o strace_ls_output.txt ls
... will write all the tracing output of 'ls' to the 'strace_ls_output.txt' file. Now all it requires is to open the file in a text editor and analyze the output to get the necessary clues.
It is common to find a lot of system function calls in the strace output. The most common of them being open(),write(),read(),close() and so on. But the function calls are not limited to these four as you will find many others too.
For example, if you look in the strace output of ls, you will find the following line:
open("/lib/libselinux.so.1", O_RDONLY) = 3
This means that some aspect of ls requires the library module libselinux.so.1 to be present in the /lib folder. And if the library is missing or in a different path, then that aspect of ls which depends on this library will fail to function. The line of code signifies that the opening of the library libselinux.so.1 is successful.
Here I will share my experience in using strace to solve a particular problem I faced. I had installed all the multimedia codecs including the libdvdcss which allowed me to play encrypted DVDs in Ubuntu Linux which I use on a daily basis. But after installing all the necessary codecs, when I tried playing a DVD movie, totem gave me an error saying that it was unable to play the movie (see the picture below). But since I knew that I had already installed libdvdcss on my machine, I was at a loss what to do.
Fig: Totem showing error saying that it cannot find libdvdcss
Then I ran strace on totem as follows :
$ strace -o strace.totem totem
... and then opened the file strace.totem in a text editor and searched for the string libdvdcss . And not surprisingly I came across this line of output as shown in the listing below.
# Output of strace on totem
open("/etc/ld.so.cache", O_RDONLY) = 26
fstat64(26, {st_mode=S_IFREG|0644, st_size=58317, ...}) = 0
old_mmap(NULL, 58317, PROT_READ, MAP_PRIVATE, 26, 0) = 0xb645e000
close(26)
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
...
open("/lib/tls/i686/cmov/libdvdcss.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/i686/cmov", {st_mode=S_IFDIR|0755, st_size=1560, ...}) = 0
...
stat64("/lib/i486-linux-gnu", 0xbfab4770) = -1 ENOENT (No such file or directory)
munmap(0xb645e000, 58317) = 0
open("/usr/lib/xine/plugins/1.1.1/xineplug_inp_mms.so", O_RDONLY) = 26
read(26, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\320\27"..., 512) = 512
fstat64(26, {st_mode=S_IFREG|0644, st_size=40412, ...}) = 0
In the above listing which I have truncated for clarity, the line in bold clearly shows that totem is trying to find the library in, among other places, the '/lib/tls/i686/cmov/' directory and the return value of -1 shows that it has failed to find it. So I realized that for totem to correctly play the encrypted DVD, it has to find the libdvdcss.so.2 file in the path it is searching.
Then I used the find command to locate the library and then copy it to the directory /lib/tls/i686/cmov/. Once I accomplished this, I tried playing the DVD again in totem and it started playing without a hitch.
Fig: Totem playing an encrypted DVD Movie
Just to make sure, I took another trace of totem and it showed that the error was rectified as shown by the bold line of output below.
# Output of the second strace on totem
open("/etc/ld.so.cache", O_RDONLY) = 26
fstat64(26, {st_mode=S_IFREG|0644, st_size=58317, ...}) = 0
old_mmap(NULL, 58317, PROT_READ, MAP_PRIVATE, 26, 0) = 0xb644d000
close(26) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
...
open("/lib/tls/i686/cmov/libdvdcss.so.2", O_RDONLY) = 26
...
stat64("/lib/tls/i686/sse2", 0xbffa4020) = -1 ENOENT (No such file or directory)
munmap(0xb645e000, 58317) = 0
open("/usr/lib/xine/plugins/1.1.1/xineplug_inp_mms.so", O_RDONLY) = 26
read(26, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\360\20"..., 512) = 512
fstat64(26, {st_mode=S_IFREG|0644, st_size=28736, ...}) = 0
Opening the man page of strace, one will find scores of options. For example, if you use the option -t, then strace will prefix each line of the trace with the time of day. One can even specify the system call functions to trace using the -e option. For example, to trace only open() and close() function system calls, one can use the command as follows:
$ strace -o strace.totem -e trace=open,close totem
The ubiquitous strace should not be confused with DTrace that ships with Sun Solaris. strace is just a single tool which takes care of a small part which is tracing a single program. Where as Sun's DTrace toolkit is much more powerful and consists of a collection of scripts which can track, tune and aid the user in troubleshooting ones system in real time. More over, dtrace is a scripting language
with close semblance to C/C++ and awk. Put another way, strace tool in GNU/Linux provides only one of the many functions provided by DTrace in Sun Solaris. That being said, strace plays an important part in aiding the user to troubleshoot ones programs by providing a view of the system calls that the program makes to the Linux kernel.
refer:
http://linuxhelp.blogspot.com/2006/05/strace-very-powerful-troubleshooting.html
This section covers briefly the terminology that you need to familiarize yourself with in order to understand the results of the operations described later in the article. Do not worry, it is not more difficult when answering to Windows Vista UAC question ;)
System Call (a.k.a. "syscall")
System Call is a mechanism for the application to make a call to the operating system. The OS usually has a limited number of the "entry points" that can be used for the basic and fundamental tasks like file I/O, memory allocation, networking etc. For example, most of the UNIX systems have only 1-3 different system calls to open a file (although various libraries offer dozens of ways to deal with a file, they all finally make the same system call). The important thing about the system call that you need to understand is that there are few of them and they are the only way for any program to interact with the OS.
Process ID (PID)
Any process in the UNIX OSes is defined by its unique numeric PID, even if the process runs several threads. From the moment you start a program till the moment it is finished it maintains the same PID. However, some processes can span other processes (usually by using one of the fork() syscalls).
File Handle
Any I/O operations in UNIX is associated with a file handle which is a positive integer number. It applies to any file on the disk, a network socket etc. A typical operation with a file on the disk, for example, is one of the following:
1. open("filename") -> returns a handle that from this moment will be associated with this open file
2. read(handle, where, how_much)
3. write(handle, from_where, how_much)
4. close(handle) -> removes the association established with open()
Handles are unique only for an individual process. There are well-known handles available by default for any application, such as 0 for the standard input, 1 for standard output and 2 for the standard error stream.
Turn out your pockets, process!
We will start from a tool that can show us a snapshot of what is happening right now with our system. There is an excellent tool that is available for many UNIX systems called "lsof" ("LiSt Open Files"). You can find more information about this tool here. Most of the Linux distributions and Mac OS X have this tool included.
Here is what this tool can do:
* Show the list of files and other resources open by a selected process or by all processes in the system
* Show the list of processes using a particular resource (like a TCP socket, for example)
You can do various types of sorting, filtering and formatting for getting exactly the results you want. The program has about 3 dozens of command line options, we will try a couple of typical and most representative examples now. Note: most of these examples require root access to the system (or you will be limited to seeing only the information that is related to your processes).
How to see the list of files opened by a running process
For example, we would like to see which resources are currently used by Apache HTTP server. First, we need to identify the process we are interested in:
[root@localhost]$ ps ax | grep httpd | grep -v grep
2383 ? Ss 0:00 /usr/sbin/httpd
2408 ? S 0:00 /usr/sbin/httpd
2409 ? S 0:00 /usr/sbin/httpd
2410 ? S 0:00 /usr/sbin/httpd
2411 ? S 0:00 /usr/sbin/httpd
2412 ? S 0:00 /usr/sbin/httpd
2414 ? S 0:00 /usr/sbin/httpd
2415 ? S 0:00 /usr/sbin/httpd
2416 ? S 0:00 /usr/sbin/httpd
Hmm...which one I am interested in? Unfortunately, there is no universal recipe to answer this question. However...lets look at the problem from another perspective. At some point I have mentioned that when you start a program, the OS creates a process. And a process can spawn other processes. So, we can assume that there may be a tree of sub-processed related to one parent. Lets see how we can get this information (this command will work on both Linux and OS X):
[root@localhost]$ ps axo pid,ppid,command | egrep '(httpd)|(PID)' |
grep -v grep
2383 1 /usr/sbin/httpd
2408 2383 /usr/sbin/httpd
2409 2383 /usr/sbin/httpd
2410 2383 /usr/sbin/httpd
2411 2383 /usr/sbin/httpd
2412 2383 /usr/sbin/httpd
2414 2383 /usr/sbin/httpd
2415 2383 /usr/sbin/httpd
2416 2383 /usr/sbin/httpd
PPID is simply Parent PID, i.e. Parent Process ID. As you can see, most of these processes have the same PPID (2383) and there is one process with the PID of 2383. This must be the parent process. If you are curious to find out who is its parent, I will let you to do this exercise yourself.
Linux has another fancy tool called "pstree". We could use it to get the same kind of information:
[root@localhost]$ pstree -p | grep httpd
|-httpd(2383)-+-httpd(2408)
| |-httpd(2409)
| |-httpd(2410)
| |-httpd(2411)
| |-httpd(2412)
| |-httpd(2414)
| |-httpd(2415)
| `-httpd(2416)
Looks better and quite self-explaining, isn't it? OK, back to the original question. Now we know that there are certain relationships with these "httpd"s. So, our target will be the process 2383.
In its simples form, "lsof" can be called like this:
[root@localhost]$ lsof -p 2383
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
httpd 2383 root cwd DIR 3,6 1024 2 /
httpd 2383 root rtd DIR 3,6 1024 2 /
httpd 2383 root txt REG 3,8 328136 261136 /usr/sbin/httpd
httpd 2383 root mem REG 3,6 130448 8167 /lib64/ld-2.5.so
httpd 2383 root DEL REG 0,8 8072 /dev/zero
httpd 2383 root mem REG 3,6 615136 79579 /lib64/libm-2.5.so
httpd 2383 root mem REG 3,6 117656 8186 /lib64/libpcre.so.0.0.1
httpd 2383 root mem REG 3,6 95480 8213 /lib64/libselinux.so.1
...
httpd 2383 root mem REG 3,8 46160 1704830 /usr/lib64/
php/modules/ldap.so
httpd 2383 root DEL REG 0,8 8120 /dev/zero
httpd 2383 root DEL REG 0,8 8122 /dev/zero
httpd 2383 root 0r CHR 1,3 1488 /dev/null
httpd 2383 root 1w CHR 1,3 1488 /dev/null
httpd 2383 root 2w REG 3,12 1077 4210 /var/log/
httpd/error_log
httpd 2383 root 3r CHR 1,9 1922 /dev/urandom
httpd 2383 root 4u IPv6 8052 TCP *:http (LISTEN)
httpd 2383 root 5u sock 0,5 8053 can't identify protocol
httpd 2383 root 6u IPv6 8057 TCP *:https (LISTEN)
httpd 2383 root 7u sock 0,5 8058 can't identify protocol
httpd 2383 root 8r FIFO 0,6 8069 pipe
httpd 2383 root 9w FIFO 0,6 8069 pipe
httpd 2383 root 10w REG 3,12 1077 4210 /var/log/
httpd/error_log
httpd 2383 root 11w REG 3,12 711 4215 /var/log/
httpd/ssl_error_log
httpd 2383 root 12w REG 3,12 0 4138 /var/log/
httpd/access_log
httpd 2383 root 13w REG 3,12 0 4151 /var/log/
httpd/ssl_access_log
httpd 2383 root 14w REG 3,12 0 4152 /var/log/
httpd/ssl_request_log
Looks quite interesting, doesn't it? Now I am going to briefly explain the meaning of each kind of entry (you can get more information from the manual page for "lsof").
COMMAND
Contains first 9 characters of the UNIX command name associated with the process
PID
The process ID. In our example it is simply the PID we have asked the information for
USER
The UNIX user owning the process
FD
File Descriptor number. Actually, it is more than the descriptor I have mentioned before. There are special descriptors identifying various resources in the UNIX system. For more details I recommend to check the man page. As you can see, everything is there: the current working directory (cwd), the process image itself (txt), memory-mapped libraries etc. The most interesting for us are the open files and sockets: you can see the numbers 0..14 followed by a suffix ("w" for writing only, "r" for reading only, "u" for both reading and writing and others). For example, you can see that Apache httpd redirects all its error output (2w) to /var/log/https/error_log. It also has a bunch of other regular files open for writing (handles 10-14, various log files). You can also see that it is listening on two TCP sockets (the port numbers shown are resolved through /etc/services file, you can always use "-n" option to disable any name resolution - useful if you, for example, know the port number and want to look for it specifically). You can see two more sockets (do not be afraid of the message "can't identify...", it is just application-specific protocol), Apache is using them to talk to its sub-processes.
TYPE
Type of the node. There is several dozens of various node types, like "REG" for the regular files, "IPv4" (or "IPv6 for IP version 6) for the Internet sockets, "CHR" for the character devices, "DIR" for the directories etc.
DEVICE
Shows the UNIX device numbers (major, minor) associated with the resource. Another interesting parameter. For example, on my system the numbers 3,12 correspond to:
[root@localhost]$ ls -l /dev/* | egrep '3,\s*12'
brw-r----- 1 root disk 3, 12 Jul 17 19:17 /dev/hda12
[root@localhost]$ mount | grep hda12
/dev/hda12 on /var type ext3 (rw)
This is true, the files are locates on one of the partitions on the first IDE disk mounted at /var.
SIZE
The size of the file or the current offset (in bytes). This number does not make sense for some of the resource types so you see blank values there.
NODE
Node number of the file, protocol identifier (for Internet sockets) etc.
Name
The name of the file or resource, if applicable. The value is quite self-explaining and probably one of the most useful.
Try using this command for other processes in your system. Probably one of the most interesting ones to look at will be your X11 server.
There is a number of filters you can use to see only the information you are interested in. For example, I want to know does my iTunes talk to someone over the network or not. Here is a command I would use something like:
[root@localhost]$ lsof -p 1559 -a -i
Which produces empty answer (exactly what I was hoping for :) ). But if I try my SSH client instead of iTunes:
[root@localhost]$ lsof -p 1586 -a -i
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
ssh 1586 nik 3u IPv4 0x40e7394 0t0 TCP macbook:50648
->oberon:ssh (ESTABLISHED)
Summary: you can get a snapshot of any live process using this command. Sometimes it is simpler to use "lsof" (especially on a system you are not familiar with) to find the location of the log file, config files for a service etc.
Who is using that file?
From time to time you may become very curious about who (or at least which process :) ) is using certain resource(s) on your system. There may be a number of reasons: a server that does not start complaining about the port that is already being used, a suspicious network connection that you see when using "netstat" command, inability to unmount a network disk or a external storage unit etc.
A traditional "fuser" command is not always good enough for these purposes. For example, if I want to see all the processes that have any open files under /etc directory. Here is the command that give me the answer:
[root@localhost]$ lsof +D /etc
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
dbus-daem 2206 dbus 5r DIR 3,6 1024 59500 /etc/dbus-1/system.d
acpid 2305 root 6r REG 3,6 236 63639 /etc/acpi/
events/video.conf
avahi-dae 2541 avahi cwd DIR 3,6 1024 63468 /etc/avahi
avahi-dae 2541 avahi rtd DIR 3,6 1024 63468 /etc/avahi
prefdm 2662 root 255r REG 3,6 1465 59322 /etc/X11/prefdm
000-delay 3831 root 255r REG 3,6 577 59446 /etc/cron.
daily/000-delay.cron
When you use "+D path" option, lsof checks every single file under the specified path and shows the information about the processes using it. This command may take some time to execute if you have a lot of files there.
Another situation: a network connection. "-i" option of lsof allows you to search for a specific network connection. The format is following: lsof -i [4|6][protocol][@hostname|hostaddr][:service|port]
"4" or "6" defines the IP protocol version. If you do not use any filters and just use "-i" you will get the list of all connections in your system. Lets find all the UDP connections that we have on a OS X box:
[root@localhost]$ lsof -i UDP
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
launchd 1 root 10u IPv4 0x393ed40 0t0 UDP *:netbios-ns
mDNSRespo 39 root 7u IPv4 0x393e790 0t0 UDP *:mdns
mDNSRespo 39 root 8u IPv6 0x393e6c0 0t0 UDP *:mdns
mDNSRespo 39 root 9u IPv4 0x41556c0 0t0 UDP *:mdns
mDNSRespo 39 root 11u IPv4 0x4155ee0 0t0 UDP *:mdns
mDNSRespo 39 root 12u IPv4 0x393dd00 0t0 UDP macbook:49727
netinfod 40 root 6u IPv4 0x393ee10 0t0 UDP localhost:
netinfo-local
syslogd 41 root 16u IPv4 0x393ec70 0t0 UDP *:49175
Directory 52 root 10u IPv4 0x3fbd1a0 0t0 UDP *:*
Directory 52 root 12u IPv4 0x4155d40 0t0 UDP *:*
ntpd 141 root 5u IPv4 0x393e520 0t0 UDP *:ntp
ntpd 141 root 6u IPv6 0x393e110 0t0 UDP *:ntp
ntpd 141 root 7u IPv6 0x393ead0 0t0 UDP localhost:ntp
ntpd 141 root 8u IPv6 0x393e860 0t0 UDP [fe80:1::1]:ntp
ntpd 141 root 9u IPv4 0x393e5f0 0t0 UDP localhost:ntp
ntpd 141 root 10u IPv6 0x393e450 0t0 UDP [fe80:5::216:
cbff:febf:a08d]:ntp
ntpd 141 root 11u IPv4 0x393e380 0t0 UDP macbook:ntp
prl_dhcpd 147 root 5u IPv4 0x393db60 0t0 UDP *:*
automount 168 root 8u IPv4 0x3fbd000 0t0 UDP localhost:1023
automount 179 root 8u IPv4 0x393e930 0t0 UDP localhost:1022
nmbd 1078 root 0u IPv4 0x393ed40 0t0 UDP *:netbios-ns
nmbd 1078 root 1u IPv4 0x393ed40 0t0 UDP *:netbios-ns
nmbd 1078 root 8u IPv4 0x393ea00 0t0 UDP *:netbios-dgm
nmbd 1078 root 9u IPv4 0x3fbd4e0 0t0 UDP 10.37.129.2:
netbios-ns
nmbd 1078 root 10u IPv4 0x3fbeba0 0t0 UDP 10.37.129.2:
netbios-dgm
nmbd 1078 root 11u IPv4 0x4155380 0t0 UDP macbook:
netbios-ns
nmbd 1078 root 12u IPv4 0x4155450 0t0 UDP macbook:
netbios-dgm
cupsd 1087 root 6u IPv4 0x3fbd340 0t0 UDP *:ipp
Do not be surprised if you do not see anything on your machine - you will not see the information about the processes that do not belong to you unless you are root.
Summary: using lsof with various filters allows you to search through your system and find out the detailed information about any local or network resource used.
Watch your steps
We have learned how to get a snapshot or your UNIX system (or any individual process living in it). But sometimes it is not enough to look at the static information. Dynamic runtime information may answer a lot of other questions - why does the program crash, why does it say "cannot open resource" without providing the resource location, is it accepting the network connection or not etc.
In the fisrt section it was mentioned that the way the UNIX program talks to the operating system (i.e. deals with almost any resource on your computer) is called "system calls". Most of the UNIX kernels, including the Linux and OS X provide a way for the user to see the list of the system calls a process makes in real time, including the parameters the process passes to the kernel. There are different tools on different systems that provide this functionality. On Linux it is called "strace", on BSD (including OS X) there are two toolss working together - "ktrace" and "kdump". We will briefly cover both of them starting from the Linux one.
The really great thing about these tools is that they use the kernel tracing facility in order to work. Thus, they work with ANY program that is running in your system, it does not have to be something you have developed and you do not need the source code of the program in order to understand (in general, of course) what it is doing.
It is important to mention again that just like in case of "lsof" you cannot see anything that you are not supposed to see, i.e. you cannot see what process is doing if it is not yours (or unless you are root user).
The simplest way to run the "strace" would be as: strace command Lets see what a simple well-known like "w" (the command that shows the users logged in and their current activities) does:
[root@localhost]$ strace /usr/bin/w 2>&1 | less
execve("/usr/bin/w", ["/usr/bin/w"], [/* 30 vars */]) = 0
brk(0) = 0x603000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0)
= 0x2aaaaaaab000
uname({sys="Linux", node="oberon.anglab.com", ...}) = 0
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such
file or directory)
open("/etc/ld.so.cache", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=103874, ...}) = 0
mmap(NULL, 103874, PROT_READ, MAP_PRIVATE, 3, 0) = 0x2aaaaaaac000
close(3) = 0
open("/lib64/libproc-3.2.7.so", O_RDONLY) = 3
read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\
240;`\314"..., 832) = 832
...
open("/proc/uptime", O_RDONLY) = 3
lseek(3, 0, SEEK_SET) = 0
read(3, "7267.86 7230.07\n", 1023) = 16
access("/var/run/utmpx", F_OK) = -1 ENOENT (No such file
or directory)
open("/var/run/utmp", O_RDWR) = -1 EACCES (Permission denied)
open("/var/run/utmp", O_RDONLY) = 4
fcntl(4, F_GETFD) = 0
...
open("/proc", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = 3
fstat(3, {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
fcntl(3, F_SETFD, FD_CLOEXEC) = 0
getdents(3, /* 35 entries */, 1024) = 1000
getdents(3, /* 38 entries */, 1024) = 1016
stat("/proc/1", {st_mode=S_IFDIR|0555, st_size=0, ...}) = 0
open("/proc/1/stat", O_RDONLY) = 4
read(4, "1 (init) S 0 1 1 0 -1 4194560 64"..., 1023) = 216
close(4) = 0
socket(PF_FILE, SOCK_STREAM, 0) = 4
...
fcntl(5, F_SETLKW, {type=F_UNLCK, whence=SEEK_SET, start=0, len=0}) = 0
alarm(0) = 1
rt_sigaction(SIGALRM, {SIG_DFL}, NULL, 8) = 0
close(5) = 0
write(1, " 21:31:48 up 2:01, 2 users, l"..., 281 21:31:48 up 2:01,
2 users, load average: 0.00, 0.00, 0.00
USER TTY FROM LOGIN@ IDLE JCPU PCPU WHAT
nick pts/0 192.168.8.18 21:12 9.00s 0.03s 0.03s -bash
nick pts/1 192.168.8.18 21:19 0.00s 0.13s 0.04s strace
/usr/bin
) = 281
exit_group(0) = ?
Process 3947 detached
Only a small part of the long output is shown here. As you can see, everything starts with the "execve" syscall. Which is correct - this is the call that executes a program in UNIX. By the way, for each of these calls you can get more information from the manual pages - just type "man execve" to learn more about the call and its parameters (as you can see the parameters are shown in the trace).And, as it is supposed to be, the execution ends with "exit_group" call with the status 0 (success).
You can see how the program being started loads the shared libraries it depends on. You can see that at some point it tries to open "var/run/utmpx" file but it does not exist (ENOENT). Then it tries to open "/var/run/utmp" file but it is not readable for this program (i.e. for this user) so it gets back (EACCES). Finally this program decides that the only way to find out who is logged in is to go through all the active processes and check where their standard input and output are attached to. In order to do it, "w" readsthe list of files and directories under "/proc" (open - getdents - close). By the way, the number returned by the open() call is the file handle. By using it, you can follow the operations with this file since all read() or write() system calls will use the handle. After getting the list of files, "w" goes over all process directories and reads the virtual files from there to get more information about the processes. And fnally it displays the results by writing them to the file descriptor 1 (standard output, stdout). Very simple and informative.
There is a number of parameters that you can use with "strace" to filter the results and get exactly the information you are looking for. For example, if you are interested only in the network operations, lets see what does "ping" command do (run it as root user):
[root@localhost]$ strace -x -e trace=network,desc ping -c 1 127.0.0.1
...
socket(PF_INET, SOCK_RAW, IPPROTO_ICMP) = 3
socket(PF_INET, SOCK_DGRAM, IPPROTO_IP) = 4
connect(4, {sa_family=AF_INET, sin_port=htons(1025), sin_addr=
inet_addr("127.0.0.1")}, 16) = 0
getsockname(4, {sa_family=AF_INET, sin_port=htons(1028), sin_addr=
inet_addr("127.0.0.1")}, [16]) = 0
close(4) = 0
setsockopt(3, SOL_RAW, ICMP_FILTER, ~(ICMP_ECHOREPLY|ICMP_DEST_UNREACH|
ICMP_SOURCE_QUENCH|ICMP_REDIRECT|ICMP_TIME_EXCEEDED|
ICMP_PARAMETERPROB),
4) = 0
setsockopt(3, SOL_IP, IP_RECVERR, [1], 4) = 0
setsockopt(3, SOL_SOCKET, SO_SNDBUF, [324], 4) = 0
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [65536], 4) = 0
getsockopt(3, SOL_SOCKET, SO_RCVBUF, [27472947287556096], [4]) = 0
fstat(1, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
setsockopt(3, SOL_SOCKET, SO_TIMESTAMP, [1], 4) = 0
setsockopt(3, SOL_SOCKET, SO_SNDTIMEO, "\x01\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00"..., 16) = 0
setsockopt(3, SOL_SOCKET, SO_RCVTIMEO, "\x01\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00"..., 16) = 0
ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, 0x7fff418a7c90) = -1
EINVAL (Invalid argument)
sendmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(0),
sin_addr=inet_addr("127.0.0.1")}, msg_iov(1)=[{"\x08\x00
\xc7\x8b\xd6\x10\x00\x01\x86\x16\xa0\x46\x00\x00"..., 64}],
msg_controllen=0, msg_flags=0}, 0) = 64
recvmsg(3, {msg_name(16)={sa_family=AF_INET, sin_port=htons(0),
sin_addr=inet_addr("127.0.0.1")}, msg_iov(1)=[{"\x45
\x00\x00\x54\xed\xd5\x00\x00\x40\x01\x8e\xd1\x7f\x00"..., 192}],
msg_controllen=32, {cmsg_len=32, cmsg_level=SOL_SOCKET,
cmsg_type=0x1d /* SCM_??? */, ...}, msg_flags=0}, 0) = 84
write(1, "PING 127.0.0.1 (127.0.0.1) 56(84"..., 106PING
127.0.0.1 (127.0.0.1) 56(84) bytes of data.
64 bytes from 127.0.0.1: icmp_seq=1 ttl=64 time=0.321 ms
) = 106
write(1, "\n", 1
) = 1
write(1, "--- 127.0.0.1 ping statistics --"..., 144---
127.0.0.1 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.321/0.321/0.321/0.000 ms
) = 144
Process 4310 detached
"-e" option allows to specify what kind(s) of system calls you are interested in.
As you can see, this program opes 2 network sockets (with the descriptors 3 and 4). One of them is used only to do getsockname() call (do "man getsockname" to get more information on this call) and then it gets closed. The one associated with the descriptor 3 is used to send and receive raw IP traffic (this is the way to send the packets using the protocols different from UDP and TCP). It configures the socket using a number of options and then finally sends an IP packet (sendmsg) to the given destination. You can even see part of the message data ("-x" option requests strace to display the hex numbers instead of non-ASCII strings). If you want to see the complete data you may want to use "-s" option to set the maximum string length you want to be displayed. After sending the message it gets back the response (recvmsg). Then it displays the output of the program.
You can use "-o" option with the file name argument to redirect the output to a text file. Usually when analyzing more complicated examples it is easier to get the trace into a file first and then study this file using text editor or "grep" command.
Mac OS X (like all BSD systems) offers "ktrace" command. In order to get similar result when analyzing the "ping" command we would have to do something like this:
[root@localhost]$ ktrace -f ping.trace -t ci ping -c 1 127.0.0.1
[root@localhost]$ kdump -f ping.trace | less
...
684 ping CALL socket(0x2,0x3,0x1)
684 ping RET socket 3
684 ping CALL getuid
684 ping RET getuid 0
684 ping CALL setuid(0)
684 ping RET setuid 0
684 ping CALL getuid
684 ping RET getuid 0
684 ping CALL getpid
684 ping RET getpid 684/0x2ac
684 ping CALL setsockopt(0x3,0xffff,0x400,0xbffffdc8,0x4)
684 ping RET setsockopt 0
684 ping CALL setsockopt(0x3,0xffff,0x1002,0xbffffdb8,0x4)
684 ping RET setsockopt 0
684 ping CALL setsockopt(0x3,0xffff,0x1001,0xbffffdb8,0x4)
684 ping RET setsockopt 0
684 ping CALL fstat(0x1,0xbffef0c0)
684 ping RET fstat 0
684 ping CALL ioctl(0x1,FIODTYPE,0xbffef08c)
684 ping RET ioctl 0
...
684 ping CALL sendto(0x3,0x5294,0x40,0,0x5260,0x10)
684 ping GIO fd 3 wrote 64 bytes
"\b\08\^R\M-,\^B\0\0\M-T\^X\240F\M-'\M^H\r\0\b
\v\f\r\^N\^O\^P\^Q\^R\^S\^T\^U\^V\^W\^X\^Y\^Z
\^[\^\\^]\^^\^_ !"#$%&'()*+,-./01234567"
684 ping RET sendto 64/0x40
684 ping CALL select(0x4,0xbffffc3c,0,0,0xbffffd90)
684 ping RET select 1
684 ping CALL recvmsg(0x3,0xbffffd24,0)
684 ping GIO fd 3 wrote 84 bytes
"E\0@\0|\M^N\0\0@\^A\0\0\^?\0\0\^A\^?\0\0
\^A\0\0@\^R\M-,\^B\0\0\M-T\^X\240F\M-'\M^H\r\0\b
\v\f\r\^N\^O\^P\^Q\^R\^S\^T\^U\^V\^W\^X\^Y\^Z
\^[\^\\^]\^^\^_ !"#$%&'()*+,-./01234567"
684 ping RET recvmsg 84/0x54
684 ping CALL write(0x1,0x18000,0x39)
684 ping GIO fd 1 wrote 57 bytes
"64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.069 ms
"
684 ping RET write 57/0x39
684 ping CALL sigaction(0x2,0xbffef8c0,0xbffef8f8)
684 ping RET sigaction 0
684 ping CALL sigaction(0xe,0xbffef8c0,0xbffef8f8)
684 ping RET sigaction 0
684 ping CALL write(0x1,0x18000,0x1)
684 ping GIO fd 1 wrote 1 byte
"
"
684 ping RET write 1
684 ping CALL write(0x1,0x18000,0x22)
684 ping GIO fd 1 wrote 34 bytes
"--- 127.0.0.1 ping statistics ---
...
"ktrace" writes the tracing information into a file ("ktrace.out" if the "-f" option is not specified) and the "kdump" tool is used to decode the file into more human-readable format. As you can see the behavior of the ping program on OS X is similar to what we saw on Linux. ktrace/kdump also have some basic filtering capabilities.
Important thing to know about strace is that by default it does not follow the child processes. I.e. if you try to trace a program than spawns other processes, you will not see the operations performed by the child processes. You will see only the fork() calls. In order to "follow" the execution of all child processes you need to use "-f" option.
Tapping the wire
Another nice feature offered by strace (and ktrace on BSD systems) is the ability to attach to any process in the system for a limited time and capture its activities. Sometimes it may help to troubleshoot the network server processes a lot. In order to "attach" to a process you need to use "-p process_ID" option offered by strace. When you are done capturing the information it is enough just to terminate strace by pressing Ctrl+C. Lets try to see what happens when Sendmail accepts an incoming connection (in the given example the number 233 is the process ID of the sendmail process).
[root@localhost]$ strace -f -p 233
And on the second console do:
[root@localhost]$ telnet localhost 25
Trying 127.0.0.1...
Connected to oberon.
Escape character is '^]'.
220 oberon ESMTP Sendmail 8.11.6p2/8.11.6-20030304; Sat, 21 Jul 2007
06:00:02 +0400
quit
221 2.0.0 oberon closing connection
Connection closed by foreign host.
And the press Ctrl+C in the first console. Here is what you will see there (use "-o" option when running strace if you want to write the output to a file):
233 select(5, [4], NULL, NULL, {5, 0}) = 1 (in [4], left {1, 750000})
233 accept(4, {sin_family=AF_INET, sin_port=htons(1226), sin_addr=
inet_addr("127.0.0.1")}, [16]) = 5
233 time([1184983202]) = 1184983202
233 sigprocmask(SIG_BLOCK, [ALRM], []) = 0
233 pipe([6, 7]) = 0
233 sigprocmask(SIG_BLOCK, [CHLD], [ALRM]) = 0
233 fork() = 20953
20953 sigprocmask(SIG_UNBLOCK, [ALRM],
233 getpid(
20953 <... sigprocmask resumed> [ALRM CHLD]) = 0
233 <... getpid resumed> ) = 233
20953 sigprocmask(SIG_UNBLOCK, [CHLD],
233 sigprocmask(SIG_UNBLOCK, [CHLD],
20953 <... sigprocmask resumed> [CHLD]) = 0
233 <... sigprocmask resumed> [ALRM CHLD]) = 0
20953 sigaction(SIGCHLD, {SIG_DFL},
233 close(6
...
20953 open("/etc/hosts", O_RDONLY) = 6
20953 fstat(6, {st_mode=S_IFREG|0644, st_size=963, ...}) = 0
20953 mmap(0, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|0x20, 4294967295, 0)
= 0x11a000
20953 read(6, "#\n# hosts\t\tThis file describe"..., 4096) = 963
20953 close(6) = 0
...
20953 write(3, "<22>Jul 21 06:00:03 sendmail[209"..., 143) = -1 EPIPE
(Broken pipe)
20953 --- SIGPIPE (Broken pipe) ---
20953 close(3) = 0
20953 close(-1) = -1 EBADF (Bad file number)
20953 sigaction(SIGPIPE, {SIG_IGN}, NULL) = 0
20953 sigprocmask(SIG_UNBLOCK, [ALRM], []) = 0
20953 sigprocmask(SIG_BLOCK, NULL, []) = 0
20953 time([1184983203]) = 1184983203
20953 unlink("./dfl6L202l20953") = -1 ENOENT (No such
file or directory)
20953 unlink("./qfl6L202l20953") = -1 ENOENT (No such
file or directory)
20953 unlink("./xfl6L202l20953") = -1 ENOENT (No such
file or directory)
20953 getpid() = 20953
20953 close(7) = 0
20953 setuid(0) = 0
20953 _exit(71) = ?
233 <... select resumed> ) = ? ERESTARTNOHAND (To be restarted)
233 --- SIGCHLD (Child exited) ---
233 wait4(-1, [WIFEXITED(s) && WEXITSTATUS(s) == 71], WNOHANG, NULL)
= 20953
233 wait4(-1, 0xbfffdf60, WNOHANG, NULL) = 0
233 sigprocmask(SIG_BLOCK, [ALRM], []) = 0
233 time([1184983203]) = 1184983203
233 open("/proc/loadavg", O_RDONLY) = 5
233 fstat(5, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
233 mmap(0, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|0x20, 4294967295, 0)
= 0x118000
233 read(5, "0.16 0.09 0.29 3/53 21316\n", 1024) = 26
233 close(5) = 0
233 munmap(0x118000, 4096) = 0
233 sigprocmask(SIG_UNBLOCK, [ALRM], [ALRM]) = 0
233 getpid() = 233
233 select(5, [4], NULL, NULL, {5, 0}
As you can see, sendmail is initially sitting there listening for the incoming connections on the file descriptor 4 (you can use "lsof" to see that it is associated with a TCP socket) - see the second parameter of the select() call. Then the incoming connection comes in and sendmail spawns another process to handle this connection (fork() is not followed by an execve() call because the child process is also sendmail). Then the child process (PID=20953) handles the SMTP dialog and finally exits.
As you can see, it is very simple. Just to add a couple of important points: do not forget to use "-f" flag when troubleshooting the server processes because most of them spawn the child processes to handle the connections. You can also specify multiple "-p" options with different PIDs to monitor many processes at the same time.
I hope this tutorial was useful. The process tracing mechanisms available in the UNIX systems give most of the people confidence that they actually control their systems and there is nothing "mysterical" running on their computers.
refer:http://myhowto.org/solving-problems/7-exploring-system-internals-with-lsof-and-strace/
-----------------------------
For tracing the system calls of a program, we have a very good tool in strace. What is unique about strace is that, when it is run in conjunction with a program, it outputs all the calls made to the kernel by the program. In many cases, a program may fail because it is unable to open a file or because of insufficient memory. And tracing the output of the program will clearly show the cause of either problem.
The use of strace is quite simple and takes the following form:
$ strace
For example, I can run a trace on 'ls' as follows :
$ strace ls
And this will output a great amount of data on to the screen. If it is hard to keep track of the scrolling mass of data, then there is an option to write the output of strace to a file instead which is done using the -o option. For example,
$ strace -o strace_ls_output.txt ls
... will write all the tracing output of 'ls' to the 'strace_ls_output.txt' file. Now all it requires is to open the file in a text editor and analyze the output to get the necessary clues.
It is common to find a lot of system function calls in the strace output. The most common of them being open(),write(),read(),close() and so on. But the function calls are not limited to these four as you will find many others too.
For example, if you look in the strace output of ls, you will find the following line:
open("/lib/libselinux.so.1", O_RDONLY) = 3
This means that some aspect of ls requires the library module libselinux.so.1 to be present in the /lib folder. And if the library is missing or in a different path, then that aspect of ls which depends on this library will fail to function. The line of code signifies that the opening of the library libselinux.so.1 is successful.
Here I will share my experience in using strace to solve a particular problem I faced. I had installed all the multimedia codecs including the libdvdcss which allowed me to play encrypted DVDs in Ubuntu Linux which I use on a daily basis. But after installing all the necessary codecs, when I tried playing a DVD movie, totem gave me an error saying that it was unable to play the movie (see the picture below). But since I knew that I had already installed libdvdcss on my machine, I was at a loss what to do.
Fig: Totem showing error saying that it cannot find libdvdcss
Then I ran strace on totem as follows :
$ strace -o strace.totem totem
... and then opened the file strace.totem in a text editor and searched for the string libdvdcss . And not surprisingly I came across this line of output as shown in the listing below.
# Output of strace on totem
open("/etc/ld.so.cache", O_RDONLY) = 26
fstat64(26, {st_mode=S_IFREG|0644, st_size=58317, ...}) = 0
old_mmap(NULL, 58317, PROT_READ, MAP_PRIVATE, 26, 0) = 0xb645e000
close(26)
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
...
open("/lib/tls/i686/cmov/libdvdcss.so.2", O_RDONLY) = -1 ENOENT (No such file or directory)
stat64("/lib/tls/i686/cmov", {st_mode=S_IFDIR|0755, st_size=1560, ...}) = 0
...
stat64("/lib/i486-linux-gnu", 0xbfab4770) = -1 ENOENT (No such file or directory)
munmap(0xb645e000, 58317) = 0
open("/usr/lib/xine/plugins/1.1.1/xineplug_inp_mms.so", O_RDONLY) = 26
read(26, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\320\27"..., 512) = 512
fstat64(26, {st_mode=S_IFREG|0644, st_size=40412, ...}) = 0
In the above listing which I have truncated for clarity, the line in bold clearly shows that totem is trying to find the library in, among other places, the '/lib/tls/i686/cmov/' directory and the return value of -1 shows that it has failed to find it. So I realized that for totem to correctly play the encrypted DVD, it has to find the libdvdcss.so.2 file in the path it is searching.
Then I used the find command to locate the library and then copy it to the directory /lib/tls/i686/cmov/. Once I accomplished this, I tried playing the DVD again in totem and it started playing without a hitch.
Fig: Totem playing an encrypted DVD Movie
Just to make sure, I took another trace of totem and it showed that the error was rectified as shown by the bold line of output below.
# Output of the second strace on totem
open("/etc/ld.so.cache", O_RDONLY) = 26
fstat64(26, {st_mode=S_IFREG|0644, st_size=58317, ...}) = 0
old_mmap(NULL, 58317, PROT_READ, MAP_PRIVATE, 26, 0) = 0xb644d000
close(26) = 0
access("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)
...
open("/lib/tls/i686/cmov/libdvdcss.so.2", O_RDONLY) = 26
...
stat64("/lib/tls/i686/sse2", 0xbffa4020) = -1 ENOENT (No such file or directory)
munmap(0xb645e000, 58317) = 0
open("/usr/lib/xine/plugins/1.1.1/xineplug_inp_mms.so", O_RDONLY) = 26
read(26, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\360\20"..., 512) = 512
fstat64(26, {st_mode=S_IFREG|0644, st_size=28736, ...}) = 0
Opening the man page of strace, one will find scores of options. For example, if you use the option -t, then strace will prefix each line of the trace with the time of day. One can even specify the system call functions to trace using the -e option. For example, to trace only open() and close() function system calls, one can use the command as follows:
$ strace -o strace.totem -e trace=open,close totem
The ubiquitous strace should not be confused with DTrace that ships with Sun Solaris. strace is just a single tool which takes care of a small part which is tracing a single program. Where as Sun's DTrace toolkit is much more powerful and consists of a collection of scripts which can track, tune and aid the user in troubleshooting ones system in real time. More over, dtrace is a scripting language
with close semblance to C/C++ and awk. Put another way, strace tool in GNU/Linux provides only one of the many functions provided by DTrace in Sun Solaris. That being said, strace plays an important part in aiding the user to troubleshoot ones programs by providing a view of the system calls that the program makes to the Linux kernel.
refer:
http://linuxhelp.blogspot.com/2006/05/strace-very-powerful-troubleshooting.html
Subscribe to:
Posts (Atom)