顯示具有 gdb 標籤的文章。 顯示所有文章
顯示具有 gdb 標籤的文章。 顯示所有文章

2008年4月24日 星期四

eXosip2 segementation fault experience

I had ever installed the osip2/eXosip2 library to test my program, but it always cause the Segmentation fault while I am running my sip user agent program.

The osip2/eXosip2 library versions are as follows:
osip2: libosip2-3.0.3-2
eXosip2: libeXosip2-3.0.3-3

The libraries include libeXosip2, libosip2, libosipparser2 as follow listing
# ls -la /usr/lib/libosip* /usr/lib/libeXosip2.* | awk '{print $8}'
/usr/lib/libeXosip2.a
/usr/lib/libeXosip2.la
/usr/lib/libeXosip2.so
/usr/lib/libeXosip2.so.4
/usr/lib/libeXosip2.so.4.2.0
/usr/lib/libosip2.a
/usr/lib/libosip2.la
/usr/lib/libosip2.so
/usr/lib/libosip2.so.2
/usr/lib/libosip2.so.2.2.0
/usr/lib/libosipparser2.a
/usr/lib/libosipparser2.la
/usr/lib/libosipparser2.so
/usr/lib/libosipparser2.so.2
/usr/lib/libosipparser2.so.2.2.0


When I run the sip ua program, it will cause Segmentation fault while receive the SIP reply message "407 Proxy Authentication Required".

SIP/2.0 407 Proxy Authentication Required
Via: SIP/2.0/UDP
[Other sip headers are omitted]

| INFO1 | Message received from: A.B.C.D:5060
| INFO1 | Message received from: A.B.C.D:5060
| INFO3 | MESSAGE REC. CALLID:951246000
| INFO1 | Message received from: A.B.C.D:5060
| INFO3 | cb_rcv4xx (id=1)
| INFO2 | eXosip: timer sec:5 usec:10000!

Segmentation fault (core dumped)

The gdb bt (backtrace) dumping is listed below

Program terminated with signal 11, Segmentation fault.
#0 0x0015298c in osip_trace@plt () from /usr/lib/libeXosip2.so.4
(gdb) bt
#0 0x0015298c in osip_trace@plt () from /usr/lib/libeXosip2.so.4
#1 0x0016e722 in __eXosip_create_proxy_authorization_header (wa=0x9eab608, rquri=0x9eab940 "sip:d-life.org",
username=0x9ea71e2 "admin", passwd=0x9ea7214 "openserrw", ha1=0x9ea7246 "", auth=0xbfd87dac,
method=0x9eaa258 "REGISTER", pCNonce=0x17cd9d "0a4f113b", iNonceCount=1) at jauth.c:484
#2 0x001556c9 in eXosip_add_authentication_information (req=0x9eaa268, last_response=0x9eab3a0) at eXosip.c:1248
#3 0x00158952 in _eXosip_register_build_register (jr=0x9ea72b8, _reg=0xbfd87e94) at eXregister_api.c:145
#4 0x00158e48 in eXosip_register_send_register (rid=1, reg=0x0) at eXregister_api.c:294
#5 0x001531ca in _eXosip_retry_register_with_auth (je=0x9eaa7f8) at eXosip.c:255
#6 0x0015362b in eXosip_retry_with_auth (je=0x9eaa7f8) at eXosip.c:385
#7 0x001538a6 in eXosip_default_action (je=0x9eaa7f8) at eXosip.c:450
#8 0x0804b704 in main (argc=3, argv=0xbfd8a014) at sip8d.c:139

The Segmentation fault is caused by the signal 11 (SIGSEGM), and was triggered from the eXosip2 eXosip_default_action api (frame 7). And the error reference seems to happen in the osip2/osipparser2 library (frame 1).

definition of /usr/include/bits/signum.h (some pieces of the definition)
/* Signals. */
#define SIGHUP 1 /* Hangup (POSIX). */
#define SIGINT 2 /* Interrupt (ANSI). */
#define SIGQUIT 3 /* Quit (POSIX). */
#define SIGILL 4 /* Illegal instruction (ANSI). */
#define SIGTRAP 5 /* Trace trap (POSIX). */
#define SIGABRT 6 /* Abort (ANSI). */
#define SIGIOT 6 /* IOT trap (4.2 BSD). */
#define SIGBUS 7 /* BUS error (4.2 BSD). */
#define SIGFPE 8 /* Floating-point exception (ANSI). */
#define SIGKILL 9 /* Kill, unblockable (POSIX). */
#define SIGUSR1 10 /* User-defined signal 1 (POSIX). */
#define SIGSEGV 11 /* Segmentation violation (ANSI). */
#define SIGUSR2 12 /* User-defined signal 2 (POSIX). */
#define SIGPIPE 13 /* Broken pipe (POSIX). */
#define SIGALRM 14 /* Alarm clock (POSIX). */
#define SIGTERM 15 /* Termination (ANSI). */

From frame 0, we can know that libeXosip2 seems something wrong with the osip_trace

But I can not get the idea (help?) from the frame 0 of the gdb backtrace. beacause when I try to list the original source code of backtrace frame 0. It just show me the
original code of the sip user agent.
(gdb) f 0
#0 0x0015298c in osip_trace@plt () from /usr/lib/libeXosip2.so.4
(gdb) list
4 */
5
6
7 int main (int argc, char *argv[])
8 {
9 sipconf *sc;
10 siprunconf *sr;
11
12 char buf[ML], tmp[ML];
13 int ch, i, call=0;

Another point, we can show the member libraries of the libeXosip2 shared library by using nm.

# nm -l -o /usr/lib/libeXosip2.so.4 | grep osip_trace
/usr/lib/libeXosip2.so.4: U osip_trace

Here we can discover that osip_trace is an undefined symbol (reference to the nm man page, "U" The symbol is undefined)

The final solution was using the newest library of the osip2/eXosip2 libraries as follow listing

Correct libraries combination (No segmentation fault)
osip2: libosip2-3.1.0
eXosip2: libeXosip2-3.1.0

Error libraries combination (Cause the segmentation fault)
osip2: libosip2-3.0.3-2
eXosip2: libeXosip2-3.0.3-3

I guess the possible reason was that osip2 and eXosip2 library have something dependency that library version number must the same to guarantee the proper working. Because the eXosip2 library is based on the osip2 library stack to develop. Maybe the eXosip2 developers doesn't consider the backward compatibility very carefully to cause this segmentation fault condition.

2007年5月6日 星期日

如何在child process 中利用gdb debug

問題描述:
跑一個process, 中間 fork ㄧ個 child process, 之後parent process 繼續執行, 發現child process 好像卡住, 不會繼續動作

Step1.
ulimit -c unlimited
設定 core dump file size 為unlimited, 預設值為0 size

ulimit 程式的主要目的是用來設定程式執行的許多參數, 包括 core file size, data segment size 還有許多其他的參數, 請利用 ulimit -a 來秀所有support 的參數
# ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
pending signals (-i) 1024
max locked memory (kbytes, -l) 32
max memory size (kbytes, -m) unlimited
open files (-n) 1024t
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 3072
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


Step2.
執行主程式, 可以看到parent process fork 出child process 的process id 為 31887 (利用 getpid()得知), 另外在 ps 也可以看到

# ps aux | grep prog
root 31886 0.1 0.4 2932 904 pts/1 S+ 00:42 0:00 ./prog
root 31887 0.0 0.4 2912 872 pts/1 S+ 00:42 0:00 ./prog
root 31888 0.0 0.4 2928 928 pts/1 S+ 00:42 0:00 ./prog


Step3.
kill -11 31887
送出 SIGSEGV signal 給 child process, child process 收到 segmentation fault 的signal, 就會 core dump 並且在同一個目錄下, 產生core.XXX (XXX為process id) 的檔案, 以這裡的例子而言就是
core.31887

/usr/include/asm/signal.h 下的定義
#define SIGSEGV 11


Step4.
利用 gdb 來debug
# gdb prog core.31887

[略]
Core was generated by `./prog'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /usr/lib/libstdc++.so.6...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/tls/libm.so.6...done.
Loaded symbols for /lib/tls/libm.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /lib/tls/libc.so.6...done.
Loaded symbols for /lib/tls/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
#0 0x0092e7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
(gdb) bt << 下bt (backtrace) 去 trace 所有的 function
#0 0x0092e7a2 in _dl_sysinfo_int80 () from /lib/ld-linux.so.2
#1 0x00a0cfc1 in recvfrom () from /lib/tls/libc.so.6
#2 0x0804f0b7 in getMessage (fd=2048, buf=0xfef6d4a0 "\001\001", len=0xfef6cfc4, srcIp=0x800, srcPort=0x800,
verbose=false) at udp.cxx:127
#3 0x0804daf6 in MyTest (myFd=3, dest=@0x800, testNum=1, verbose=false, reply=0xfef6df18) at stun.cxx:2188
#4 0x0804a460 in get_mess (req=0xfef6df50, reqtype=QT_Reflexive, reply=0x0, type=0xfffffe00)
at myclient.cxx:632
#5 0x0804a8c6 in refresh (reqfd=3, svr=0xfef6ea60) at myclient.cxx:572
#6 0x08049299 in main (argc=1, argv=0xfef6eb24) at sip8test.c:71

因為 frame 0, 1 都是system call
frame 2, 3, 4, 5, 6 都是我們程式自己寫的 function
所以可以懷疑程式是卡在 fram 2 getMessage function 當中, 也就是 udp.cxx 的第127 行

Step5.
frame 就是有許多不同層 stack 的意思, 在gdb 中可以利用frame 去jump 到某ㄧ個特定的stack (frame), 來print variable value

(gdb) frame 2 << 我們這裡執行 frame 2 來看stack2 的variable value, 可以用 p XXX 來看
#2 0x0804f0b7 in getMessage (fd=2048, buf=0xfef6d4a0 "\001\001", len=0xfef6cfc4, srcIp=0x800, srcPort=0x800,
verbose=false) at udp.cxx:127
127 *len = recvfrom(fd,


Step6.
這時可以list 來看 code 的內容, 找出錯誤的原因

(gdb) list
122 assert( originalSize > 0 );
123
124 struct sockaddr_in from;
125 int fromLen = sizeof(from);
126
127 *len = recvfrom(fd,
128 buf,
129 originalSize,
130 0,
131 (struct sockaddr *)&from,


Embedded system debug
# toolChain_gdb daemon coreFile
For example
# /opt/v6_be_uclibc/bin/arm_v6_be_uclibc-gdb daemon coreFile