前言

1
2
3
4
5
6
7
8
9
10
11
问: gdb是什么?
答: 强大的UNIX下命令行调试工具。

问: gdb能干什么?
答: 让你随心所欲的驾驭你的程序;Start、Stop、Examine、Change。

问: 我们为什么要学习gdb?
答: 欲善其事,必先利其器;利用gdb进一步的定位程序异常。

问: 本次分享的宗旨?
答: gdb的介绍和使用入门,为大家抓虫多提供一个选择。

抓虫从0开始

前期准备

1.包含有调试信息的可执行文件

2.编译时加-g选项即可,不建议开优化选项

GDB的启动

1
2
3
4
gdb <program>
gdb <program> core
gdb <program> <PID>
(gdb) file <program>

抓虫流程

实战1 : GDB基础命令的使用

1.1、示例程序(example_1.cpp)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
#include <iostream>
#include <string>

using namespace std;
int foo(int m, int n)
{

return 1;
}
int foo(int n)
{
int result=0;
for (int i=0;i<=n;i++)
{
result+=n;
}
return result;

}

int main()
{

string s1="dafdf";
char * s2;
// s1=s2;

int sum =0;
for (int i=1;i<=100;i++)
{
sum+=i;
}

cout<<"result[1-100]="<<sum<<endl;
cout<<"result[1-250]="<<foo(250)<<endl;

return 0;
}

1.2、调试准备
编译命令:g++ -g -Wall -o example_1 example_1.cpp

1.3、启动gdb

1
2
3
4
5
6
7
8
$ gdb example_1
GNU gdb Red Hat Linux (6.3.0.0-1.96rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1".

1.4、辅助性命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
(gdb) cd ..
Working directory /home/work/testers/sgc.
(gdb) shell ls
autotest client Makefile spanti spantispam_if.h study
(gdb) cd study/
Working directory /home/work/testers/sgc/study.
(gdb) pwd
Working directory /home/work/testers/sgc/study.
(gdb) help run
Start debugged program. You may specify arguments to give it.
Args may include "*", or "[...]"; they are expanded using "sh".
Input and output redirection with ">", "<", or ">>" are also allowed.

With no arguments, uses arguments last specified (with "run" or "set args").
To cancel previous arguments and run with no arguments,
use "set args" without arguments.

1.5、设置断点命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
//查看源代码信息
(gdb) l
17 result+=n;
18 }
19 return result;
20
21 }
22
23 int main()
24 {
25
26 string s1="dafdf";
(gdb) l 1
1 #include <iostream>
2 #include <string>
3
4
5 using namespace std;
6 int foo(int m, int n)
7 {
8
9 return 1;
10
(gdb) l example_1.cpp:16
11 }
12 int foo(int n)
13 {
14 int result=0;
15 for (int i=0;i<=n;i++)
16 {
17 result+=n;
18 }
19 return result;
20
(gdb) l foo(int)
8
9 return 1;
10
11 }
12 int foo(int n)
13 {
14 int result=0;
15 for (int i=0;i<=n;i++)
16 {
17 result+=n;
1
2
3
4
5
6
7
8
9
//设置断点
(gdb) b 17
Breakpoint 1 at 0x400c07: file example_1.cpp, line 17.
(gdb) b main
Breakpoint 2 at 0x400c27: file example_1.cpp, line 26.
(gdb) info br
Num Type Disp Enb Address What
1 breakpoint keep y 0x0000000000400c07 in foo(int) at example_1.cpp:17
2 breakpoint keep y 0x0000000000400c27 in main at example_1.cpp:26

1.6、执行控制命令

1
2
3
4
5
6
7
8
9
10
11
12
13
(gdb) r
Starting program: /home/work/testers/sgc/study/example_1

Breakpoint 2, main () at example_1.cpp:26
26 string s1="dafdf";
(gdb) c
Continuing.
result[1-100]=5050

Breakpoint 1, foo (n=250) at example_1.cpp:17
17 result+=n;
(gdb) n
15 for (int i=0;i<=n;i++)

1.7、程序信息查看命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
//查看变量信息
(gdb) p result
$1 = 250
(gdb) p s1
$2 = 1431655765
(gdb) disp result
1: result = 250
(gdb) c
Continuing.

Breakpoint 1, foo (n=250) at example_1.cpp:17
17 result+=n;
1: result = 250
(gdb) info locals
i = 1
result = 250

//查看栈信息
(gdb) bt
#0 foo (n=250) at example_1.cpp:17
#1 0x0000000000400cc1 in main () at example_1.cpp:38
(gdb) info f
Stack level 0, frame at 0x7fbffff8a0:
rip = 0x400c07 in foo(int) (example_1.cpp:17); saved rip 0x400cc1
called by frame at 0x7fbffff910
source language c++.
Arglist at 0x7fbffff890, args: n=250
Locals at 0x7fbffff890, Previous frame's sp is 0x7fbffff8a0
Saved registers:
rbp at 0x7fbffff890, rip at 0x7fbffff898
(gdb) f 0
#0 foo (n=250) at example_1.cpp:17
17 result+=n;

1.8、修改环境命令

1
2
3
4
5
6
7
8
9
10
11
12
13
14
(gdb) set var i=97
(gdb) p i
$5 = 97
(gdb) print i=98
$6 = 98
(gdb) ignore 1 300
Will ignore next 300 crossings of breakpoint 1.
(gdb) finish
Run till exit from #0 foo (n=250) at example_1.cpp:17
0x0000000000400cc1 in main () at example_1.cpp:38
38 cout<<"result[1-250]="<<foo(250)<<endl;
Value returned is $8 = 38500
(gdb) quit
The program is running. Exit anyway? (y or n) y

不要放过core文件

问 :Core文件是什么?
答 :a disk file containing an image of the process’s memory at the time of termination


问 :Core的作用?
答 :
    1、让你在调试时,不用花费大量等待程序出错;
    2、让你避免了单步调试的烦恼
    3、让你定位错误所在

启动方式

1
$ gdb <program> core

查看程序信息的常用命令

1、查看栈信息:bt, f n, up/down,info frame
2、查看变量信息:info args|locals

实战2 : core文件调试

2.1、示例程序(crash2.c)

1
2
3
4
5
6
7
8
9
10
11
12
#include <string.h>

void Strcpy(char *to , char *from)
{
strcpy(to , from);
}
int main()
{
char *s = NULL;
Strcpy(s, "abcdefg");
return 0;
}

2.2、查看信息

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
$ gdb ./crash2 core.19562
GNU gdb Red Hat Linux (6.3.0.0-1.96rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu"...Using host libthread_db library "/lib64/tls/libthread_db.so.1".

Core was generated by `./crash2'.
Program terminated with signal 11, Segmentation fault.
Reading symbols from /lib64/tls/libc.so.6...done.
Loaded symbols for /lib64/tls/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
#0 0x000000302af6f9a4 in strcpy () from /lib64/tls/libc.so.6
(gdb) bt
#0 0x000000302af6f9a4 in strcpy () from /lib64/tls/libc.so.6
#1 0x00000000004004c5 in Strcpy (to=0x0, from=0x4005dc "abcdefg") at crash2.c:5
#2 0x00000000004004e5 in main () at crash2.c:10
(gdb) f 0
#0 0x000000302af6f9a4 in strcpy () from /lib64/tls/libc.so.6
(gdb) up
#1 0x00000000004004c5 in Strcpy (to=0x0, from=0x4005dc "abcdefg") at crash2.c:5
5 strcpy(to , from);
(gdb) info args
to = 0x0
from = 0x4005dc "abcdefg"

//至此,已经清楚的发现了问题所在,to指针为空

进阶之多线程程序调试

多线程调试常用命令

1
2
3
4
5
6
$ info <...>  // 强大的查看命令,如info threads
$ attach/detach <pid> // 挂载到进程
$ thread <thread_no> // 切换到线程
$ thread apply <thread_no_list> <cmd>
// 对于list中的thread,执行cmd
$ break <linenum> thread <threadno> if ...

实战3 : 多线程程序调试

3.1、正常的示例程序(good_thread.c)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
#include <pthread.h>
#include <stdio.h>
#include <sys/time.h>
#include <string.h>
#define MAX 10
pthread_t thread[2];
pthread_mutex_t mut;
int number=0, i;
void *thread1()
{
        printf ("thread1 : I'm thread 1\n");
        for (i = 0; i < MAX; i++)
        {
                printf("thread1 : number = %d\n",number);
                pthread_mutex_lock(&mut);
                        number++;
                pthread_mutex_unlock(&mut);
                sleep(2);
        }
        printf("thread1 :主函数在等我完成任务吗?\n");
        pthread_exit(NULL);
}
void *thread2()
{
        printf("thread2 : I'm thread 2\n");
        for (i = 0; i < MAX; i++)
        {
                printf("thread2 : number = %d\n",number);
                pthread_mutex_lock(&mut);
                        number++;
                 pthread_mutex_unlock(&mut);
                sleep(3);
        }
        printf("thread2 :主函数在等我完成任务吗?\n");
        pthread_exit(NULL);
}
void thread_create(void)
{
        int temp;
        memset(&thread, 0, sizeof(thread));          //comment1
       /*创建线程*/
        if((temp = pthread_create(&thread[0], NULL, thread1, NULL)) != 0)       //comment2
                printf("线程1创建失败!\n");
        else
                printf("线程1被创建\n");
        if((temp = pthread_create(&thread[1], NULL, thread2, NULL)) != 0) //comment3
                printf("线程2创建失败");
        else
                printf("线程2被创建\n");
}
void thread_wait(void)
{
        /*等待线程结束*/
        if(thread[0] !=0) {                   //comment4
                pthread_join(thread[0],NULL);
                printf("线程1已经结束\n");
        }
        if(thread[1] !=0) {                //comment5
                pthread_join(thread[1],NULL);
                printf("线程2已经结束\n");
        }
}
int main()
{
        /*用默认属性初始化互斥锁*/
        pthread_mutex_init(&mut,NULL);
        printf("我是主函数哦,我正在创建线程,呵呵\n");
        thread_create();
        printf("我是主函数哦,我正在等待线程完成任务阿,呵呵\n");
        thread_wait();
        return 0;
}

3.2、演示过程

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
(gdb)
21 printf ("thread1 : I'm thread 1\n");
22
23 for (i = 0; i < MAX; i++)
24
25 {
26
27 printf("thread1 : number = %d\n",number);
28
29 pthread_mutex_lock(&mut);
(gdb) b 27
Breakpoint 1 at 0x40079e: file good_thread.c, line 27.
(gdb)
51 for (i = 0; i < MAX; i++)
52
53 {
54
55 printf("thread2 : number = %d\n",number);
56
57 pthread_mutex_lock(&mut);
58
59 number++;
(gdb) b 57
Breakpoint 2 at 0x400838: file good_thread.c, line 57.
(gdb) r
Starting program: /home/work/testers/sgc/study/goodthread
[Thread debugging using libthread_db enabled]
[New Thread 182894112416 (LWP 22783)]
ÎÒÊÇÖ÷º¯ÊýŶ£¬ÎÒÕýÔÚ´´½¨Ị̈߳¬ºÇºÇ
[New Thread 1084229984 (LWP 22786)]
Ïß³Ì1±»´´½¨
thread1 : I'm thread 1
[Switching to Thread 1084229984 (LWP 22786)]

Breakpoint 1, thread1 () at good_thread.c:27
27 printf("thread1 : number = %d\n",number);
(gdb) bt
#0 thread1 () at good_thread.c:27
#1 0x000000302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#2 0x000000302afc6003 in clone () from /lib64/tls/libc.so.6
#3 0x0000000000000000 in ?? ()
(gdb) info threads
[New Thread 1094719840 (LWP 22787)]
3 Thread 1094719840 (LWP 22787) 0x000000302afc5fc4 in clone () from /lib64/tls/libc.so.6
* 2 Thread 1084229984 (LWP 22786) thread1 () at good_thread.c:27
1 Thread 182894112416 (LWP 22783) 0x000000302afc5fc4 in clone () from /lib64/tls/libc.so.6
(gdb) thread 1
[Switching to thread 1 (Thread 182894112416 (LWP 22783))]#0 0x000000302afc5fc4 in clone () from /lib64/tls/libc.so.6
(gdb) bt
#0 0x000000302afc5fc4 in clone () from /lib64/tls/libc.so.6
#1 0x000000302b805d86 in do_clone () from /lib64/tls/libpthread.so.0
#2 0x000000302b806846 in pthread_create@@GLIBC_2.2.5 () from /lib64/tls/libpthread.so.0
#3 0x00000000004008fd in thread_create () at good_thread.c:91
#4 0x00000000004009a9 in main () at good_thread.c:135

3.3、死锁示例程序(multi_thread.c)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <pthread.h>

#define THREAD_NUM 20

pthread_mutex_t AccountA_mutex;
pthread_mutex_t AccountB_mutex;
struct Account {
char account_name[1];
int balance;
};
struct Account accountA = {'A', 100000};
struct Account accountB = {'B', 200000};
void * accountAB (void* amount_ptr) {
int amount = *((int*)amount_ptr);
pthread_mutex_lock(&AccountA_mutex);
if (accountA.balance < amount) {
printf("There is not enough memory in Account A!\n");
pthread_mutex_unlock(&AccountA_mutex);
pthread_exit((void *)1);
}
accountA.balance -=amount;
sleep(2);
pthread_mutex_lock(&AccountB_mutex);
accountB.balance +=amount;
pthread_mutex_unlock(&AccountA_mutex);
pthread_mutex_unlock(&AccountB_mutex);
}

void * accountBA (void* amount_ptr) {
int amount = *((int*)amount_ptr);
pthread_mutex_lock(&AccountB_mutex);
if (accountB.balance < amount) {
printf("There is not enough memory in Account B!\n");
pthread_mutex_unlock(&AccountB_mutex);
pthread_exit((void *)1);
}
accountB.balance -=amount;
pthread_mutex_lock(&AccountA_mutex);
accountA.balance +=amount;
pthread_mutex_unlock(&AccountB_mutex);
pthread_mutex_unlock(&AccountA_mutex);
}

int main(int argc, char* argv[]) {
int threadid[THREAD_NUM];
pthread_t pthread[THREAD_NUM];
void* thResState;
int res, flag;
int transfer_amount[THREAD_NUM] = {100, 200, 300, 400,100,200,300,400,500,600,700,800,900,800,700,600,500,400,300,200};

pthread_attr_t attr;
pthread_attr_init(&attr);
pthread_attr_setscope(&attr, PTHREAD_SCOPE_SYSTEM);

for(flag=0; flag<THREAD_NUM; flag++){
if(flag%2 == 0){
if (threadid[flag] = pthread_create(&pthread[flag], &attr , accountAB, \
(void*)&transfer_amount[flag]) < 0){
printf("Thread %d creation failed\n", flag);
exit (1);
}
}
else{
if (threadid[flag] = pthread_create(&pthread[flag], &attr , accountBA, \
(void*)&transfer_amount[flag]) < 0) {
printf("Thread %d creation failed\n", flag);
exit (1);
}
}
}

for(flag=0; flag<THREAD_NUM; flag++){
res = pthread_join(pthread[flag], &thResState);
if (res != 0){
perror("Thread join failed");
exit(-1);
}
printf("thread success id %u state code %d\n",threadid[flag],thResState);
}

printf("Transitions are in progress..");
printf("\nAll the money is transferred !!\n");
}

3.4、错误定位

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
$ gcc -g -o multi_thread multi_thread.c –lpthread
$ ./multi_thread
$ ps -eLF |grep multi
work 21675 19997 21675 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21676 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21677 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21678 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21679 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21680 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21681 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21682 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21683 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21684 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21685 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21686 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21687 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21688 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21689 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21690 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21691 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21692 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21693 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21694 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread
work 21675 19997 21695 0 21 52124 464 0 00:25 pts/8 00:00:00 ./multi_thread


$ gdb
GNU gdb Red Hat Linux (6.3.0.0-1.96rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
(gdb) attach 21680
Attaching to process 21680
Reading symbols from /home/work/testers/sgc/study/multi_thread...done.
Using host libthread_db library "/lib64/tls/libthread_db.so.1".
Reading symbols from /lib64/tls/libpthread.so.0...done.
[Thread debugging using libthread_db enabled]
[New Thread 182894112416 (LWP 21675)]
[New Thread 1283537248 (LWP 21695)]
[New Thread 1273047392 (LWP 21694)]
[New Thread 1262557536 (LWP 21693)]
[New Thread 1252067680 (LWP 21692)]
[New Thread 1241577824 (LWP 21691)]
[New Thread 1231087968 (LWP 21690)]
[New Thread 1220598112 (LWP 21689)]
[New Thread 1210108256 (LWP 21688)]
[New Thread 1199618400 (LWP 21687)]
[New Thread 1189128544 (LWP 21686)]
[New Thread 1178638688 (LWP 21685)]
[New Thread 1168148832 (LWP 21684)]
[New Thread 1157658976 (LWP 21683)]
[New Thread 1147169120 (LWP 21682)]
[New Thread 1136679264 (LWP 21681)]
[New Thread 1126189408 (LWP 21680)]
[New Thread 1115699552 (LWP 21679)]
[New Thread 1105209696 (LWP 21678)]
[New Thread 1094719840 (LWP 21677)]
[New Thread 1084229984 (LWP 21676)]
Loaded symbols for /lib64/tls/libpthread.so.0
Reading symbols from /lib64/tls/libc.so.6...done.
Loaded symbols for /lib64/tls/libc.so.6
Reading symbols from /lib64/ld-linux-x86-64.so.2...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0

(gdb) bt
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x00000000432041e0 in ?? ()
#2 0x00000000432049f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()
(gdb) info threads
21 Thread 1084229984 (LWP 21676) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
20 Thread 1094719840 (LWP 21677) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
19 Thread 1105209696 (LWP 21678) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
18 Thread 1115699552 (LWP 21679) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
17 Thread 1126189408 (LWP 21680) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
16 Thread 1136679264 (LWP 21681) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
15 Thread 1147169120 (LWP 21682) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
14 Thread 1157658976 (LWP 21683) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
13 Thread 1168148832 (LWP 21684) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
12 Thread 1178638688 (LWP 21685) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
11 Thread 1189128544 (LWP 21686) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
10 Thread 1199618400 (LWP 21687) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
9 Thread 1210108256 (LWP 21688) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
8 Thread 1220598112 (LWP 21689) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
7 Thread 1231087968 (LWP 21690) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
6 Thread 1241577824 (LWP 21691) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
5 Thread 1252067680 (LWP 21692) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
4 Thread 1262557536 (LWP 21693) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
3 Thread 1273047392 (LWP 21694) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
2 Thread 1283537248 (LWP 21695) 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
1 Thread 182894112416 (LWP 21675) 0x000000302b806ffb in pthread_join () from /lib64/tls/libpthread.so.0

(gdb) thread apply all bt

Thread 21 (Thread 1084229984 (LWP 21676)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x0000000000000000 in ?? ()

Thread 20 (Thread 1094719840 (LWP 21677)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x00000000414011e0 in ?? ()
#2 0x00000000414019f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()

Thread 19 (Thread 1105209696 (LWP 21678)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x0000000041e021e0 in ?? ()
#2 0x0000000041e029f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()

Thread 18 (Thread 1115699552 (LWP 21679)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x00000000428031e0 in ?? ()
#2 0x00000000428039f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()

Thread 17 (Thread 1126189408 (LWP 21680)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x00000000432041e0 in ?? ()
#2 0x00000000432049f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()

Thread 16 (Thread 1136679264 (LWP 21681)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x0000000043c051e0 in ?? ()
#2 0x0000000043c059f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()

Thread 15 (Thread 1147169120 (LWP 21682)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x00000000446061e0 in ?? ()
#2 0x00000000446069f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()
---Type <return> to continue, or q <return> to quit---

Thread 14 (Thread 1157658976 (LWP 21683)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x00000000450071e0 in ?? ()
#2 0x00000000450079f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()

Thread 13 (Thread 1168148832 (LWP 21684)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x0000000045a081e0 in ?? ()
#2 0x0000000045a089f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()

Thread 12 (Thread 1178638688 (LWP 21685)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x00000000464091e0 in ?? ()
#2 0x00000000464099f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()

Thread 11 (Thread 1189128544 (LWP 21686)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x0000000046e0a1e0 in ?? ()
#2 0x0000000046e0a9f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()
Thread 10 (Thread 1199618400 (LWP 21687)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x000000004780b1e0 in ?? ()
#2 0x000000004780b9f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()

Thread 9 (Thread 1210108256 (LWP 21688)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x000000004820c1e0 in ?? ()
#2 0x000000004820c9f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()

Thread 8 (Thread 1220598112 (LWP 21689)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x0000000048c0d1e0 in ?? ()
---Type <return> to continue, or q <return> to quit---
#2 0x0000000048c0d9f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()

Thread 7 (Thread 1231087968 (LWP 21690)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x000000004960e1e0 in ?? ()
#2 0x000000004960e9f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()

Thread 6 (Thread 1241577824 (LWP 21691)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x000000004a00f1e0 in ?? ()
#2 0x000000004a00f9f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()

Thread 5 (Thread 1252067680 (LWP 21692)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x000000004aa101e0 in ?? ()
#2 0x000000004aa109f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()
Thread 4 (Thread 1262557536 (LWP 21693)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x000000004b4111e0 in ?? ()
#2 0x000000004b4119f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()

Thread 3 (Thread 1273047392 (LWP 21694)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x000000004be121e0 in ?? ()
#2 0x000000004be129f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()

Thread 2 (Thread 1283537248 (LWP 21695)):
#0 0x000000302b80adfb in __lll_mutex_lock_wait () from /lib64/tls/libpthread.so.0
#1 0x000000004c8131e0 in ?? ()
#2 0x000000004c8139f0 in ?? ()
#3 0x000000302b807bd4 in pthread_mutex_lock () from /lib64/tls/libpthread.so.0
#4 0x0000000000000000 in ?? ()

---Type <return> to continue, or q <return> to quit---
Thread 1 (Thread 182894112416 (LWP 21675)):
#0 0x000000302b806ffb in pthread_join () from /lib64/tls/libpthread.so.0
#1 0x0000000000400b69 in main (argc=1, argv=0x7fbffffa08) at multi_thread.c:76

总结

GDB是一个好用的抓虫工具,随意控制进程,随便查看内存和环境;上述只是新手学习过程中的一些记录,欢迎深入探讨。