Linux Command - tcpdump/ethereal/wireshark

show the packets with specified length
# tcpdump -nvi eth0 port ssh and greater 1500

//tcpdump filter which tcp header src port isn't 3389 and destination port isn't 3389 too.
tcpdump -n -N -i eth1 tcp[0:2] != 3389 and tcp[2:2] != 3389

tcpdump -n -N -i eth1 tcp[0:2] = 445 or tcp[2:2] = 445

tcpdump -n -N -i eth1 'tcp[0:2]!=161 and tcp[2:2]!=161 and tcp[0:2]!=445 and tcp[2:2]!=445 and tcp[0:2]!=139 and tcp[2:2]!=139'



//下面兩個例子都是要求tcpdump去capture src/dest port > 1024 並且不等於3389port的connection
tcpdump -n -N -i eth1 'tcp[0:2] & 0xfc00!=0 and tcp[2:2] & 0xfc00!=0 and tcp[0:2]!=3389 and tcp[2:2]!=3389'

tcpdump -n -N -i eth1 'tcp[0:2]>1024 and tcp[2:2]>1024 and tcp[0:2]!=3389 and tcp[2:2]!=3389'


將tcpdump 的內容寫到檔案，以供tcpreplay 或 ethereal 來寫
tcpdump -n -i br0 ip host 170.116.11.94 and port 80 -w file

看某一個特定subnet
tcpdump -n -i br0 ip net 192.168.200.0/24

TCP Flags
URG ACK PSH RST SYN FIN
## tcp[13]==17 --> FIN, ACK
## tcp[13]==2 --> SYN
## tcp[13]==18 --> SYN, ACK
## tcp[13]==4 --> RST
# tcpdump -n -i br0 host 192.168.200.1 and tcp[13]==17 or tcp[13]=2

將tcpdump 的內容寫到檔案，以供tcpreplay 或 ethereal 來寫
tcpdump -n -i br0 ip host 170.116.11.94 and port 80 -w file


關於tcpdump 抓檔時, 所設定的-s snaplength 過小時, tcpreplay 在replay traffic 時, 會有的處理方式
tcpdump -s snaplen

tcpreplay -u
can be
pad -- pad the end of the packet with zeros
trunc -- re-adjusting the length in the IP header

-u or untruncate
When a packet is truncated in the capture file because the snaplen was too small, this option will pad the end of the packet with zeros, or truncate (trunc) it by re-adjusting the length in the IP header. The trunc option will only alter IPv4 packets, all others will be sent unmodified.

tcpreplay -R
tcpreplay -r , ex.
# tcpreplay -r 30
tcpreplay -m 0.1

Some early experience
//使用前須先執行，經過測試好像不用執行這行指令也可以work的樣子，不是很確定
//將eth0 interface設定為promisc mode，亦即網路上所有流經本地端的封包，都會被parsing
#ifconfig eth0 promisc

//將interface eth的promisc mode disable
#ifconfig eth0 -promisc

//不要使用promiscuous mode的情形下來節取packet，尚在測試中，不是很確定
#tcpdump -p

//dump source ip address 為 143.158.11.94的packet
#tcpdump src host 143.158.11.94

//dump source ip address & destination ipaddress為143.158.11.96的packet
#tcpdump host 143.158.11.96

//dump source port & destination port為80的packet
#tcpdump port 80

//dump destination network為143.158.11.0/24 的packet
# tcpdump dst net 143.158.11.0/24

# tcpdump -nvi eth2 ip net 172.16.0.0/16

>> dump a network wioth specified netmask
#./tcpdump -i wan1 -s 0 net 198.145.245.0 mask 255.255.255.0 -w aaa.pcap
>> or like this format
#./tcpdump -i wan1 -s 0 net 198.145.245.0/24 -w aaa.pcap

promiscuous mode 是指 ethernet card 接收全部 packet 的一種模式，正常的情況, ethernet card 是用 mode 3.

ethernet card 的receive mode 有:

01h turn off receiver
02h receive only packes sent to this interface
03h mode 2 plus broadcast packets
04h mode 3 plus limited multicast packets
05h mode 3 plus all multicast packets
06h all packets(promiscuous mode)
07h raw mode for serial line only(v1.10+)

你可能執行類似 tcpdump 的網路監聽程式, 才會有上述的 error mesg

網卡可以置一種模式叫混雜模式（promiscuous），在這種模式下工作的網卡能夠接收到一切通過它的數據，而不管實際上數 據的目的地址是不是他。這實際上就是我們SNIFF工作的基本原理：讓網卡接收一切他所能接收的數據。

Q. How to set network card in the promiscuous mode?
A.
# ifconfig eth0 promisc
// Be sure to replace "eth0" with your own network interface in case it's "wlan0" or something else.
// To remove promiscuous mode, type:
# ifconfig eth0 -promisc

ethereal/wireshark Analyzes => Display Filters can input the filtering rule to search the specified packets
e.g.
* tcp.flags.reset == 1
* tcp.len >= 1500
* frame.number==24333 (each packet is called a frame in the ethereal/wireshark, so we can use frame as the display filter component
* tcp.analysis.flags (use wireshark tcp analysis result)
* tcp.analysis.lost_segment
* tcp.analysis.retransmission
* tcp.analysis.fast_retransmission
* cdp.checksum_bad==1 || edp.checksum_bad==1 || ip.checksum_bad==1 || tcp.checksum_bad==1 || udp.checksum_bad==1


Advanced wireshark search tips
include:
Display filter:
e.g.
frame contains fe:3d:dd:36
or other display filtering rules
Hex value:
fe:3d:dd:36 or fe3ddd36
String:
"babala" (guessing usage...)

ipv6 filtering rules in the wireshark
icmpv6
ipv6.src
ipv6.dst

ipv6.addr == ff02::1:2


We can setup the coloring rule in the wireshark to separate the different packets type in the following method.
Select View -> Coloring Rules
Setup the coloring rules to display different color according the filtering rules.

wireshark will sniff all the packets destinated the Network Card buffer. And print the packets in the raw socket format.

We can disable the TCP checksum verification by the following steps:
1. Select Edit->Preferences
2. Select protocols -> TCP from the left frame of current window
3. Disable the option of "Validate the TCP checksum if possible"
Then the checksum error of TCP packets will be ignored by the wireshark.

TCP principle
* TCP often send an ACK packet while received a data packet, the acknowlegement number is the received packet sequence number plus received packet payload length.

* Sometimes TCP will send an Cumulative ACK if a preACK is not yet sent out because the data of sender is quickly enough (often less than 500 ms)
(According the RFC documents: TCP ACK generation[RFC 1122, RFC 2581])

* TCP will send a duplicated-ACK while it receive a miss-order packet (miss one packet and receive the afterward packet, wireshark call "TCP Previous segment lost").

* After receive a out-of-order packet (missing the previous packets, wireshark label "TCP Previous segment lost"), the wireshark will label all the following received packet "TCP Retransmission" until the packet sequence of previous received out-of-order packet.

* In wireshark, after receiving a packet labeled "TCP Previous segment lost", packet "MPacket". And then immediately (<0.001) TCP Out-Of-Order" and not call it "TCP Retransmission". Because just invert these two packet receiving time, the tcp sequence order will be correct.

Some different tcp analysis in the wireshark
Dup ACK (Due to miss a packet of a next sequence number, this is labeled a "TCP Previous segment lost" in the wireshark)
Retransmission (Due to happen the specified RTT timeout and no ack received)
Fast Retransmission (Due to receive 3 ack packet with the same acknowledgement number) TCP Previous segment lost (Due to miss a packet which should in the next order, but receive a wrong order packet)


Refer the Fast Retransmission description as following:
http://en.wikipedia.org/wiki/Fast_retransmit

Fast Retransmit is an enhancement to TCP which reduces the time a sender waits before retransmitting a lost segment.
A TCP sender uses timers to recognize lost segments. If an acknowledgement is not received for a particular segment within a specified time (a function of the estimated Round-trip delay time), the sender will assume the segment was lost in the network, and will retransmit the segment.
The fast retransmit enhancement works as follows: if a TCP sender receives three duplicate acknowledgements with the same acknowledge number (that is, a total of four acknowledgements with the same acknowledgement number), the sender can be reasonably confident that the segment with the next higher sequence number was dropped, and will not arrive out of order. The sender will then retransmit the packet that was presumed dropped before waiting for its timeout.

Linux Command - vi, use Vi to delete ^M signature

My experience

在Unix/BSD中，要顯示

^M等符號的方法，
按住Ctrl + V + M就會產生" ^M "的符號

實際例子：
當從windows傳送檔案到Unix/BSD之後，
或是Unix互傳檔案時，沒有使用ascii mode，而使用binary mode時

文字檔的每行行末會出現 ^M 的符號，這樣很難看，希望將這個符號刪除。

利用vi的替代功能

將^M改為空白符號
:%s/(Ctrl + V +M)//g
:1,$s/(Ctrl + V +M)//g

不過在剛拿到這些scripts(經由E-Mail得到)，發現其在DOM內用vi觀看時，會發現在每行的行尾都有一個^M符號，而在Linux Source下觀看則沒有，使得每次想run該script時，都必須先進vi，以人工的方式將這個符號刪除，但是由於資料是儲存在ramdisk中，故每次重新開機後，資料就會恢復原狀，這個問題困擾我滿久的。
因為剛開始就懷疑原因可能是UNIX和DOS在做文字檔轉換的時候，所多出來的符號，但是原本是朝向由vi內部去做符號搜尋替換的方向，或是由vi的特殊功能來改善，但是試過很多方法沒有效果。
不過現在發現這個工具”dos2unix”，在嘗試性的run過一遍後，竟然驚奇的發現原來的^M符號不見了，現在就將方法詳列於下
//指令格式dos2unix –n infile outfile
// If the dest file is new created file, you should add "-n" parameter
# dos2unix –n wanfo.html wanfo

// If the dest file is already existed, you should add "-o" parameter or just leave the parameter empty
# dos2unix -o wanfo.html
# dos2unix wanfo.html

原本wanfo.html內有^M符號，經過下面指令執行後，產生出來的wanfo，並沒有^M符號


some experiences of other advisor
http://newbiedoc.sourceforge.net/text_editing/vi.html#SEARCHING

Searching and Replacing

Searching text is done with the command /xxx for a forwards search or ?yyy for a backwards search. n will skip to the next occurrence.When specifying / without argument vi will default to the argument of the last search.

Global Search and Replace:

The magic command is

:line1,line2s/old_string/new_string/g

The /g is optional, it means 'do the replace everytime'.If not specified, vi will replace only the first occurrence in each line.

Special ^XX characters

To search for a ^XX character, you must use Ctrl-v (^v) in order to disable interpretation of the Ctrl commands.

A useful example:

Windows (MS-DOS) text files use RETURN/LINEFEED to end every line; Mac uses only RETURN; and Unix/Linux uses only NEWLINE (which is the same as the linefeed in DOS). To use the linux programming style:

\r\n = chr(13)chr(10) = MS-DOS

\r = chr(13) = Mac

\n = chr(10) = Linux/Unix

So when displaying an msdos ascii file with vi (or with any other text editor), you will find each and every line ended by a ^M (it's character 13, aka \r, aka ENTER). When displaying a mac ascii file, you will have a single line with a ^M at what should be each end of line.

Our MSDOS text file should look like this:

Friday the 13th^M ^M ^M ^M Dear Sir,^M ^M ....

And our mac text file should look like that

"Friday the 13th^M^M^MDearSir,^M^M...."

[The Macintosh->Unix conversion isn't easy to do with vi macros, so we'll concentrate on msdos/windows->Unix]

MS-DOS/Windows -> UNIX conversion:

In order to remove these ugly ^M, you search for them and replace them by....nothing!

So first, let's search for those weird ^M ... but, how can you search for character 'ENTER'?

By preceeding it with ^V (Control-V). Any keystroke after ^V is accepted literally -- that is, it won't have its usual command function, if it's something like ESC, ENTER, ^Z, etc...

What the following command tells VI to do is to replace the first (since the /g option isn't set, but anyway, we only expect one) ^M on every line, with nothing (there is nothing between the last two slashes: //):

This is what you type

:1,$s/^V^M//

(where ^V is Control-V, and ^M is ENTER or Control-M)

note that VI doesn't display the ^V, so you'll only see

:1,$s/^M//

This what you actually see

And it should work...

[FYI the "text-edition task force" is working on an elegant way to convert mac ascii files to unix, but the first research campaign hasn't brought us much]

file manipulation experience

/* Change the permissions of all the sub files and folder
Remove the read/write/excute rights of group and others of all the files and subfolders of the specified folder
Using chmod command to excute
*/

# chmod -R go-r,go-w,go-x ./folder/

# chmod -R g-rwx,o-rwx ./folder2/

Socket experience

Server
socket
bind
listen
accept
close

Client
socket
connect
close

While client issue the connect, the physical packet sequence is like

1. SYN (client -> server)
2. SYN+ACK (server->client)
3. ACK

But I found that server will auto send the FIN wjile the client is trying to connect the server.
So the connection will closed immediately while the client is trying to connect the server.

4. FIN+ACK
5.ACK

Afterall the client will also send the FIN to close the connection.

6. FIN+ACK
7. ACK

The server will respond the error
Socket Error: Transport endpoint is not connected

When a program is terminated using the exit, it will send the reset to the connected socket.

Endians

All processors must be designated as either big endian or little endian.
Intel's 80x86 processors and their clones are little endian.
Sun's SPARC, Motorola's 68K, and the PowerPC families are all big endian. The Java Virtual Machine is big endian as well.

Intel 80x86 is little endian (host byte order). The least significant byte first stored in the memory. So the memory dump is from the least significant byte to the most significant byte.

The network byte order format is big endian. So we need to transfer the byte order if we want to transfer data from Intel 80x86 machine onto the network.

For example.
General speaking, the packet format of adrress/port is like following syntax

Appearing in the physical packet
Address: 8c61 12ab ==> 140.97.18.171
Port: c268 ==> 0xc268 == 49768 (big endian format, network byte order, from the most significant byte to the least significant byte)

sendmail mail pool?

/var/spool/clientmqueue

sendmail mail pool?


The following message is copied from forum

> Hi, all
>   I found there is a directory named /var/spool/clientmqueue and many
> many file below this directory, what's use of this directory and the
> files under it?
>   thanks.

When submitting mail by using sendmail as a mail submission program,
sendmail copies all messages to /var/spool/clientmqueue first. sendmail is
a setgid smmsp program and thus gives the any user the permission to do so
(/var/spool/clientmqueue belongs to user and group smmsp).

Later, another sendmail process, the sendmail localhost-only (FreeBSD
default config) mail transfer agent (MTA) copies the messages from
/var/spool/clientmqueue to /var/spool/mqueue and sends them to their
destination.

When files accumulate in /var/spool/clientmqueue, you probably don't run
the sendmail localhost MTA, and thus the mails don't get send.

Linux Command - cvs, get specified date source from cvs server

we can use the following date format to retrieve the specified source date version
# cvs -q up -D 2008-08-13 00:00
# cvs st -v ./Makefile
===================================================================
File: Makefile Status: Up-to-date

Working revision: 1.18
Repository revision: 1.18 [omit]/Makefile,v
Sticky Tag: (none)
Sticky Date: 2008.08.12.16.00.00
Sticky Options: (none)


# cvs -q up -D "2010-01-18 18:00:00 +0800" //Because the Taipei time minutes was +08:00
or
# cvs -q up -D "2010-01-18 10:00:00 UTC"
# cvs st -v ./File.c
File: BufferManager.c Status: Up-to-date

Working revision: 1.2
Repository revision: 1.2 [omit]/File.c,v
Sticky Tag: (none)
Sticky Date: 2010.01.18.10.00.00
Sticky Options: (none)



# checkout a specified date/time version of project code in the UTC (Coordinated Universal Time). This command was worked for me.
# cvs checkout -D "2009-05-04 14:48:00 UTC" modulename

# The following format is not yet correct executed in my environment yet, but it was suggested by the net
cvs checkout -D "2009-05-04 14:48:00 +0800" modulename >> "+0800" means that the CVS UTC time will minus the "0800" and the real checkout time (UTC/GMT time) will be "2009-05-04 06:48:00"
or cvs update -D "2009-05-04 14:48:00 +0800" modulename

Note:
Greenwich Mean Time (GMT) can be considered equivalent to UTC time

// cvs command format
cvs [Global options] cvs_command [Common options] [cvs_command_individual_options]

//
# cvs -q up -r TSA_V1272_PS -D "2010-01-17 04:00:00 -0800"

File: Tunnel_TCP.c Status: Needs Patch

Working revision: 1.182.2.1.2.3
Repository revision: 1.182.2.1.2.4 /home/cvs/tunnelcode/TunnelSystem/TunnelSystem/Tunnel_TCP.c,v
Sticky Tag: TSA_V1272_PS (branch: 1.182.2.1.2)
Sticky Date: (none)
Sticky Options: (none)