2010年6月10日 星期四

Clound Computing - Hadoop

Hadoop - http://hadoop.apache.org/

Unified Modeling Language (UML)

The de factor standard of Software Enginering - Unified Modeling Language (UML)

http://www.sparxsystems.com.au/resources/uml2_tutorial/



http://www.cnblogs.com/oomusou/archive/2007/02/20/652899.html

(原 創) association,aggregation,composition有什麼差別? (OO) (UML) (C/C++)

class之間有三種關係,inheritance,implementation和association。inheritance和 implementation在C++、C#、Java都有直接支援,所以不難懂,但association,aggregation和 composition在語言並沒有知接支援,到底三者有什麼差別呢?

首先看association,下圖是個典型的 assoication class diagram。



association的表示法是實線加上箭 頭,有兩個重點,navigability和multiplicity,navigability就是箭頭指的方向,指別的class的,表示負責維護 association關係,在此class中有data member存著被指class的reference(pointer),multiplicity則是上面的數字,離自己class比較遠的,表示對方 class和自己class之間的關係,如上圖,1個School可以有多個Student,而1個Student只能有一個School,而一個 Student可以選1到6個Course,1個Course可被1到多個Student選。

association常見的問題是,到底 箭頭該怎麼指才對?一般來說,若是一對多的關係,是由一指向多,因為通堂一個class會有個array或vector儲存多個物件,但這並非絕對,只能 說通常如此,如上圖的Student對Course是一對多,且Course對Student也是一對多,但他選擇了由Course負責 association關係,所以完全看設計需要,實務上,建議如UML for Java Programmer中文版p.3-15那樣,加上stereotype,詳細的敘述是屬於哪一種association,將來我會再專文介紹。

若 以C++表示,association的程式碼如下
1class A {
2 private:
3 B* itsB;
4}
;

再 來看aggregation

aggregation的表示法由空diamond和箭頭 表示(選的這張圖是比較舊的UML格式,所以沒有箭頭),空diamond表示whole,箭頭表示part。若以英文表示,就是has a的關係,上圖是典型一對多的表示法。

若以C++表示,aggregation的程式碼如下
1class Node {
2 private:
3 vector<Node*> itsNodes;
4}
;

最 後是composition,表示法與aggregation的差異在於變成實diamond,其他完全一樣,跟aggregation在意義的差異在於 composition強調『同生共死』,當System物件死亡時,Component物件也要跟著死亡,但aggregation是『生死有命』,當 System物件死亡時,Component物件並不特別去處理。

若以C++表示,composition的程式碼如下
1class Car {
2 public:
3 virtual ~Car() {delete itsCarb;}
4 private:
5 Carburetor* itsCarb
6}
;

由 以上程式可知,composition須由destructor去處理,而aggregation則不必。

在C#、Java這類有 garbage collection的語言,composition幾乎不會用到,但在C++,只要用到pointer,就得自己去delete,所以在 composition對於C++就很重要。


UML keywords
alt ==> if else
opt ==> switch case
loop ==> for/while loop

2010年6月6日 星期日

Linux Service - Firewall, NAT - iptables

Enable the FTP Passive mode to login ftp server which installed iptables firewall rules

[root@localhost net]# /sbin/modprobe nf_conntrack_ftp
[root@localhost net]# /sbin/modprobe nf_nat_ftp

[root@localhost net]# /sbin/lsmod | grep ftp
nf_nat_ftp 6721 0
nf_conntrack_ftp 10725 1 nf_nat_ftp
nf_nat 18393 3 nf_nat_ftp,ipt_MASQUERADE,iptable_nat
nf_conntrack 50453 7 nf_nat_ftp,nf_conntrack_ftp,xt_state,ipt_MASQUERADE,iptable_nat,nf_nat,nf_conntrack_ipv4


# cat /etc/sysconfig/iptables
*nat
:PREROUTING ACCEPT [2071492:180971620]
:POSTROUTING ACCEPT [2118:149186]
:OUTPUT ACCEPT [1956:140466]
-A POSTROUTING -s 172.16.0.0/255.255.0.0 -o eth8 -j MASQUERADE
-A POSTROUTING -s 17.17.0.0/255.255.0.0 -o eth8 -j MASQUERADE
COMMIT

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:RH-Firewall-1-INPUT - [0:0]
-A INPUT -j RH-Firewall-1-INPUT
-A RH-Firewall-1-INPUT -i lo -j ACCEPT
-A RH-Firewall-1-INPUT -p icmp --icmp-type any -j ACCEPT
-A RH-Firewall-1-INPUT -p 50 -j ACCEPT
-A RH-Firewall-1-INPUT -p 51 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp --dport 5353 -d 224.0.0.251 -j ACCEPT
-A RH-Firewall-1-INPUT -p udp -m udp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 631 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 80 -j ACCEPT
-A RH-Firewall-1-INPUT -p tcp -m tcp --dport 8080 -j ACCEPT
-A RH-Firewall-1-INPUT -m multiport -p tcp --destination-ports 111,635,52845 -j ACCEPT
-A RH-Firewall-1-INPUT -m multiport -p udp --destination-ports 111,635,52845 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -m tcp -p tcp --sport 20 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW,ESTABLISHED -m tcp -p tcp --dport 21 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state ESTABLISHED,RELATED -m tcp -p tcp --sport 1024: --dport 1024: -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A RH-Firewall-1-INPUT -m state --state NEW -m tcp -p tcp --dport 904 -j ACCEPT
-A RH-Firewall-1-INPUT -j REJECT --reject-with icmp-host-prohibited
#-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT



# /sbin/iptables -L -v -n
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
1936 141K RH-Firewall-1-INPUT all -- * * 0.0.0.0/0 0.0.0.0/0

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 1473 packets, 143K bytes)
pkts bytes target prot opt in out source destination

Chain RH-Firewall-1-INPUT (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 icmp type 255
0 0 ACCEPT esp -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT ah -- * * 0.0.0.0/0 0.0.0.0/0
0 0 ACCEPT udp -- * * 0.0.0.0/0 224.0.0.251 udp dpt:5353
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 udp dpt:631
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:631
12 830 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:8080
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 111,635,52845
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 111,635,52845
1328 98880 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED tcp spt:20 ==> The rule seems not the root rule of the FTP passive mode problem
4 206 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW,ESTABLISHED tcp dpt:21
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED tcp spts:1024:65535 dpts:1024:65535 ==> The rule seems not the root rule of the FTP passive mode problem
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:22
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:904
592 40943 REJECT all -- * * 0.0.0.0/0 0.0.0.0/0 reject-with icmp-host-prohibited

The difference between FTP active and passive mode

Active FTP vs. Passive FTP, a Definitive Explanation

http://slacksite.com/other/ftp.html

Contents:

* Introduction
* The Basics
* Active FTP
* Active FTP Example
* Passive FTP
* Passive FTP Example
* Other Notes
* Summary
* References
* Appendix 1: Configuration of Common FTP Servers


Introduction

One of the most commonly seen questions when dealing with firewalls and other Internet connectivity issues is the difference between active and passive FTP and how best to support either or both of them. Hopefully the following text will help to clear up some of the confusion over how to support FTP in a firewalled environment.

This may not be the definitive explanation, as the title claims, however, I've heard enough good feedback and seen this document linked in enough places to know that quite a few people have found it to be useful. I am always looking for ways to improve things though, and if you find something that is not quite clear or needs more explanation, please let me know! Recent additions to this document include the examples of both active and passive command line FTP sessions. These session examples should help make things a bit clearer. They also provide a nice picture into what goes on behind the scenes during an FTP session. Now, on to the information...

The Basics

FTP is a TCP based service exclusively. There is no UDP component to FTP. FTP is an unusual service in that it utilizes two ports, a 'data' port and a 'command' port (also known as the control port). Traditionally these are port 21 for the command port and port 20 for the data port. The confusion begins however, when we find that depending on the mode, the data port is not always on port 20.

Active FTP

In active mode FTP the client connects from a random unprivileged port (N > 1023) to the FTP server's command port, port 21. Then, the client starts listening to port N+1 and sends the FTP command PORT N+1 to the FTP server. The server will then connect back to the client's specified data port from its local data port, which is port 20.

From the server-side firewall's standpoint, to support active mode FTP the following communication channels need to be opened:

* FTP server's port 21 from anywhere (Client initiates connection)
* FTP server's port 21 to ports > 1023 (Server responds to client's control port)
* FTP server's port 20 to ports > 1023 (Server initiates data connection to client's data port)
* FTP server's port 20 from ports > 1023 (Client sends ACKs to server's data port)

When drawn out, the connection appears as follows:
In step 1, the client's command port contacts the server's command port and sends the command PORT 1027. The server then sends an ACK back to the client's command port in step 2. In step 3 the server initiates a connection on its local data port to the data port the client specified earlier. Finally, the client sends an ACK back as shown in step 4.

The main problem with active mode FTP actually falls on the client side. The FTP client doesn't make the actual connection to the data port of the server--it simply tells the server what port it is listening on and the server connects back to the specified port on the client. From the client side firewall this appears to be an outside system initiating a connection to an internal client--something that is usually blocked.

Active FTP Example

Below is an actual example of an active FTP session. The only things that have been changed are the server names, IP addresses, and user names. In this example an FTP session is initiated from testbox1.slacksite.com (192.168.150.80), a linux box running the standard FTP command line client, to testbox2.slacksite.com (192.168.150.90), a linux box running ProFTPd 1.2.2RC2. The debugging (-d) flag is used with the FTP client to show what is going on behind the scenes. Everything in red is the debugging output which shows the actual FTP commands being sent to the server and the responses generated from those commands. Normal server output is shown in black, and user input is in bold.

There are a few interesting things to consider about this dialog. Notice that when the PORT command is issued, it specifies a port on the client (192.168.150.80) system, rather than the server. We will see the opposite behavior when we use passive FTP. While we are on the subject, a quick note about the format of the PORT command. As you can see in the example below it is formatted as a series of six numbers separated by commas. The first four octets are the IP address while the last two octets comprise the port that will be used for the data connection. To find the actual port multiply the fifth octet by 256 and then add the sixth octet to the total. Thus in the example below the port number is ( (14*256) + 178), or 3762. A quick check with netstat should confirm this information.

testbox1: {/home/p-t/slacker/public_html} % ftp -d testbox2
Connected to testbox2.slacksite.com.
220 testbox2.slacksite.com FTP server ready.
Name (testbox2:slacker): slacker
---> USER slacker
331 Password required for slacker.
Password: TmpPass
---> PASS XXXX
230 User slacker logged in.
---> SYST
215 UNIX Type: L8
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> ls
ftp: setsockopt (ignored): Permission denied
---> PORT 192,168,150,80,14,178
200 PORT command successful.
---> LIST
150 Opening ASCII mode data connection for file list.
drwx------ 3 slacker users 104 Jul 27 01:45 public_html
226 Transfer complete.
ftp> quit
---> QUIT
221 Goodbye.


Passive FTP

In order to resolve the issue of the server initiating the connection to the client a different method for FTP connections was developed. This was known as passive mode, or PASV, after the command used by the client to tell the server it is in passive mode.

In passive mode FTP the client initiates both connections to the server, solving the problem of firewalls filtering the incoming data port connection to the client from the server. When opening an FTP connection, the client opens two random unprivileged ports locally (N > 1023 and N+1). The first port contacts the server on port 21, but instead of then issuing a PORT command and allowing the server to connect back to its data port, the client will issue the PASV command. The result of this is that the server then opens a random unprivileged port (P > 1023) and sends the PORT P command back to the client. The client then initiates the connection from port N+1 to port P on the server to transfer data.

From the server-side firewall's standpoint, to support passive mode FTP the following communication channels need to be opened:

* FTP server's port 21 from anywhere (Client initiates connection)
* FTP server's port 21 to ports > 1023 (Server responds to client's control port)
* FTP server's ports > 1023 from anywhere (Client initiates data connection to random port specified by server)
* FTP server's ports > 1023 to remote ports > 1023 (Server sends ACKs (and data) to client's data port)

When drawn, a passive mode FTP connection looks like this:
In step 1, the client contacts the server on the command port and issues the PASV command. The server then replies in step 2 with PORT 2024, telling the client which port it is listening to for the data connection. In step 3 the client then initiates the data connection from its data port to the specified server data port. Finally, the server sends back an ACK in step 4 to the client's data port.

While passive mode FTP solves many of the problems from the client side, it opens up a whole range of problems on the server side. The biggest issue is the need to allow any remote connection to high numbered ports on the server. Fortunately, many FTP daemons, including the popular WU-FTPD allow the administrator to specify a range of ports which the FTP server will use. See Appendix 1 for more information.

The second issue involves supporting and troubleshooting clients which do (or do not) support passive mode. As an example, the command line FTP utility provided with Solaris does not support passive mode, necessitating a third-party FTP client, such as ncftp.

With the massive popularity of the World Wide Web, many people prefer to use their web browser as an FTP client. Most browsers only support passive mode when accessing ftp:// URLs. This can either be good or bad depending on what the servers and firewalls are configured to support.

Passive FTP Example

Below is an actual example of a passive FTP session. The only things that have been changed are the server names, IP addresses, and user names. In this example an FTP session is initiated from testbox1.slacksite.com (192.168.150.80), a linux box running the standard FTP command line client, to testbox2.slacksite.com (192.168.150.90), a linux box running ProFTPd 1.2.2RC2. The debugging (-d) flag is used with the FTP client to show what is going on behind the scenes. Everything in red is the debugging output which shows the actual FTP commands being sent to the server and the responses generated from those commands. Normal server output is shown in black, and user input is in bold.

Notice the difference in the PORT command in this example as opposed to the active FTP example. Here, we see a port being opened on the server (192.168.150.90) system, rather than the client. See the discussion about the format of the PORT command above, in the Active FTP Example section.

testbox1: {/home/p-t/slacker/public_html} % ftp -d testbox2
Connected to testbox2.slacksite.com.
220 testbox2.slacksite.com FTP server ready.
Name (testbox2:slacker): slacker
---> USER slacker
331 Password required for slacker.
Password: TmpPass
---> PASS XXXX
230 User slacker logged in.
---> SYST
215 UNIX Type: L8
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> passive
Passive mode on.
ftp> ls
ftp: setsockopt (ignored): Permission denied
---> PASV
227 Entering Passive Mode (192,168,150,90,195,149).
---> LIST
150 Opening ASCII mode data connection for file list
drwx------ 3 slacker users 104 Jul 27 01:45 public_html
226 Transfer complete.
ftp> quit
---> QUIT
221 Goodbye.


Other Notes

A reader, Maarten Sjouw, pointed out that active FTP will not function when used in conjunction with a client-side NAT (Network Address Translation) device which is not smart enough to alter the IP address info in FTP packets.

Summary

The following chart should help admins remember how each FTP mode works:

Active FTP :
command : client >1023 -> server 21
data : client >1023 <- server 20

Passive FTP :
command : client >1023 -> server 21
data : client >1023 -> server >1023

A quick summary of the pros and cons of active vs. passive FTP is also in order:

Active FTP is beneficial to the FTP server admin, but detrimental to the client side admin. The FTP server attempts to make connections to random high ports on the client, which would almost certainly be blocked by a firewall on the client side. Passive FTP is beneficial to the client, but detrimental to the FTP server admin. The client will make both connections to the server, but one of them will be to a random high port, which would almost certainly be blocked by a firewall on the server side.

Luckily, there is somewhat of a compromise. Since admins running FTP servers will need to make their servers accessible to the greatest number of clients, they will almost certainly need to support passive FTP. The exposure of high level ports on the server can be minimized by specifying a limited port range for the FTP server to use. Thus, everything except for this range of ports can be firewalled on the server side. While this doesn't eliminate all risk to the server, it decreases it tremendously. See Appendix 1 for more information.