Sec-Art: Python

Showing posts with label Python. Show all posts

Installing Python Interpreter in Windows 10

Python is a popular high-level programming language used for general purpose programming. But unlike Linux/Unix based operating systems, python interpreter does not come pre-packaged with windows 10. It means we have to install it by ourselves.

Installing python in Windows 10 in very easy, we just need to download the installer file and install it. You can download python interpreter for windows from below link :

Web Scrapping with Python using BeautifulSoup module

Web Scrapping (also known as web harvesting or web data extraction ) is a technique used to extract information and data from websites. In this tutorial we are going to use a python module called beautifulSoup for web scrapping. It is very powerful python library for extract data from HTML and XML files. But note that BeautifulSoup does not send any page requests to website, so we have to do this by using other modules like urllib2, requests etc. Now first we need to install BeautifulSoup Library. To install the BeautifulSoup module use the below command :

 sudo apt-get install python-bs4

or you can also install with pip

 pip install beautifulsoup4

or in-order to install BeautifulSoup from source, Download the source from here

https://www.crummy.com/software/BeautifulSoup/bs4/download/4.6/

Then extract it and run the setup.py file

 tar xvf beautifulsoup4-x.x.tar.gz
 python setup.py install

Installing HTML Parser :

In Beautiful Soup we need to suppy an html parser to process the data. With python an html parser ('html.parser') comes built in, But we can also instal some more powerfull html parsers like xml, html5lib etc. To install below parsers use the below command :

 sudo apt-get install python-lxml
 sudo apt-get install python-html5lib

 pip install lxml
 pip install html5lib

The lxml parser is very fast and can be used to quickly parse given HTML. But the html5lib is a bit slow as compared to lxml, but it is also very useful parser. To clearify the difference between all the parsers, try the below code :

 $ python
 
 >>> from bs4 import BeautifulSoup
 >>> code = """<html>
 <HEAD>
 <title>This is test
 </HEAD>
 <body>
 <p>Hello world This is test</p>
 </html>"""

html.pareser :

 >>> psr1 = BeautifulSoup(code, 'html.parser')
 >>> print psr1
 <html>
 <head>
 <title>This is test
 </title></head>
 <body>
 <p>Hello world This is test</p>
 </body></html>

xml parser :

 >>> psr2 = BeautifulSoup(code, 'xml')
 >>> print psr2
 <?xml version="1.0" encoding="utf-8"?>
 <html>
 <HEAD>
 <title>This is test
 </title>
 <body>
 <p>Hello world This is test</p>
 </body></HEAD></html>

lxml parser :

 >>> psr3 = BeautifulSoup(code, 'lxml')
 >>> print psr3
 <html>
 <head>
 <title>This is test
 </title></head>
 <body>
 <p>Hello world This is test</p>
 </body></html>

html5lib parser :

Basics of beautifulSoup :

Now here are some basic example of how to use BeautifulSoup Library for Web Scrapping, we are taking a simple html code for demonstration :

 <html>
 <head>
  <title>Simple Web Page</title>
 </head>
 <body>
  <p class='heading'>Hello world : A Sample Page test</p>
  <p class='sub-heading'>This the Sub-heading or Description</p>
  <table>
    <tr>
   <th>Firstname</th><th>Lastname</th> <th>Age</th>
  </tr>
  <tr>
   <td>Frank</td><td>Castle</td><td>40</td>
    </tr>
    <tr>
   <td>Jack</td><td>Rietcher</td><td>45</td>
    </tr>
  </table>
  <a href="http://wikipedia.org/">Wikipedia</a>
  <a href="http://youtube.com">Watch Videos</a>
  <a href="http://google.com">Search Something</a>
  <a href="http://facebook.com">Find Your Friends</a>
 <body>
</html>

Store the above code in a variable in python interpreter

 code = """<html>
 <head>
  <title>Simple Web Page</title>
 </head>
 <body>
  <p class='heading'>Hello world : A Sample Page test</p>
  <p class='sub-heading'>This the Sub-heading or Description</p>
  <table>
    <tr>
    <th>Firstname</th><th>Lastname</th> <th>Age</th>
  </tr>
  <tr>
   <td>Frank</td><td>Castle</td><td>40</td>
    </tr>
    <tr>
   <td>Jack</td><td>Rietcher</td><td>45</td>
    </tr>
  </table>
  <a href="http://wikipedia.org/">Wikipedia</a>
  <a href="http://youtube.com">Watch Videos</a>
  <a href="http://google.com">Search Something</a>
  <a href="http://facebook.com">Find Your Friends</a>
 <body>
</html>"""

Now import the BeautifulSoup library :

 from bs4 import BeautifulSoup

Now create a bs4 object to parse the data, we going to use lxml parser in this example :

 soup = BeautifulSoup(code, 'lxml')

Now with the soup objects we can access all elements of the html page throgh tag names and its attributes. For example to get the title of page

 >>> soup.title
 <title>Simple Web Page</title>

 >>> soup.title.string
 u'Simple Web Page'

 >>> soup.title.name
 'title'

To get the paragraph :

  >>> soup.p
 <p class="heading">Hello world : A Sample Page test</p>
 
 >>> print soup.p.contents
 [u'Hello world : A Sample Page test']

 >>> soup.p.string
 u'Hello world : A Sample Page test'

To get all paragraph

  >>> soup.find_all('p')
 [<p class="heading">Hello world : A Sample Page test</p>, <p class="sub-heading">This the Sub-heading or Description</p>]

Extracting table information

   >>> soup.body.table
 <table>\n<tr>\n<th>Firstname</th><th>Lastname</th> <th>Age</th>\n</tr>\n<tr>\n<td>Frank</td><td>Castle</td><td>40</td>
 \n</tr>\n<tr>\n<td>Jack</td><td>Rietcher</td><td>45</td>\n</tr>\n</table>
 
 >>> soup.body.table.tr
 <tr>\n<th>Firstname</th><th>Lastname</th> <th>Age</th>\n</tr>
 
 >>> soup.body.table.tr.th
 <th>Firstname</th>
 
 >>> soup.body.table.tr.find_all('th')
 [<th>Firstname</th>, <th>Lastname</th>, <th>Age</th>]

printing all rows data in table

 >>> for dat in soup.body.table.find_all('tr'):
 ...      for ind in dat.find_all('td'):
 ...             print ind.string
 ... 
 Frank
 Castle
 40
 Jack
 Rietcher
 45

Getting link information

  >>> soup.body.a
 <a href="http://wikipedia.org/">Wikipedia</a>
 
 >>> soup.body.find_all('a')
 [<a href="http://wikipedia.org/">Wikipedia</a>, <a href="http://youtube.com">Watch Videos</a>, <a href="http://google.com">Search Something</a>, 
 <a href="http://facebook.com">Find Your Friends</a>]

print all the links :

 >>> for link in soup.body.find_all('a'):
 ...     print link['href']
 ... 
 http://wikipedia.org/
 http://youtube.com
 http://google.com
 http://facebook.com

With next_sibling and previous_sibling we can navigate between page elements that are on the same level

 >>> soup.body.p
 <p class="heading">Hello world : A Sample Page test</p>

 >>> soup.body.p.next_sibling
 u'\n'

 >>> soup.body.p.next_sibling.next_sibling
 <p class="sub-heading">This the Sub-heading or Description</p>

Parsing a Webpage with urllib2 and BeautifulSoup :

first imoprt all the necessary libraries

 from bs4 import BeautifulSoup
 import urllib2

Now to get the page, send the get request to the page url https://www.w3.org, and parse it with BeautifulSoup

 page = urllib2.urlopen("https://www.w3.org")
 soup = BeautifulSoup(page, 'lxml')

Now get the title of the page

 >>> soup.title.string
 u'World Wide Web Consortium (W3C)'

 >>> soup.body.p
 <p class="bct"><span class="skip"><a accesskey="2" href="#w3c_most-recently" tabindex="1" title="Skip to content (e.g., when browsing via audio)">Skip</a></span></p>

Harvesting all the links

 >>> for link in soup.find_all('a'):
 ...     print link['href']
 ... 
 /
 /standards/
 /participate/
 ------------
 -----------
 https://www.w3.org/WAI/videos/standards-and-benefits.html
 https://www.w3.org/WAI/videos/standards-and-benefits.html
 http://lists.w3.org/Archives/Public/site-comments/
 http://twitter.com/W3C
 http://www.csail.mit.edu/
 http://www.ercim.eu/
 http://www.keio.ac.jp/
 http://ev.buaa.edu.cn/

Printing all the text from page :

 print (soup.get_text())

And with like the above we can collect information by parsing the web pages.

Conclusion :

The BeautifulSoup is very powerful library to parse the HTML and XML documents and collecting data from it. At above we saw some very basic example of how to use it. And For more detail information please check the documentation page at here : Official Documentation

A Beginners Guide to requests Library in Python : How to Use requests library in Python

Most existing Python modules for sending HTTP requests are extremely verbose and cumbersome. Python’s builtin urllib2 module provides most of the HTTP capabilities you should need, but the api is thoroughly broken. It requires an enormous amount of work (even method overrides) to perform the simplest of tasks.

How to use urllib2 module in Python : A Beginners guide to urllib2 module

urllib2 is a Python module for fetching URLs. It offers a very simple interface, in the form of the urlopen function. The urllib2 is capable of fetching URLs using a variety of different protocols and it also supports slightly complex interface for handling common situations - like basic authentication, cookies, proxies and so on.

Banner Grabbing with Python

In the previous post we saw the basics of banner grabbing techniques. Now we are trying to grab banners with some python scripts. In python we are going to use socket module to connect to target services. The socket module exposes the low-level C API for communicating over a network using the BSD socket interface.

The below python script simply convert Hexadecimal inputs into ASCII Characters

 #!/usr/bin/python  
 import sys  
 if(len(sys.argv) != 2):  
   print "\n\tUsage: %s <hex_values>\n" % sys.argv[0]  
   print "\t  Ex: %s 41424344\n" % sys.argv[0]  
   sys.exit(1)  
 tl = len(sys.argv[1])  
 x = 0  
 str = ""  
 while((x+2) <= tl):  
   ch = chr(int(sys.argv[1][x:x+2], 16))  
   print "[0x" + sys.argv[1][x:x+2] + "] = " + ch  
   str += ch  
   x += 2  
 print  
 print "String= " + str + "\n"

Code for HTTP Sniffing with Raw Socket in Python

here is the code:

HttpSniff.py

 #!/usr/bin/python  
   
 import socket  
 import struct  
 import binascii  
   
 def mac_print(mac):  
  mac_ad = "%.2x:%.2x:%.2x:%.2x:%.2x:%.2x" % (ord(mac[0]), ord(mac[1]), ord(mac[2]), ord(mac[3]), ord(mac[4]), ord(mac[5]))  
  return mac_ad  
   
 RawSocket = socket.socket(socket.PF_PACKET, socket.SOCK_RAW, socket.htons(0x0800))  
   
 while True:  
  packet = RawSocket.recvfrom(65565)  
    
  # Check for the TCP packets  
  IpHeader = packet[0][14:34]  
  TcpHeader = packet[0][34:54]  
  ip_hdr = struct.unpack("!B8s1s2s4s4s", IpHeader)  
  tcp_hdr = struct.unpack("!HHLLB7s", TcpHeader)  
  if binascii.hexlify(ip_hdr[2]) == "06" and (tcp_hdr[0] == 80 or tcp_hdr[1] == 80):  
  # Check for the TCP protocol and port 80 [HTTP]  
    
  # Extracting the Mac Address from EtherNet Header  
  dst_mac = mac_print(packet[0][0:6])  
  src_mac = mac_print(packet[0][6:12])  
   
  # Extracting the IP address from IP header  
  src_ip = socket.inet_ntoa(ip_hdr[4])  
  dst_ip = socket.inet_ntoa(ip_hdr[5])  
   
  # Extracting Source and Destination Port  
  src_port = tcp_hdr[0]  
  dst_port = tcp_hdr[1]  
   
  # Calculating the length of data  
  eth_length = 14  
  iph_length = ip_hdr[0]  
  iph_length = (iph_length & 0xF) * 4  
  tcph_length = tcp_hdr[4]  
  tcph_length = (tcph_length >> 4) * 4  
  hdr_length = eth_length + iph_length + tcph_length  
  data_length = len(packet[0]) - hdr_length  
  Data = packet[0][hdr_length:]  
  if Data == None:  
   continue  
  else:  
   # print all The Data  
   print "Source { IP : " + str(src_ip) + " | Mac : " + src_mac + " | Port : " + str(src_port) + " }"  
   print "Dest. { IP : " + str(dst_ip) + " | Mac : " + dst_mac + " | Port : " + str(dst_port) + " }"  
   print "Data : " + Data  
   print "---------------------------------------"

run this code with root privilege otherwise it may not work, and also you need to generate some http traffic by yourself.

 ajay@ubuntu:~$ sudo ./HttpSniff.py  
 Source { IP : 192.168.56.1 | Mac : 0a:00:27:00:00:00 | Port : 47708 }  
 Dest. { IP : 192.168.56.101 | Mac : 08:00:27:5e:26:c3 | Port : 80 }  
 Data :   
 ---------------------------------------  
 Source { IP : 192.168.56.1 | Mac : 0a:00:27:00:00:00 | Port : 47708 }  
 Dest. { IP : 192.168.56.101 | Mac : 08:00:27:5e:26:c3 | Port : 80 }  
 Data :   
 ---------------------------------------  
 Source { IP : 192.168.56.1 | Mac : 0a:00:27:00:00:00 | Port : 47708 }  
 Dest. { IP : 192.168.56.101 | Mac : 08:00:27:5e:26:c3 | Port : 80 }  
 Data : GET / HTTP/1.1  
 Host: 192.168.56.101  
 User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:47.0) Gecko/20100101 Firefox/47.0  
 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8  
 Accept-Language: en-US,en;q=0.5  
 Accept-Encoding: gzip, deflate  
 Connection: keep-alive  
 If-Modified-Since: Mon, 27 Jun 2016 17:03:35 GMT  
 If-None-Match: "2cf6-536458034b832-gzip"  
 Cache-Control: max-age=0

Simple Python Web Server with CGI

Here is the code for python CGI Web server :

CgiServer.py

 #!/usr/bin/env python  
   
 import BaseHTTPServer  
 import CGIHTTPServer  
 import cgitb; cgitb.enable() # This line enables CGI error reporting  
   
 ServerHandler = CGIHTTPServer.CGIHTTPRequestHandler  
 ServerHandler.cgi_directories = ["/"]  
 HttpServer = BaseHTTPServer.HTTPServer(("", 8000), ServerHandler)  
 HttpServer.serve_forever()

and here's the code for a sample page:

TasgePage.py

 #!/usr/bin/env python  
   
 print """Content-type:text/html\r\n\r\n  
 <html>  
 <head>  
  <title>Test Page</title>  
 </head>  
 <body>  
  <center><h1>Hello world. This is The Test Page</h1></center>  
 </body>  
 </html>"""

then set the executable permissions for both file and run the CgiServer.py

 $ chmod +x CgiServer.py  
 $ chmod +x TestPage.py  
 $ ./CgiServer.py

now oen your web browser and type the following url

http://localhost:8000/TestPage.py

voila!!, thats it. For more information visit : https://wiki.python.org/moin/CgiScripts

Simple Echo Server in python using Sockets

A simple echo server which just echo the client supplied data or string

code : Server.py

 #!/usr/bin/python  
   
 import socket  
   
 tcpSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)  
 tcpSocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)  
   
 tcpSocket.bind(("0.0.0.0", 8000))  
 tcpSocket.listen(2)  
   
   
 while 1:  
  print "Waiting for a Client ... "  
  (client, (ip, sock)) = tcpSocket.accept()  
   
  print "Received connection from : ", ip  
  client.send("Press Return or Ctrl+C to close..\n")  
  print "Starting ECHO output ... "  
   
  data = 'dummy'  
   
  while len(data):  
  data = client.recv(2048)  
  if len(data)==1:  
   print "Closing connection with ", ip  
   client.close()  
   print "Connection closed successfully.!!"  
   print "---------------------------------"  
   break  
  if len(data)==0:  
   print "Some Error in connection with ", ip  
   print "Connection closed with ", ip  
   print "---------------------------------"  
   break  
  print "Client sent:", data  
  client.send(data)  
   
 tcpSocket.close()

But the above server code process only a single client at a time.

Now with the use of threading we can solve this problem. Here is the second echo server which handle multiple connection with threads

Threaded_server.py

 #!/usr/bin/python  
   
 import thread  
 import socket  
   
 tcpSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)  
 tcpSocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)  
   
 tcpSocket.bind(("0.0.0.0", 8000))  
 tcpSocket.listen(2)  
   
 def conn_handler(client, ip, thread_id):  
      print "[T%d]Received connection from : %r" % (thread_id, ip)  
      client.send("Press Return or Ctrl+C to close..\n")  
      print "[T%d]Starting echo output..." % thread_id  
      data = 'dummy'  
      while len(data):  
           data = client.recv(2048)  
           if len(data)==1:  
                print "[T%d]Closing connection with %r" % (thread_id, ip)  
                client.close()  
                print "[T%d]Connection closed successfully.!!" % thread_id  
                print "------------------------------------"  
                break  
           if len(data)==0:  
                print "[T%d]Some Error in connection with %r" % (thread_id, ip)  
                print "[T%d]Connection closed with %r" % (thread_id, ip)  
                print "------------------------------------"  
                break  
           print "[T%d]Client sent: %s" % (thread_id, data)  
           client.send(data)  
   
   
 thread_id = 0  
 while 1:  
      thread_id = thread_id + 1  
      print "Waiting for Client ...\n"  
      (client, (ip, sock)) = tcpSocket.accept()  
   
      try:  
           thread.start_new_thread(conn_handler, (client, ip, thread_id, ))  
      except:  
           print "Error: Unable to start thread [T%d]\n" % thread_id  
   
 tcpSocket.close()

Multi-Process Echo Server

 #!/usr/bin/python  
   
 from multiprocessing import Process  
 import socket  
 import os  
 import signal  
   
 tcpSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)  
 tcpSocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)  
   
 tcpSocket.bind(("0.0.0.0", 8000))  
 tcpSocket.listen(2)  
   
 def conn_handler(client, ip, Child_id):  
      print "[C%d]Received connection from : %r" % (Child_id, ip)  
      print "[C%d]Starting echo output... " % Child_id  
      data = 'dummy'  
      while len(data):  
           data = client.recv(2048)  
           if len(data)==0:  
                print "[C%d]Closing connection with %r" % (Child_id, ip)  
                client.close()  
                print "[C%d]Connection closed with %r" % (Child_id, ip)  
                os.kill(os.getpid(), signal.SIGTERM)  
           print "[C%d]Client sent: %s" % (Child_id, data)  
           client.send(data)  
   
 def main():  
      Child_id = 0  
      while 1:  
           Child_id = Child_id + 1  
           print "Waiting for Client ...\n"  
           (client, (ip, sock)) = tcpSocket.accept()  
   
           try:  
                Process(target=conn_handler, args=(client, ip, Child_id)).start()  
           except:  
                print "Error: Uable to start Child Process.!![C%d]" % Child_id  
        
      tcpSocket.close()  
   
   
 if __name__ == "__main__":  
      main()

you can try out with nc, to close the connection just press 'Ctrl + C' or use the below client.py code to communicate with multiprocess_server.py

Client.py

 #!/usr/bin/python  
 import socket  
 import sys  
 tcpSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)  
 tcpSocket.connect((sys.argv[1], int(sys.argv[2])))  
 print "Input some string : ['quit' to exit]"  
 data = "dummy"  
 while 1:  
      data = raw_input("|> ")  
      if data=="quit":  
           tcpSocket.close()  
           break  
      tcpSocket.sendall(data)  
      result = tcpSocket.recv(2048)  
      print result