Web Requests

2 main tools :

• cURL
• Browser DevTools

Concepts :

• http
• https
• http request and responses and their headers
• http methods and response codes
• interacting with APIs

HTTP.

Scheme and host are mandatory

Our browser usually look first in the local ‘/etc/hosts’, if the domain can not be resolved, it will then ask to a DNS Server.

Return status 200 to web browser if everything is ok.

Using cURL :

• cURL does not render the html/js/css
• curl -O lal.com/index.html will download the file
• curl -s -O will silent the process
• curl -h to see all other options
• man curl to view the full curl manual page

exemple :

cat page.php to open the file

Browser DevTools

Ctrl + shift: show devtools

Ctrl+ shift + e: network tab

Ctrl+ shift + k: console tab

Man in the middle attack

In http, all of the data is transferred in clear text. This mean that anyone between the source and destination can perform a man in the middle attack. For example, username and password can be extract from http requests.

https transfer data in an encrypted format. Soon web browsers would only allow this kind of request.

https communication steps :

• client hello
• server hello
• ssl certificate exchange
• handshake
• communication

curl -k to skip ssl certificate check

http requests and responses

REQUESTS

http 1 sends request as clear text

http 2 sends request as binary in  a dictionary form

RESPONSES

http responses can return html, json, pdf file, style sheets, images, scripts etc….

2 parts separated by a newline :  headers + body

in order to view the full http request we can write

curl inlanefreight.com -v

Network tab in DevTools are responsible for web requests

HTTP headers

General headers:

• Date
• Connection

Entity headers:

• Content-type
• Media-type
• Boundary
• Content-Length
• Content-Encoding

Request headers:

• Host
• User-Agent
• Referer
• Accept
• Cookie
• Authorization

Response headers:

• Server
• Set-Cookie
• WWW-Authenticate

Security headers:

• Content-Security-Policy
• Strict-Transport-Security
• Referrer-Policy

Curl -I or curl-I to get only the header response

Requests methods:

• GET : request a resource
• POST : send data to server
• HEAD : request the header (such as get but without the data)
• PUT : create new resource on the server (useful to load malicious resources)
• DELETE : deletes existing resource (can lead to DOS by deleting critical files on server)
• OPTIONS : returns information about the server
• PATCH : applies partial modifications to the resource
• And much more

Mostly use in web app : GET and POST

Rests APIS also rely on PUT and DELETE

Returns code

HTTP GET

Basic HTTP auth with GET

If in header

WWW-Authenticate: Basic realm=”Access denied”

or stand http address as mentioned above

HTTP POST

Unlike http GET which places user parameter in the URL, http POST places user parameters within the http Request Body with the following benefits:

• Lack of logging
• Less encoding requirements
• More data can be sent (url have a max length)

In DevTools if we click on a POST request, we can see the raw data being sent.

To send a POST request we must use the -X POST flag to curl and then specify the data being sent with the -d flag.

Exemple

We can use the -L flag to follow the redirection of the url

Once authenticated we receive a cookie so our browser can persist our authentication and we don’t need to login everytime.

Use -v flag to view the Set-Cookie value

We can then use the cookie value to launch any GET request

or in the header

Having a valid cookie may be enough to get authenticated into web applications. This can be an essential part of some web attacks, like Cross-Site Scripting!

Json in post request

In the header, we would see content-type:application/json. We can specify all header parameter in the curl command with the -H flag

CRUD API and REST API

There are several types of APIs. Many APIs are used to interact with a database.

in general APIs performs 4 mains operations on databases:

• Create : POST
• Read : GET
• Update : PUT
• Delete : DELETE

 

Same principle are also used in REST API. API will return a json string as result. Exemple of CRUD API request and results (-s to silent curl, |jq to nicely represent the json

 

 

curl -s http://<SERVER_IP>:<PORT>/api.php/city/london | jq

[

{

“city_name”: “London”,

“country_name”: “(UK)”

}

]

Introduction no networking

The internet is made of a multitude of small networks.

• an FQDN (www.hackthebox.eu) only specifies the address of the “building” and
• an URL (https://www.hackthebox.eu/example?floor=2&office=dev&employee=17) also specifies the “floor,” “office,” “mailbox” and the corresponding “employee” for whom the package is intended.

We usually subdivide company network in small networks with their own properties/securities. Printers are usually a weak point in networks and store a lot of sensitive informations (it is due to how windows works).

Network types:

Wide Area Network : WAN à the Internet. The WAN address is the one generally accessed by internet. Generally speaking, the primary way we identify if the network is a WAN is to use a WAN Specific routing protocol such as BGP and if the Schema in use int not within RFC 1918.

Local Area Network : LAN à Internal Networks (Home or office). This network will typically assign IP Addresses designated for local use (RFC 1918, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16)

Wireless Local Area Nerwork : WLAN à Internal Networks accessible via WIFI. Same as WLAN

Virtual Private Network : VPN à Connects multiple network site to one LAN:

• Site to Site VPN : both client and server are Network Devices (ie : Routers, Firewalls). Commonly used to join company networks together over the Internet.
• Remote Access VPN : involve the client’s computer creating a virtual interface that behaves as if it is on client’s network
• SSL VPN : VPN that is done within web browser

Global Area Network : GAN à Global network (the internet): Glass fibers international network

Metropolitan Area Network : MAN à Regional network (connects multiple LANs)

Wireless Personal Area Network : WPAN à Personal network (Bluetooth)

Network topologies:

Topology is a typical arrangement of physical and logical connection of devices (computers, switches, bridges, routers,…) in a network. Computers are hosts such as clients and servers.

Connections:

Wired connections:

• Coaxial cabling
• Glass fiber cabling
• Twisted pair cabling

Wireless connections:

• Wifi
• Cellular
• Satellite

Topology is a virtual form or structure of a network, basic types are:

• Point to point : telephony
• Star
• Mesh : WAN and MAN
• Hybrid
• Bus
• Ring
• Tree
• Daisy chain : CAN

Proxies:

A proxy act as a mediator in the middle of the connection. (different than a gateway that cannot inspect the contents of the traffic). Technically a VPN is not a proxy. There 3 main types of proxies:

• Dedicated Proxy / Forward Proxy : carry out a request, can act as a filter, filter outgoing request
• Reverse Proxy : filter ingoing request
• Transparent Proxy

Networking Models:

OSI Model : 7 Layers

TCP/IP Model : 4 Layers

Network layers protocols:

• IPV4/IPV6
• IPsec
• ICMP
• IGMP
• RIP
• OSPF

IP addresses :

• Each host is identified by a Media Access Control address (MAC). If hosts are located on different network, then MAC addresse is not enough to establish a connection
• Addressing on the internet is done via the IPv4 or IPv6 address (Building of the receiver, MACà exact position of the appartement in the building)

IPv4 structure:

4 bytes consisting of 8 bit groups (octets) ranging from 0-255. Converted in readbale decimal numbers.

Ex : 127.0.0.1

IPv4 allows 4 billions unique addresses. The ip address is divided into a host part and a network part. The router assigns the host part, the network administrator assigns the network part. On the internet, this is IANA which allocates and manages the unique ips.

Classes of ip:

A smaller division of these classes into smaller network is done with subnet mask.

Last address represent the broadcast address of a network.

Octet format:

128 – 64-32-16-8-4-2-1

1 0 0 0 0 0 0 0 gives 128

1 0 0 0 0 0 0 1 gives 129

CIDR : Classless Inter_Domain Routing

CIDR : 192.168.10.39/24. 24 represent the qty of 1-bits in the subnet-mask

1^st,2^nd,3^rd, and two first bits of the 4^th octet are the network part

Last 6 bits of the 4^th octet are the host part

All bits of host part to 0 à subnet network address

All bit of the host part to 1 à broadcast address

When an ip packet is delivered it contains a sender address and a destination address. The MAC address consists of 6  octets. The first half (3 octets) identifies the manufacturer (OUI : Organization Unique Identifier). The last part is called the Individual Address Part or NIC Network Interface Controller.

During transmission of a packet :

• If a host with IP target is in same subnet, the delivery is made directly to the target
• IF a host belongs to a different network, the delivery is addressed to the MAC
• Address resolution protocol (ARP) is made to solve MAC—IP correspondence

MAC addresses :

Each host in a network has its own 48-bit (6 octets) Media Access Control (MAC) address, represented in hexadecimal format. MAC is the physical address of our network interface. There are different format for MAC address :

• Ethernet
• Bluetooth
• WLAN

MAC address is configured by the manufacturer, but can be changed at least temporarily. Some examples of MAC address

 

Http request Footprint

Attributs de l’en-tête http

User Agent : OK

Accept : NOK

Content encoding : NOK – pas utile

Content language : OK

Headers.sec-ch-ua.name : NOK

Headers.sec-ch-ua.name :  NOK

Headers.sec-ch-ua-platform.name : NOK

Possibilité d’ajouter du javascript?

Plateforme

Utilisation de cookiers

Fusau horaire

Langage

Canvas

Liste des polices de caractères (JS) 10.200.20.0/27

Navigator properties

Product

Hardware concurrency

Largeur de l’écran

Hauteur de l’écran

Permissions

Vendor WebGL

Renderer WebGL

Paramètres WebGL

Contexte audio

Agencement du clavier

Batterie

Connexion

To collect browser footprint does not need to install a cookie. Several techniques can be used to collect data :

• Http headers
• Javascript
• Flash plugin
• Html5 Canvas
• Plugin Detect

Certificats : not verifying a certificate can expose application to security risks, such as man-in-the-middle attacks.

Cookies data can be retrieved to identify a unique user. Google analytics id typically expires every 2 years and combined with IP address can be a good parameter to identify a unique device in a graph.

_ga values are unique for a website and for a user. If the same user goes to another website, it will generate another _ga value.

_gid has a lower expiration value.