Scripting HTTP Clients and Web Interactions

Explore the HTTP protocol's request-response model, its stateless nature, and how cookies enable sessions, covering methods like GET and POST, status codes, and ports 80 and 443.

Understand the http get request and how a client retrieves resources using the get method, headers, and query parameters. Learn why the body is empty and how http is stateless.

Post operations send data to the server by placing form field values or JSON in the request body, targeting a URL with the content type, and https protects the body.

read html forms to understand how form method and action generate dynamic get or post http requests, where get puts data in the url and post in the body.

Explore wget, a portable command-line tool for interacting with web content and downloading files, supporting get and post requests across protocols with option-driven syntax and help redirection.

Use wget to issue HTTP get requests and retrieve resources by URL, saving index.html or the full page, while logging transfer metrics with -o and headers with -S.

Crafted http get and post teaches using wget to build precise get and post requests with parameters, headers, and body data for login and user detail retrieval.

Learn to download files with wget from a web location, save them to a specified path with -P, manage multiple URLs with -I, and run in background with -B.

Use wget to mirror a website with a recursive download, convert links to local references, pull CSS/JS/images, prevent climbing above root, store content locally, enabling offline browsing.

Compare authentication options for git scripts, including direct user/password, interactive prompts, form-based posting, basic auth, and token headers, noting password exposure and https requirements.

Curl is a cross-platform command line tool for interacting with http, enabling data transfers, and get or post requests, and retrieving headers.

Master http get requests with curl to fetch a web page's html from a url. Use --include to see headers, or --head to retrieve only headers.

Post data to a server using http post operations by encoding form fields and sending them with curl --data to login.php.

Learn how curl handles sessions and cookies using the -b and -c options to read from and write to a cookie file, enabling stateful post-login interactions with a web app.

Learn to script file uploads with curl using the -F/--form option for multipart/form-data and post to upload.php. Handle single or multiple files and optional put-based uploads with -T.

Use curl with the capital O option to save a remote file locally. This mirrors the remote content to your current directory for an efficient download.

Explore curl authentication options, including basic auth via dash U or URL-embedded credentials, and form-based or token-based approaches using the dash H header for OAuth or bearer tokens.

Learn how curl uses https with tls to encrypt data in transit and perform mutual authentication via client certificates (pem) and options like --cert, passphrase handling, and --insecure.

Issue a curl http get with no parameters to a target url and retrieve the raw html without rendering. Http is stateless, so each resource requires a separate request.

Demonstrates posting a login form to a vulnerable web app with curl, using -b and -c to manage cookies, sending username and password, and retrieving a PHP session ID.

Demonstrate retrieving a remote html page with curl and saving it locally as dvwa.html using the lowercase -o option, and review the transfer stats.

Explore Python scripting to build http clients with the requests library, including get, post, put requests, sessions, cookies, authentication, and file transfers via pip.

Learn how to issue http get requests in python with the requests library, capture response.text, and understand how browsers render pages vs. scripting single-request responses.

Learn to post data with the Python requests module by specifying a target URL, building a post body from form fields and values, and handling the server response.

Create and use a single http session with requests to perform get and post operations, sharing cookies across requests and maintaining the php session id.

Upload a file from client to server in under ten lines of Python using the requests module, including binary file handling and posting to a target URL.

Use Python and the requests module to download a remote file by fetching it and saving its binary content to a local file, following redirects and printing the status.

Learn how to implement header based authentication by sending an access token in a custom authorization header and use HTTP basic authentication with a username and password in Python requests.

Demonstrate a simple Python http client that issues a GET request to a target URL using the requests library. Print the HTML response to the console.

Demonstrate issuing a post request with a Python client to a login page, sending a post body, and verifying a 200 status and an empty cookie dictionary before session handling.

Demonstrate maintaining a stateful session in Python by posting login data, capturing the session cookie, and issuing a follow-up get request under the same session to confirm identical cookies.

Download a compressed python resource file containing code examples from this section. Use the provided code as is, and modify variables to run it in your environment.

Learn to use PowerShell's invoke web request to perform get and post Http requests, manage a session, and inspect the basic HTML web response object with content, headers, and status.

Learn how to issue get requests with PowerShell's invoke web request, store the response object, and inspect content, headers, and the http response code for pages, links, and images.

Explore posting data with PowerShell using invoke web request, specifying post method and a form body with username and password, then handling responses and status codes.

PowerShell demonstrates sessions with Invoke-WebRequest, using a session variable and web session options to keep initial and subsequent requests under the same session, reuse cookies, and share session state.

Upload files from client to server using the put method with invoke web request, set the local file path and content type, and ensure the server accepts put.

Explore two PowerShell approaches to download files for any file type, using invoke web request with -out file and the web client's download file method.

Explore PowerShell authentication for web requests, using invoke-webrequest with basic, bearer, and OAuth; manage credentials, build headers, and secure tokens for secure HTTP interactions.

Build a PowerShell web client that performs an HTTP GET using Invoke-WebRequest, explores the response object, and inspects status, content, headers, forms, and links.

Demonstrate using PowerShell's Invoke-WebRequest to post form data to a login page on a vulnerable web app, build a form body, and verify a 200 status code.

Demonstrate managing a web session in PowerShell by posting credentials, reusing a session variable, retrieving cookies, and navigating authenticated pages like index.php.

Download a compressed PowerShell code resource file containing many examples from this section. Use the code as is, but modify sections or variable values to run it in your environment.

Explore the course's additional content section, featuring student questions, requests, demonstrations, and resources that didn't make the initial deadline, plus code cheat sheets and new lessons—check back often.

Explore a curl cheat sheet, a single-page reference with basic syntax and practical examples. Download the PDF to print and quickly reference curl options.

Learn to issue an HTTP GET with pure JavaScript using the XMLHttpRequest object in an HTML page, configure the request, send it, and display the 200 response.

Learn to issue a post request in pure JavaScript using the XML HTTP request object within an HTML page, sending JSON data and handling a 200 response.

Master scripting http interactions to extract information, simulate users, post data, and use get, post, and put with authentication and file uploading and downloading across cross-platform tools.