Advanced Web Scraping with Python using Scrapy & Splash

Explore scraping centrista real estate listings by analyzing network requests, identifying xhr payloads for Montreal residential rentals, and using postman to replicate browser api calls without splash or selenium.

Learn to debug web APIs with Postman by sending get and post requests, configuring headers and json payloads, and using a two-step approach for scraping—update query, then get inscriptions.

Implement update query and get inscriptions requests in Scrapy by overriding start_requests. Send a post with a JSON payload, configure headers, and set a callback to process the response.

Post to the get inscriptions endpoint with a start position that increments by 12. Parse the JSON response to HTML, then extract listings using Scrapy selectors and XPath.

Learn to implement robust XHR pagination in Scrapy by tracking the start position key and the count key, calculating the increment, and issuing post requests to retrieve all listings.

Determine if a page needs JavaScript, then use Splash or Selenium to render HTML and scrape listing addresses and descriptions with Scrapy, building absolute URLs and passing meta data.

Increase the virtual machine memory and CPU for splash in docker, then configure splash with restart always and port mapping to reduce 504 gateway timeout errors when scraping.

Learn to bypass 504 gateway timeout by optimizing splash scripts for Scrapy and Splash. Disable images and JavaScript, abort CSS requests, and return HTML to speed up scraping.

Bypass 504 http errors by deploying a splash cluster with aquarium and a load balancer, scaling splash instances via docker compose to render javascript sites and optimize scraping.

Explore scraping the Steam official store to pull top selling games, including image URL, game name, platform, release date, user reviews, and pricing, by building a Scrapy spider.

define each data point as a field in items.py, instantiate the steam item, and extract game url, image url, game name, release date, and pricing with XPath.

Learn to extract multi-platform game data by processing and cleaning data points, utilizing xpath to capture platform spans, parsing class attributes, and building a get_platforms function for a scrapy spider.

Extract review summaries from the data tooltip HTML using XPath in a Scrapy workflow, avoiding hover-based tricks, and clean results with remove_tags while handling missing data gracefully.

Scrape the discount rate, original price, and discounted price using XPath selectors with contains and normalize-space, then clean results with strip and lstrip in Scrapy.

Handle pagination by locating the next page button via its class and href with XPath. Stop after a set item count using a close spider setting, e.g., 100 items.

Move data processing from the spider class to items.py and use an item loader with input and output processors to clean and assemble scraped fields.

Learn to convert fields to strings in Scrapy using input and output processors with take_first. Apply this to game name, game url, img url, and release date, and verify outputs.

Process the review summary field by removing HTML tags and converting to a string, then extract supported platforms from class attributes using map compose and item loader.

Write input and output processors for original price, discounted price, and discount rate in scrapy, converting strings to selectors, cleaning with strip, and joining results into a single string.

learn to run a spider in real time with scrapy rt from a browser endpoint to fetch the top 100 steam games, avoiding a database pipeline.

Set up a Flask web app to fetch scraped game data from Scrapy via a JSON API and display it as a clean JSON response.

Learn how to render templates in Flask with render_template, set up the templates folder, and pass variables and lists using Jinja to create dynamic HTML.

Build responsive interfaces with bootstrap in a Flask template, wiring a static css bundle, a navbar, and a centered jumbotron that showcases Steam top 100 games scraped with Scrapy.

Modify a Flask app to render index.html with a games list, build a Bootstrap grid of three-card rows, and display each game's image, name, platforms, and URL.

Learn to locate and leverage a website's API for scraping by using devtools to inspect network requests, identify XHR calls, and fetch listing data as JSON, even across pagination.

build a scrapy project named Zillow and a spider to fetch Zillow data, bypass reCAPTCHA by using a real user agent, adjust robots.txt and download delay, and parse json.

Test API responses against browser data by simulating requests in incognito mode, inspecting headers, and injecting the cookie header into your Scrapy spider to ensure matching results.

Learn to inject cookie headers in scrapy requests by overriding start_requests or using a cookies dict, and build a custom cookie parser with SimpleCookie to convert cookie strings to dicts.

Scrape json objects with scrapy by converting the response body to a dict with json.loads. Map fields to a scrapy item using an item loader and take first output processor.

Master pagination techniques for web scraping listings across pages. Decode and update the search query state to inject the next page number and build the new URL with Scrapy.

Learn to download and store house images with Scrapy's images pipeline, configure image fields and Pillow, enable the pipeline in settings, and test a spider that saves images locally.

Modify the Scrapy images pipeline to name images using the house id by passing the id through request meta and overriding get_media_requests and file_path.

Design a Tkinter-based desktop app to run scrapy spiders, allowing dynamic spider selection, feed type choice (json or csv), and inputs for output path and dataset name.

Load dynamic spiders into a desktop app using spider loader to populate a dropdown, then capture chosen spider and prepare execution with the execute button.

Prevent a tkinter desktop app from freezing by running the spider in a separate thread using Python's threading module, addressing Twisted main-thread rules to keep the UI responsive.