Web Scraping: Extract Structured Data from Web Pages

What you're probably confused about right now

"Isn't scraping just copying text off a webpage?"

Not even close. Scraping means programmatically extracting specific fields while handling errors and staying compliant.

One-line definition

Web scraping is requesting a page via code, parsing the HTML, and pulling out the data you actually need.

Real-life analogy

Copying a contact list by hand vs. automatically collecting and organizing it into a spreadsheet.

Minimal working example

import requests
from bs4 import BeautifulSoup

resp = requests.get("https://example.com", timeout=10)
soup = BeautifulSoup(resp.text, "html.parser")
print(soup.title.string)

Quick quiz (5 min)

1. Extract the page title and all h2 tags.
2. Add retry logic (2 attempts on failure).
3. Output a count of scraped items.

Quiz answer guide & grading criteria

• Answer direction: write runnable code that covers the core requirements and edge cases from the prompt.
• Criterion 1 (Correctness): Main flow produces correct results, key branches execute.
• Criterion 2 (Readability): Clear variable names, no excessive nesting.
• Criterion 3 (Robustness): Basic protection against null values, type errors, or unexpected input.

Take-home task

Build a small scraper: crawl a public page listing and export to JSON.

Acceptance criteria

You can independently:

• Scrape with requests + BeautifulSoup
• Handle timeouts and empty results
• Explain compliance boundaries

Common errors & debugging steps (beginner edition)

• Can't read the error: start from the last line -- find the error type (TypeError, NameError, etc.), then trace back to the line in your code.
• Not sure about a variable's value: throw in a temporary print(var, type(var)) at key points to verify data looks right.
• Changed code but nothing happened: make sure the file is saved, you're running the right file, and your terminal is in the correct venv.

Common misconceptions

• Misconception: if you can access a webpage, you can scrape it freely.
• Reality: respect robots.txt and the site's terms of service.