You finished the tutorial. You wrote 20 lines of Python, pressed run, and a clean list of product names filled your terminal. It felt like magic. So you ran it again the next morning, and this time you got an error, an empty page or a polite message asking if you are a robot.
Welcome to the part most beginner courses skip. Your code probably did nothing wrong. What changed is that the website noticed you.
This guide walks through why that happens, what each error is telling you and how to build a scraper that lasts longer than a day.
Why It Worked the First Time
A website does not usually judge you on one request. One visit looks like anyone else. Trouble starts with patterns, and scrapers are very good at creating patterns.
Look at what your script sends. Every request carries a User-Agent header, which identifies the application making the request. Browsers send long strings naming the browser and operating system. The Python requests library announces itself with something like python-requests followed by a version number. To a server, that is a plain statement that a script is knocking.
Then there is speed. A person reads a page, scrolls and clicks. Your loop can ask for 100 pages in the time it takes to sip your coffee. Sites protect themselves with rate limiting, and a reply of status 429 means you have sent too many requests in a given amount of time. It may even include a Retry-After header telling you how long to wait.
Finally, there is your address. All of that traffic comes from one IP. Once a site has flagged it, every later run starts with a bad reputation.
When Residential Proxies Make Sense
Beginners often jump straight to tools when the real problem is manners. So fix your script first, using the steps below. If your project is legitimate and has simply outgrown a single connection, this is where proxies come in.
A proxy sends your request through another address. Datacenter proxies are cheap, but their ranges are easy to recognize. Residential proxies route traffic through rotating household IPs, so requests look like ordinary home connections and can come from different locations. That matters for jobs such as checking regional prices or search results from another city.
They are not a license to ignore a site’s rules. Used well, a proxy spreads a reasonable workload. Used badly, it just lets you annoy a server from more places, and that usually ends the same way. Treat it as a later step in your learning, not the first one.
Decode the Message You Got
Each failure tells you something different. Learning to read them is a real skill, and it will help you far beyond scraping.
403 Forbidden
The server understood you and said no. This often points to your headers, your IP reputation or a rule that blocks automated tools. Print the response body before you change anything. The text often explains the reason.
429 Too Many Requests
This is the friendliest one, because the site is telling you exactly what is wrong. Slow down, read the Retry-After value and wait.
200 OK, but no data
This one confuses beginners most. The request worked, yet your list is empty. Usually the content is loaded later by JavaScript, so it is missing from the HTML that requests received. Check the page source in your browser, not the inspector. A browser automation tool such as Playwright may help here.
A CAPTCHA page
You got a 200 response, but the HTML is a challenge page. Save the response to a file and open it. Many beginners spend hours fixing a parser that was never given the real page.
It works in Chrome but not in Python
This is a sign that your request looks different from a browser’s. Compare the headers in your browser’s Network tab with the ones your script sends. That single habit can save you an afternoon.
Build a Scraper That Behaves
Most blocks disappear when you act like a considerate guest. Start with these habits:
- Read the rules first. Check robots.txt and the site’s terms before you collect anything.
- Add delays. Put a pause between requests, and randomize it a little. The time.sleep function is your friend.
- Use a session. A requests.Session keeps cookies and reuses connections, which is more efficient for you and the server.
- Handle errors. Catch failed responses, wait longer after each retry and stop after a few attempts.
- Save your results. Store pages locally while you develop, so you are not requesting the same page 50 times to fix one bug.
- Look for an API. Many sites offer one. It is cleaner, more stable and clearly allowed.
Robots.txt Is a Great First Lesson
There is an underrated bit of web history here. The Robots Exclusion Protocol lets site owners tell crawlers what they may access. RFC 9309 formalized it in 2022, and it makes clear that these rules are not a form of access authorization. Nothing technically stops you from ignoring them. That is exactly why respecting them says something about how you work.
Python even ships with a tool for this. The urllib.robotparser module can read a site’s file and tell you whether a URL may be fetched, using its can_fetch method. It can also report any crawl delay the site asks for. It is only a few lines, and it is a great addition to your first project.
Being Blocked Means You Are Learning
Every working scraper has been blocked many times along the way. The block is your first real lesson in how the web works: HTTP, headers, sessions, rate limits and the people who run the servers.
Keep a small notebook of every error you hit and what fixed it. In a few months, that notebook will be worth more than any tutorial.
So do not give up on that 20-line script. Open the response, read what it says and fix one thing. Then run it again, a little more slowly this time.
