cross-posted from: https://lemmy.sdf.org/post/58474758

Someone brought the HTtrack app to my attention in this thread. Superficially it’s a great concept. But it did not work for me.

The author of HTtrack says they respect the robots.txt files. I’m not sure if this is my problem. But what an absurd line to draw. Why should a realtime GUI user have more privilege to view a website than a CLI user?

If anything, it should be the other way around.

  • People who lack the priviledge of having Internet at home face this nasty discrimination of being treated like a bot after they make the extra effort of commuting to a public library just to fetch a website for offline viewing later. This 2nd-classing of a demographic who is already marginalized is quite despicable.
  • People with Internet at home can schedule HTtrack to run at an off-peak time with a narrow bandwidth put less burden on the server than the realtime GUI users who obviously hit the site mostly during peak times.
  • (update) Some people are on measured rate Internet connections that give tiny daytime quotas and generous late night quotas. Tools like HTtrack are needed to manage this. Which ultimately benefits the more privileged Internet users who have no constraints.

Update

There is a poorly worded -s0 option to ignore robots.txt. Fooled some people into thinking the tool uses robots.txt to direct the fetches.

  • evenwichtOPM
    1·
    5 days ago

    Are you calling Firefox a scraper then?

    I’m sure there’s a way to wire it up that way.

    Yes, you simply turn off images. Then Firefox is a scraper from the server standpoint. I use FF that way when operating under a bandwidth quota. Scraping a page is far less of a burden on the server than fetching the whole page (images, js, and all) for GUI rendering.

    It’s bound to have a Turing complete language in it somewhere

    Turing completeness is wholly irrelevant. A fetch operation does not absolutely need a turing complete language. But you will struggle to render the content for a graphical presentation without a turing complete language.

    From a practical standpoint there’s a spectrum of activities that approximately go from Browsing to Scraping to DDoS in that order that really only differ in terms of scale.

    Not “in that order”. Browsing graphically causes every image to be fetched. It’s much heavier than the HTML text that a scraper pulls. 10,000 gui users each fetching a page will strain your webserver faster than 10,000 html text pages being scraped.

    Non-graphical users suffer from marginalization driven by the shitty attitude that “bots are bad and humans should alter their behavior and fetch images to look less like a bot as we don’t give a shit about oppressing bots.”

    It sounds like what you’re describing is just timeshifted browsing. I doubt anyone notices unless you’re really describing what you’re trying to do incorrectly.

    Timeshifted browsing is purpose of HTtrack. An AI operation doesn’t use HTtrack which is designed to organize the browsing session into interactive projects which launch a local GUI browser for offline viewing. An AI op has zero interest in that. A scraping operation uses curl, wget, beautiful soup/python, etc.

    until WHOOM! an AI company scrapes their whole site and their hosting cost jumps that month, then the next month and the next as well because companies that would destroy copies of rare books to feed to their training don’t care.

    It’s this kind of commentary that distorts public perception.

    Scraper bots themselves serve many users, often more than the original site could afford to host. A scraper bot that grabs just raw text builds a database and makes the information more useful to more people than the original site. E.g. instead of each traveler manually visiting the website of every single airline (a costly image-rich bloat high in CO₂ emissions), they can go to a site that has scraped Ryanair and others. The aggregated text info is more useful. Consumers then only have to visit /one/ airline website after seeing all the info side-by-side. This is much less strain on resources.

    You’ll often hear/read “follow” used by itself to imply “follow the rules,” in this case “follow the rules specified by robots.txt.”

    That’s what I was saying. Hence why the claim that HTtrack follows locations listed in robots.txt is nonsense.

    From a technical standpoint an LLM recommends trying this code snippet to make httrack ignore the robots.txt file:

    I appreciate the tip. Indeed -s0 is what I worked out that I needed. You seem bent against AI so it’s a bit unexpected that you would be using it. I avoid AI because I generally boycott the corporations pushing the LLMs. But I’m not generally opposed to LLM tech. I have a local language translator and probably need to look into other LLMs that can run locally.