I could really use a commandline mechanism for saving a webpage locally for offline viewing later. Saving the page in a single file is ideal.

“singlefile”, the browser extension

For GUI users there is a quite useful extension (single-file). That can be run from the CLI by installing single-file-cli. But I am not enthusiastic about the bloaty option. You must first install some obscure package manager called “Deno”, which does not exist in the official Debian repositories (thus has not proven to be worthy for mainstream public use, and also complicates every Debian upgrade). Then it uses JavaScript to instrument a bloaty GUI browser. Not great.

Worth noting that there is Org-mode integration but this does not get around the the non-Debian dependency on Deno.

wget

A page can be fetched using this command:

$ wget -U 'stopDiscriminatingAgainstWg3t' -c -P "$dir" -E -H -k -K --xattr -p "$url"
$ printf '%s' "$url" > "$dir"/url.txt

The printf is needed because wget just dumps a big tree of files with no indication of which file must be opened to render the whole page. Indeed it sucks. Saving the URL gives us a fighting chance at locating the file but it’s still a pain. And from there, the files are still scattered on your hard drive.

There is a non-Debian tool called HTMLArk. Again, I am not thrilled about non-Debian stuff. But it claims to be able to produce a single HTML file from a scattered collection of files. I have not tried it but in principle it can be used to tidy up the mess dumped by wget.

ArchiveBox and Grunt-inline

ArchiveBox looks interesting. But it’s non-Debian so not exactly spot-on. Same issue with Grunt-inline. There is some grunt stuff in the Debian repos but not this tool specifically.

webpages2html

This is a python script but the script and its dependencies are non-debian. Requires using pip which is a disaster of a tool that is barely suitable for developers but hardly suitable for the end users who end up getting pushed into it b/c there is no proper pkg manager.

What else?

Any decent option I have overlooked?

  • Freyi
    4·
    4 days ago

    I put links to pages I want to read in a local HTML file and run PyPlucker on it, which scrapes up to a specified depth (usually 1). Then I send the resulting PDB file to my old Palm T3 over Bluetooth and read it all offline. There’s a desktop viewer but I haven’t tested that yet.

    Before that I used to use a Chromium extension “EpubPress” that dumps all open tabs into an EPUB, but that relied on a 3rd party service and no longer works. There’s a project called EpubPressX which tries to do it fully local but it didn’t work for me. There’s also offpunk which is a text-only local CLI offline reader.

    • evenwichtOPM
      1·
      3 days ago

      I usually want to archive a page that can be faithfully reproduced, sometimes for borderline forensic purposes. I usually don’t have time during a browsing session to inspect the results of a saved page for quality. If I pop into a public hotspot I want to save and go.

      But later on when I’m reading and managing the offline content, I might decide the graphics on a page are unwanted. So perhaps it would be useful to do w3m -dump "$my_local_GUI_page.html" > lean_capture.txt && rm $my_local_GUI_page.html". I’ll have to try that.

      (update) I works to run w3m -dump "$my_local_GUI_page.html" > lean_capture.txt but strangely the file:// scheme is not understood. E.g. this does not work: w3m -dump file://./"$my_local_GUI_page.html".

      (update 2) If the webpage is MIME-formatted, w3m does not handle it and just dumps the base64 blobs.

  • pmjv
    31·
    4 days ago

    do you care about it looking like the webpage, or do you just want the text ala reader-mode? if the latter, then

    links2 -dump 'url'

    • evenwichtOPM
      2·
      3 days ago

      Probably rare that I would opt for a lossy text version, but I’ll consider it in some cases along the same lines as my response to the w3m suggestion.

    • evenwichtOPM
      1·
      2 days ago

      Glad to see it’s in the official Debian repos. That encourages me to try it and fiddle around with it.

      The landing page says: “It allows you to download a World Wide Web site from the Internet”, which gives the feeling of a 1990s project. Which isn’t bad… It suggests the dev is probably not keen to use silly modern day bloaty garbage like JS, npm, electron, python, and awkward pkg managers. And indeed it’s in C.

      Although I had to wonder how it notably differs from wget -c -E -H -k -K … Then I saw the --single-file option. So that pretty much settles it – I must check it out. This may be the perfect tool. Thanks for the tip!

      (update) I installed it. Looks promising.

      (update 2) Superficially i love the design… the way it has a server and remembers sessions. A lot of thought went into the design. But I can’t get it to work. There is a config for “proxy”; does not say whether it’s an HTTP proxy or SOCKS. It does not indicate if an URL is expected or just a hostname. My proxy setting does not stick. And no data is fetched in the end. Hard to know if respect for robots.txt is hindering it.

    • evenwichtOPM
      2·
      3 days ago

      Looks great! It’s a shame it’s not in the Debian official repos. I wonder why. I have to say I’m tempted to try it because at least there is an unofficial Debian pkg so we need not use an obscure or language-specific pkg manager.

    • evenwichtOPM
      2·
      4 days ago

      I did not think that curl could interpret HTML and fetch all elements needed to render a webpage. What would be a sample commandline for curl?

      • graycube@lemmy.world
        1·
        4 days ago

        While -o would simply the url to a file, it does not grab all the resources for the page. You would probably need to write a recursive script that parses the page to fetch dependencies. You might be able to use the lynx web browser. I think chromium also has a command line interface, but it may create a virtual head to render the page before dumping it to a file.