How to download Lemmy post, as it is shown in the Lemmy UI

2 months ago by da_cow (she/her) to c/webdev

So, what I want is to download lemmy posts (including comments) as raw html through a script. The problem I have is, that curl by default does not include the css of the page. Is there a way I can download the posts webpages as they are being displayed?

DmMacniel 2 points 2 months ago

Mhm you could run a script through the HTML and look for stylesheet hrefs and download next to them.

save

path: 0 25256624, hotness: undefined, score: 2, children: 3

cows_are_underrated 2 points 2 months ago

I was more looking at doing this with a tool like wget. I got the following command, that downloads the post and its assets, but for some reason it then does not actually work:

wget --recursive --no-clobber --page-requisites --html-extension --convert-links --restrict-file-names=windows --domains "feddit.org" --no-parent  "feddit.org/post" "https://feddit.org/post/33942692

The advantage of wget is, that it also accepts multiple links at once, so it is quite easy do download multiple posts at once (which is my goal)

save

path: 0 25256624 25257431, hotness: undefined, score: 2, children: 2

apparia 4 points 2 months ago

If this is what you are seeing that makes it "not work":

It seems to be implemented in a script that runs when the saved page loads and hides the content, even though that content is actually there. You can tell wget to ignore scripts (which aren't very useful in a download anyway, as they mostly pertain to interactivity) with --reject js. I tried this and was able to view the downloaded post properly.

I've no idea why the script does that, one could probably look at the lemmy-ui source code to figure it out.

save

path: 0 25256624 25257431 25258777, hotness: undefined, score: 4, children: 1

cows_are_underrated 2 points 2 months ago

That was my exact problem. Thanks for the solution.

save

path: 0 25256624 25257431 25258777 25260135, hotness: undefined, score: 2, children: 0

refalo 1 point 2 months ago

why html? just saving the json it fetches in the first place would be a lot easier to work with IMO, but it depends on what you're doing with it

save

path: 0 25264023, hotness: undefined, score: 1, children: 4

elvith 1 point 2 months ago

I second that - can't you just use the API and request the post and its comments as json?

Unless you want to explicitly archive the post as a human viewable HTML, the structured json from the API is almost always superior to further analyze, archive,... the posts.

save

path: 0 25264023 25265557, hotness: undefined, score: 1, children: 3

cows_are_underrated 2 points 2 months ago

Archiving thwmin a human readable form is exactly what I want to do.

save

path: 0 25264023 25265557 25271318, hotness: undefined, score: 2, children: 2

elvith 2 points 2 months ago

Then I'm not sure if curl is up to the task in this case - most modern websites are PWAs that dynamically request data from their Backen and display it. Curl just fetches the initial HTML but doesn't execute the JS and thus might not see the content of the page as it isn't there yet the moment it gets saved.

Some PWAs "render" the first page contents in your HTML so that it is included and do not need to be requested.

Maybe try the alternative frontends if some of them work better? Also remember that feddit.org and many other instances deploy Anubis and might lock you out from making simple requests with curl

save

path: 0 25264023 25265557 25271318 25272444, hotness: undefined, score: 2, children: 1

cows_are_underrated 2 points 2 months ago

Got it working with wget

save

path: 0 25264023 25265557 25271318 25272444 25272757, hotness: undefined, score: 2, children: 0

anzo 1 point 2 months ago

Fwiw there's SinglePage extension for web browsers. Maybe you can get a headless way to run that... Which frontend would you save from? I'd say i rather prefer to get json data from backend

save

path: 0 25266045, hotness: undefined, score: 1, children: 1

cows_are_underrated 1 point 2 months ago

I am preferring the standard Lemmy UI, but I already got it working.

save

path: 0 25266045 25271307, hotness: undefined, score: 1, children: 0

Sv443 1 point 2 months ago

Your browser dev tools network tab can be set to filter by CSS files. Then you just copy those URLs, and sort out the unnecessary garbage if there is some (ad frameworks, etc.). Since CSS won't really be changing a lot, you can just download it once and include it statically using your script and some <link rel="stylesheet" href="./path/to/file.css"> tags in the <head>

save

path: 0 25258897, hotness: undefined, score: 1, children: 0

What is web development?

Rules/Guidelines

Related Communities

Wormhole

What is web development?

Rules/Guidelines

Related Communities

Wormhole