s in the past hour

Since Saturday lemmy.ml and lemmy.world were not sending content correctly. The problem seems cleared up now.

Query speed is Lemmy’s main performance bottleneck, so we really appreciate any help database experts can provide.

I have been pleading that Lemmy server operators install pg_stat_statements extension and share metrics from PostgreSQL. https://lemmy.ml/post/1361757 - a restart of PostgreSQL server is required for the extension to be installed. I suggest this be part of 0.18 upgrade. Thank you.

A reminder to move to smaller instances for a better experience

A reminder that this constant advice people blindly parrot to install and flock to smaller instance has now created something like 1000 new servers in 50 days that are poorly run and already going offline as quickly as they went online.

Github Issue 2910 is the kind of PostgreSQL problems that the developers ignored for months and people still defend the developer choices to have the code doing real-time counting of every single comment and post for numbers nobody needs to needs done in real-time.

PostgreSQL is voodoo to this project, they do everything they can to avoid going to !postgresql@lemmy.ml community and asking for help, learning 101 about how to fix their SQL TRIGGER logic like Github Issue 2910 spelled out June 4.

Right now it looks to me like Lemmy is built all around live real-time data queries of the SQL database. This may work when there are 100 postings a day and an active posting gets 80 comments... but it likely doesn't scale very well. You tend to have to evolve to a queue system where things like comments and votes are merged into the main database in more of a batch process (Reddit does this, you will see on their status page that comments and votes have different uptime tracking than the main website).

On the output side, it seems ideal to have all data live and up to the very instant, but it can fall over under load surges (which may be a popular topic, not just an influx from the decline of Twitter or Reddit). To scale, you tend to have to make some compromises and reuse output. Some kind of intermediate layer such as every 10 seconds only regenerate the output page if there has been a new write (vote or comment change).

don’t necessarily see the complexity (not saying it’s not there

It's the lack of complexity that's kind of the problem. Doing direct SQL queries gets you the latest data, but it becomes a big bottleneck. Again, what might have seemed to work fine when there were only 5000 postings and 100,000 total comments in the database can start to seriously fall over when you have reached the point of accumulating 1000 times that.

I don't think people know (how end-users will cope with the distributed choices of Lemmy). Reddit 2023 is nothing at all like Lemmy. One could be considered a household name for regular users of the Internet, the other a return to something more like FidoNet.

I come from the BBS days of the early 1980's and even social media radio before that. I come from IRL user group meetings, held at public library and after-hours company meeting rooms. It has always bothered me that current-day subreddits have mostly no identity to the moderators and that moderation is often behind the scenes.

I guess it's like "corporate experience" that people expect this day in society... that you can walk into a generic franchise chain bar and grill and not really care who the owner/operator and bouncers are of your hangout. Anyone can start a topic/ conversation and there is just some anonymous janitorial crew who is supposed to clean up the overflowing mess if (non-venue) spam or hate messages enter into the space.

The mechanisms of who pays for the venue and the moderators also was a topic most people never bothered to think about. Like it was some taxpayer-funded city park and perhaps the admin police might spot check if anyone was causing a tragedy in that there commons. But reality is that it was a profit-seeking venue charging a cover charge in the form of selling copies of your contribution and changing the tone of your meeting space by controlling the jukebox that visitors hear in terms of advertising messages inserted into the conversation space.

Lemmy seems small, owner/operator focused, and you get a sense that each instance is like some small bar and grill where you can come and meet some strangers or friends to discuss some topics under house rules. Your tips help pay for the hosting and the jukebox isn't piped in memes from advertisers.

I remember when Reddit had known owners with known ideals, but that was very long ago. I've found making it big (with the associated wealth) changes people. One owner even committed suicide over his society ideals about sharing information. Ultimately I feel like a lack of topic participation by the moderators and owners alike made people thoughtless as to their own role in building a human community and people often felt like they were fighting machines and code.

sorry if this meandered off topic, but lately I've had some long-time friends ask me 'what is Reddit" since it is in the news lately, and I find it hard to explain what Reddit used to be (before new Reddit and the addition of images/video) vs. the corporate-like entity we know today that our contributions and participation helped empower over the past 17 years. I've used it mostly daily for all that time, and I have been unhappy with society's dehumanizing direction for too many years.

/ramble from a disturbed mind.

Most common cause is people changing their language settings in their profile. It's a daily occurrence. The app really needs to tell people "25 messages not displayed because you are only viewing in Spanish".

Based on looking at the code and the relatively small size of the data, I think there may be fundamental scaling issues with the site architecture. Software development may be far more critical than hardware at this point.

It’s not on every comment,

My testing with latest code is that it is indeed on every single comment INSERT, every new comment. I have the ability to view my live data while using Lemmy: https://lemmyadmin.bulletintree.com/...

Every one of the 1486 rows on that table gets +1 on comment when I post a new comment on my instance.

it’s mostly triggered on deletions and edits

That is not correct. Edits do not change the count of comments column on site_aggregates - because the number isn't changing. Deletes in Lemmy are also not SQL DELETE statements, they are just a delete column in the table, so that DELETE PostgreSQL trigger only gets run when a end-user cancels their Lemmy account in their profile.

The project had gone on for 4 years without a lot of testing... old code like login form had all kinds of problems, etc. Lemmy-ui had almost no ability to cope with errors from the backend, and often error messages didn't even exist for the API calls. There was a huge rush to fix so many areas that were outright not working.

Great, thank you. The network is much more stable and working solid!

There is no built-in “real-time” methods for admins via the UI to identify suspicious activity from their users, I am only able to fetch this data directly from the database. I don’t think it is even exposed through the rest api.

The people doing the development seem to have zero concern that their all the major servers are crashing with nginx 500 errors on their front page under routine moderate loads, nothing close to a major website. There is no concern to alert operators of internal federation failures, etc.

I am only able to fetch this data directly from the database.

I too had to resort to this, and published an open source tool - primitive and non-elegant, to try and get something out there for server operators: !lemmy_helper@lemmy.ml

That feature you linked to is to flair users.... there is a different issue to flair posts: https://github.com/LemmyNet/lemmy/issues/317

Welcome to the Lemmy network.

There is a pretty steep learning curve to this, and there have been a lot of little tweaks to the install that it's hard to know for certain what is causing your problems.

Lemmy does not bring in messages automatically. You have to subscribe to each community from one user on your server. I see some of your listed communities have zero subscribers: https://globe.pub/communities?listingType=All&page=1

You also have many that do list 1 subscriber, did it ever go past "pending" for that one user to "joined" for the community?

Are you running on Ubuntu 22.04 server? How did you do the install?

On the technical topic of renaming a domain of a Lemmy server... I think it is worth experimenting with the code. At minimum, I think it should be an option to try and keep the same login/passwords for users from the old install of Lemmy. But even that could prove tricky if a particular domain changed underllying ownership more than once - and user@domain became rewritten by an entirely different person. I guess in the real-world people do often get mail for previous residence of a house.

My biggest concern is legality because Lemmy claims to support privacy. I honestly think it's a bad idea to claim privacy because you run into so many problems. If the user never knows that their lemmy instance changed names and can't find it again, etc. Especially on technical topics, 15+ years of having Reddit keep messages from deleted user accounts offered a lot of great search engine hits. With Lemmy, a person moving to a different instance and deleting their account, so much content is going to get black-hole in favor of 50 instances having copies of a meme post or trivial website link - and solid original content (often in comment discussions) gets removed.

And I would like to see a federation-wide policy that all bots must be clearly identified as bots (an attribute on their account). And features in the site code to block all bots as a user preference.

Lemmy federation, server to server, is having a number of problems. Comments and postings aren't always getting copied over to all the servers who have subscribers. For over a week, Lemmy.ml has had regular nginx 500 error codes on the front page, sometimes you have to refresh 3 or 4 times to get a page to load.

Growing pains.

When subscribing to communities it occasionally says “Subscribe Pending”

Your specific issue is documented and I opened a fresh issue on it today: https://github.com/LemmyNet/lemmy/issues/3203

Why is there a lack of gifs/videos on Lemmy?

Lemmy's internal data performance is so horribly slow and crash-causing that I think the last thing they want is even more popular data.

Video is simply the most superior type of media there is, and I think that not having easy access to it on Lemmy is hurting it.

Video is more data, popularity is more data. For whatever reason, at every turn, I've seen developers turn away from scaling options like Memcache, Redis, or just abandoning ORM data management and rewriting the data interfaces by hand....

since the sites on which the videos are hosted can track you.

That's already true for images that are hot linked routinely, so I don't think video really changes it.

I've been baffled since June why data and fixing lemmy's data coding hasn't been front and center. It's pretty wild to witness so many come to Lemmy and then turn away... Elon Musk has been flocking people, Reddit, etc. It's as if the project wants to make code that won't work on any data. It's baffeling.

Up until early July, Lemmy was damned if you do, damned if you don't. Federation had massive performance overhead due to some bugs and each additional instance that went online and subscribed to the big 4 popular servers was causing an even worse load problem than if say 30 users had joined directly. Especially instances that wanted a fully populated All listing, that meant every single thing was being sent to the server even if nobody was really reading that stuff.

And things like searching for topic content are going to be pretty limited given these newer servers don't have much history.

The aftermath of this attempt to scale is that there is also likely a lot of duplicate data, conversations that are mostly repetitive and posts to the same topics. Let alone the bugs Lemmy has federating deletes and moderation removal that doesn't impact direct users on the main servers as much.

This basically shuts my idea down

it's not very difficult to modify the code for something like this.... and closing off registration wont' let anyone else login and create new content form your istance.

Personally the load on the major servers by having one more instance that subscribes to everything is why I think people should back off from creating more than the 1500 instances Lemmy network already has. Delivery of every single vote, comment, post 24 hours a day just so one person can read content for an hour or two a day.

That makes sense for email systems where all that content doesn't have to be sent, but for Lemmy it's a huge amount of overhead.