To be clear, $3k is an accurate, but unacceptable amount.

As in that's what it's actually costing us, but it's not what it should be costing. I'd imagine more like $250 is what we should be paying if I wasn't using AWS in the silly way I am.

I'm admitting up front that I've been more focused on developing rather than optimising operating costs because I could afford to be a little frivolous with the cost in exchange for not having to worry about doing server stuff.

Even when the Reddit thing happened I was wilfully ignoring it, trying to solve the scaling issues instead of focusing on the increased costs.

And so I didn't notice when Lemmy was pushing a terabyte of data out of the ELB a day. And that's what got me.

About half that $3k is just data transfer costs.

Anyhow the notice was just to let our users know what is going on and that there'll be some maintenance windows in their future so it doesn't surprise anyone.

We have a plan and it will all work out.

Don't panic or have any kneejerk reactions, it's just an FYI.

I have been watching my love tie herself in knots over the last several days, having to deal with the drama that has been brought on, trying her best to bring everyone back together.

There's been bad behaviour from both sides, and I'm really disappointed to see that some of the worst of it came from our users, who didn't keep to the moral high ground, disregarded our instance rules and stoopped to levels of behaviour worse than that leveled against them.

There have been accusations against us (or Ada specifically) that we are a safe harbour for bad behaviour and cause harm to trans people through our inaction.

This is perhaps the cruelest accusation they could have leveled at Ada, as she works tirelessly to maintain a safe space for our community, and while I was hoping, for all the effort that she was investing into this issue, that she could make it work despite my own reservations, this last attack on her impeccable morality has made me very angry.

I'm sorry for those that wanted to remain federated, sorry that it came to this, but I am glad it's over now, purely for the mental health of my precious beloved.

Just want to say, I don't blame anyone else but myself.

I certainly don't blame anyone at 196.

I hope I'm really clear about that. It's one of the reasons I specifically didn't name 196 in my announcement.

We've got a solution planned, we've already started to implement it and have the image transfer issue solved already.

We can afford to cover this ridiculous AWS bill, I just need to do some maintenance work so this doesn't continue because I can't continue to line Jeff Bezos' pockets like this indefinitely.

So, one thing I'd mention is the systems and admin work involved in running an instance.

This is on top of the community moderation, and involves networking with other instance admins, maintaining good relations, deciding who to defeferate from, dealing with unhappy users, etc.

Then there's the setup and maintenance of the servers, security, hacks, DDoSing, backups, redundancy, monitoring, downtime, diagnosis, fixing performance issues, patching, coding, upgrades etc.

I wouldn't be here doing this without @ada. We make a formidable team, and without any self effacement, we are both at the top of our respective roles with decades of experience.

Big communities also magnify the amount of work involved. We're almost at the point where we are starting to consider getting additional people involved.

Moreover we're both here for the long haul, with the willingness and ability to personally cover the shortfall in hosting costs.

I'm not trying to convince you to stay here. But in addition to free hardware, you're going to need a small staff to do these things for you, so my advice is to work out if you have reliable AND trustworthy people (because these people will have access to confidential user data) who are committed to do this work long term with you. Where will you be in 3 years, 5, 10?

How do you know I haven't always been the hacker who's in control? :D

Migration has been completed!

It wasn't an actual emojo. The script processed the SQL header column names as an emojo and tried to add them. Unfortunately publicUrl is not a valid URL, so lemmy's /api/v3/site metadata endpoint started returning an error relative URL without a base instead of the JSON that the website was expecting and so the site just stopped working for everyone the next time it tried to load that url.

Migration complete.

The pict-rs upgrade is ongoing.

From what I can tell it'll be about another 5 hours. I'm going to have to go to bed and check on it in the morning.

Unfortunately the stock-standard lemmy-ui doesn't like it that pict-rs is migrating to a new version of database and not serving images, so it's stubbornly just not working at all.

Okay, so that was way more painful than expected... /sigh

Our best haj, Shonky (they/them) is available now, over at their own Github repository for use when referring to Blåhaj Lemmy.

I'm guessing we'll need them to make an appearance for the Canvas template.

Okay, I'm taking down the server now to upsize thew instance hardware.

Server hardware is now upsized. I will be upgrading and rebuilding the 0.18.1-rc.4 lemmy now. The database hardware upsize will be in about 19 minutes time.

You are super welcome, lovely.

It brings Ada and I a whole heap of pleasure running these instances and it's largely knowing that we're making a difference to our users, and providing a safe space for you all to grow and flourish that makes it all worth it for us.

Database upsize completed. Still working on 0.18.1 upgrade, will be one further (software) restart in about an hour and then we'll be done and running the latest release candidate.

Apparently I was wrong, it's decided it's done and working again now.

All fixed!

The maintenance has completed and everything is now running from our new database.

Thanks for your patience while we performed this migration.

@ada@lemmy.blahaj.zone does a fantastic job of moderating, and it constantly amazes me that she manages to keep her cool (and calms me down when I get riled up too!).

Thanks for being the moderator to my sysadmin, babe. I couldn't do it without you.

https://mlmym.lemmy.blahaj.zone