Hey julian,

thank you for your reply. I should have probably been more verbose, my bad. I cannot call .deleteAccount directly, at least I would not know how to. Because I am using a Python script on my local machine which has remote read/write access to the database on which the forum runs. I also have a dump of the forum and its database with which I can play around.

My initial idea was just to quickly shut down the forum, let my script run on the DB (which just takes seconds to run, even when deleting thousands of users and doing queries in between for related documents). But that leaves me with a functional but still malformed NodeBB instance. I then looked at the NodeBB Write API. I only got the DELETE /api/v3/users/{uid} route to work yesterday. Which is very slow. I let it ran over night on the dummy forum, and it only chewed through half of the users over night (because the estimate is 33 hours of runtime seems correct). I will have a look at DELETE /api/v3/users again today, i.e., the bulk deletion route. When I tried to use it the first time, NodeBB rejected my payload with the IDs to delete (I already split up things into chunks of 100 user ids). But that was probably my fault somewhere.

So, the information I am seeking is:

1. Are there any deeper insights into the database structure of NodeBB? I am already aware of NodeBB: Database Structure, but it does not seem to explain why my forum.net/users shows thousands of empty pages when I just manually delete the users.
2. The problem we are having does not strike me as unique, given how popular NodeBB is among spammers. But when I google for 'nodebb bulk delete users', the only relevant hit I find is this ten years old posting with no real answer (and now also my own posting here as the second hit). Are there any known workflows/plugins for this? I have either a list of UIDs or a set of deletion criteria, and want to bulk delete these users.

I could probably write a NodeBB plugin, so that I can call .deleteAccount, but that would be a whole lot more work than I planned to invest.

Cheers, zipit

The user's page loads uids from various sorted sets based on the sort option slected. So you need to remove the deleted uids/usernames from these sorted sets. https://github.com/...

I think I took these into account:

 def DeleteUser_Deprecated(self, user: dict) -> None:
    """Deletes the given #user and all its related documents from the database.

    .. warning::
    
        This method does not work properly. Use `DeleteUsers` instead. Even when we follow what
        is documented in the NodeBB database structure documentation, we will end up with a
        deletion anomaly. The NodeBB instance will be somewhat functional, but effectively 
        malformed. There is probably one of these 'user sorted sets' missing, but I was not in
        the mood to read the code base to find out which one it is.

    """
    raise DeprecationWarning("Use DeleteUsers() instead.")

    if not isinstance(user, dict):
        raise TypeError("#user must be a dict.")
    
    # Could probably be optimized into a single query, but this more readable and fast enough.
    uid: int = int(user["_key"][len("user:"):]) #: The user ID to delete.
    docs: list[dict] = [user] #: The document to delete.

    # E.g., user:1234:usernames, user:1234:emails, etc.
    docs += list(self.Find({"_key": { "$regex": f"^user:{uid}:"}}))
    # E.g., uid:1234:posts,  uid:1234:bans:timestamp, etc. Most of them should be irrelevant, 
    # for us since our users all have no posts.
    docs += list(self.Find({"_key": { "$regex": f"^uid:{uid}:"}}))
    # All the cases where the key is static and UID is mentioned as in some field in the document.
    for key, field in (
        ("users:banned", "value"),
        ("users:banned:expire", "value"),
        ("users:flags", "value"),
        ("users:joindate", "value"),
        ("users:notvalidated", "value"),
        ("users:online", "value"),
        ("users:postcount", "value"),
        ("users:reputation", "value"),
        ("users:uid", "value"),
        ("username:uid", "score"),
        ("userslug:uid", "score"),
        ("email:uid", "score"),
        ("users:website", "value"),
        ("users:aboutme", "value"),
        ):
        docs += list(self.Find({"_key": key, field: uid}))

    # Some special cases.
    docs += list(self.Find({"_key": "email:sorted", "value": {"$regex": f":{uid}$"}}))
    docs += list(self.Find({"_key": "username:sorted", "value": {"$regex": f":{uid}$"}}))
    docs += list(self.Find({"_key": {f"$cid:\\d+:uid:{uid}"}}))
    docs += list(self.Find({"_key": { "$regex": f"^tid:\\d+:posters$"}, "value": uid}))

    # Gather all IDs of the documents and bulk delete them.
    ids: list[bson.ObjectId] = [doc["_id"] for doc in docs if "_id" in doc]
    result: pymongo.results.DeleteResult = self.Objects.delete_many({"_id": { "$in": ids}})
    if result.deleted_count != len(ids):
        print(f"Failed to delete {len(ids) - result.deleted_count} documents. "
                f"Deleted count: {result.deleted_count}")

But as you can see, I deprecated this method, because at least for me it did leave me with a malformed instance. I have worked today towards the bulk deletion endpoint of the WriteAPI, and got something that deletes users in chunks of 100 users at a somewhat acceptable speed.

Will post my code here once I am done, in case someone faces this problem too in the future. Thanks for the help so far!

Cheers, zipit

zipit if the accounts have no actual content you can just call .deleteAccount as that's more lightweight.

The reason why user deletion takes so long is because of all those cross referenced sets. There are probably opportunities for optimization there.