How many proper open-source LLMs are there?

2 months ago by hendrik to c/localllama

Sometimes I get into arguments about AI and people comment weird stuff here on Lemmy. Like we have ethical AI models (meaning they're not made up of unethically sourced training material). Or we got plenty "libre" or "free and open-source" AI models.

Am I missing something? Or is this just a bunch of people trying to bullshit me? Is anyone here aware of such large language models and would be so kind as to educate me? I mean we do have a ton of open-weights models. But proper open-source or ethical??? I've been following the news for quite a while and I'm aware of a very select few I'd call LLMs with training datasets available. I'm aware of:

Pythia by EleutherAI (up to 12B, from 2023) and Dolly 2 by Databricks (12B, also from 2023)
BLOOM by BigScience (176B, from 2022)
GPT-2 by OpenAI (1.5B, from 2019)
The recreation of LLaMA by the RedPajama-Project
Apertus by the ETH Zürich

Edit: In the comments we got:

LLM360 with K2-65B, Crystal-7b, Amber-7B
the Soofi-Project
OLMoE

That list needs a lot of footnotes, though. Some of them come with restrictions, terms and conditions... And none of them can do modern tasks like AI-assisted coding. The only one getting close is the Swiss model. Sadly the earlier version I tried didn't perform well. Seems they released an updated v1.5 version this summer, I wonder if that's better and anybody uses it for real-world applications.

So... Back to my initial question: Are there models out there with datasets available? I mean except the exactly one somewhat usable one I already found?

keepthepace 16 points 2 months ago

LLM 360 probably deserves a mention there, trying to be open, reproducible and publishing intermediate training checkpoints.

meaning they’re not made up of unethically sourced training material

I am old enough to remember the days before 'piracy' was considered unethical. I still stand by that. Authors need to be compensated but also "publication" is supposed to give some rights to the "public". Sharing non-commercially things you love over internet used to be a gray zone that should have been the subject of a political debate. Instead, right holders decided that the letter of the law should strictly apply to conditions it was never designed for. I have little empathy for them moaning now that AI training, while not formally illegal, should be considered unethical.

A copyright reform is long overdue. Let's discuss it seriously instead of playing the game of "who can lobby the most?". In the meantime, enjoy the gray zone that refusing to update 19th century laws to the century of internet causes.

path: 0 25600136, hotness: undefined, score: 16, children: 4

Rhaedas 6 points 2 months ago

I was trying to reply to this, bringing up the controversy that erupted when image generators were able to use prompts with artist names to simulate their style, and how that shows how blurry the copyright line is. As I wrote it, I realized how complicated the issue really is, and how little I understand current copyright laws. Some may argue there isn't any vagueness, an artist's work in any format is theirs and shouldn't be copied for other use, training or otherwise. But we all use other people's work in our own, don't we? There is no solid line where one is ethical and one isn't. Plagiarism without proper attribution has always been wrong, but if you take the same content and tweak it a bit, it's not. LLMs can and do that, through statistical randomness, while using someone else's material, just like human brains would.

The compensation issue is definitely a problem and where the ethics come into play. Even if many people freely read a book in a library, it has been bought at some point. But how do you determine who was unfairly "stolen" from? Isn't it basically everyone at this point?

path: 0 25600136 25601770, hotness: undefined, score: 6, children: 2

hendrik 4 points 2 months ago

I know I'm way too nitpicky... But I feel I should point out there's some confusion hidden here...

There's stuff which is moral to do. And stuff which is legal to do. Those aren't the same! When talking about who can mention what, what copyright law demands people to do and those blurry lines, we're concerned with legality. That doesn't have anything to do with ethics. (At least not directly.)

path: 0 25600136 25601770 25603011, hotness: undefined, score: 4, children: 0

keepthepace 2 points a month ago

Compensation is actually the only issue. If they were fairly paid, most artists would be ok with people enjoying their works in any way they can.

path: 0 25600136 25601770 25613371, hotness: undefined, score: 2, children: 0

hendrik 1 point 2 months ago

Thanks! Yes, definitely deserves a mention.

path: 0 25600136 25602914, hotness: undefined, score: 1, children: 0

Eyekaytee 12 points 2 months ago

https://eurollm.io/ ?

Looks like they go over their data sets here:

https://www.sciencedirect.com/...

https://huggingface.co/togethercomputer

Hopefully someone else knows more, I don't really care, I just want Europe to catch up to China/America otherwise it won't matter how ethical the LLM is because everyone will be using Chinese/American anyway like 99% of people already are

path: 0 25599415, hotness: undefined, score: 12, children: 0

hok 9 points 2 months ago

I see Olmo by AllenAI frequently used in research. The publish all the checkpoints of training and all training sources. https://allenai.org/olmo / https://huggingface.co/collections/allenai/olmo-3

Here's the link to the dataset used to train Olmo 3: https://huggingface.co/...

It's fairly modern. Olmo 3.1 was released January 2026.

path: 0 25603909, hotness: undefined, score: 9, children: 1

jerkface 1 point 2 months ago

Awesome, I thought there were none.

path: 0 25603909 25606391, hotness: undefined, score: 1, children: 0

hirihit640 7 points 2 months ago

I recall OpenAI saying to a courtroom judge that modern AI requires training off copyrighted material. And obviously the AI labs cannot just release those.

Distlled models (aka models that train themselves off the responses from other AI models) could probably release all the AI responses they used for training, since afaik AI generated content is not copyrightable..

However I suspect that most modern models are built from previous models. For example Claude Fable is probably based on Claude Sonnet or something. And for a downstream model to be fully "open source" and reproducible, you would need to reproduce the upstream model too. So even if a downstream model didn't use copyrighted data (eg it only used distills), one of the upstream models likely did.

path: 0 25600826, hotness: undefined, score: 7, children: 5

jerkface 4 points 2 months ago

They could still release a manifest of hashed contents and metadata.

path: 0 25600826 25606350, hotness: undefined, score: 4, children: 4

hendrik 1 point 2 months ago

What would be an application for this? More transparency for court cases? Or enable people to search for an alternative source for the data, like a torrent?

path: 0 25600826 25606350 25607222, hotness: undefined, score: 1, children: 3

jerkface 3 points 2 months ago

It would allow users to audit the dataset for ethical concerns, to interrogate the set to understand what a model does and does not "know", to identify weaknesses and biases, and to ensure that nothing has been inserted specifically to engineer prompt responses. Just examples off the top of my head.

path: 0 25600826 25606350 25607222 25607588, hotness: undefined, score: 3, children: 2

hendrik 2 points 2 months ago

Knowledge should be relatively impossible, though, if you just get the names and hashsums of the files? I mean I guess you could look up if a specific file you already have is in it. Might have some application. It needs to be very detailed, though. If they just call it GitHub-dump-timestamp or Wikipedia-dump-2026 with some overall hash, that doesn't tell us anything.

path: 0 25600826 25606350 25607222 25607588 25607710, hotness: undefined, score: 2, children: 1

kata1yst 3 points 2 months ago

https://huggingface.co/...

path: 0 25600782, hotness: undefined, score: 3, children: 1

hendrik 2 points 2 months ago

Uh, that one looks nice. Thanks!

path: 0 25600782 25602880, hotness: undefined, score: 2, children: 0

SuspiciousCarrot78 2 points a month ago

path: 0 25671607, hotness: undefined, score: 2, children: 1

hendrik 1 point a month ago

Looks fine, thanks!

path: 0 25671607 25671990, hotness: undefined, score: 1, children: 0

elucubra 1 point 2 months ago

Many of the Chinese LLMs have open source models, even frontier models.

path: 0 25600474, hotness: undefined, score: 1, children: 2

sukhmel 13 points 2 months ago

Open source or open weights?

path: 0 25600474 25600552, hotness: undefined, score: 13, children: 0

hirihit640 11 points 2 months ago

Please read the whole post. OP is asking for the datasets used to train the models.

path: 0 25600474 25600718, hotness: undefined, score: 11, children: 0

e0qdk 1 point 2 months ago

Nemotron is a fairly significant model family in this space, IIRC. e.g. here's one of the recent releases: https://huggingface.co/...

Edit: Not fully open -- see thread below.

path: 0 25605221, hotness: undefined, score: 1, children: 4

hendrik 4 points 2 months ago

Huh. It doesn't come with all the datasets, though. Does it? The model card contains a long list of "Private Non-publicly Accessible Datasets", both by NVIDIA and by third parties. And some of the stuff just says "Undisclosed".

path: 0 25605221 25605776, hotness: undefined, score: 4, children: 3

e0qdk 2 points 2 months ago

"Private Non-publicly Accessible Datasets"

Huh. I thought they were fully open. I missed that. Most of the data sets and methodology is open though. (Maybe I'm remembering an older model? Or maybe I just fell for their marketing a while back... 😔️)

NVIDIA's own proprietary stuff looks largely like machine translation examples, and some sort of game logs (Sudoku and Minesweeper 🤨️), plus a few things I'm not quite sure I understand ("Multitool Typewriter Hard"? Is that for OCR maybe? Edit: or maybe they mean 'multitool' in an agentic sense... and that's for tool call reasoning?).

The third-party stuff is these six entries:

Global Regulation
TAUS Translation Memory
Scale HLE
HackerRank Coding
RL data for Search
Mercor SWE-AgentsV1

I wonder how hard it would be to rebuild something equivalent that is fully open. 🤔️

path: 0 25605221 25605776 25606177, hotness: undefined, score: 2, children: 2

hendrik 2 points 2 months ago

Sudoku and Minesweeper could very well be some internal project names and have nothing to do with what the words mean. I have no clue.

Seems to me their announcements ooze with marketing speech. I bet they put it all in one sentence to mislead about the datasets. Put a comma there and omitted the word "partly". It's technically borderline correct. I'll have to read up on it, whether that's a good amount of data, or just some tiny fraction. It's not looking good, though.

path: 0 25605221 25605776 25606177 25607191, hotness: undefined, score: 2, children: 1

e0qdk 2 points 2 months ago

It could be, but using the games as logical puzzle solver/reasoning tests might have value -- I don't know.

path: 0 25605221 25605776 25606177 25607191 25607260, hotness: undefined, score: 2, children: 0

TheJesusaurus -27 points 2 months ago

Who cares? Might as well so how many organic grass fed Nazis there are. FOSS doesn't clean off the stink of AI

path: 0 25599769, hotness: undefined, score: -27, children: 4

Eyekaytee 10 points 2 months ago

Your brain on Lemmy

path: 0 25599769 25601137, hotness: undefined, score: 10, children: 0

jerkface 5 points 2 months ago

If the dataset is ethical and can be inspected, and the model is self-hosted, what is your objection

path: 0 25599769 25606382, hotness: undefined, score: 5, children: 2

hendrik 2 points 2 months ago

I don't think this is an easy debate to have. Even those kinds of models have effects on society, on people, on the environment. A lot of resources are needed for research and training. Same for inference even if you own the computer. And we'd need to agree on what's ethical. Usually that means treat the entire world like a self-service outlet and just take everything unless someone calls you and explicitly said no. At that point it'll become a nasty technical problem because there's no easy way to get rid of data after the fact.

Models could in theory be ethically sourced, but used for evil. Which is gonna be a problem to society. And then all of the debate is hypothetical because that's not even what's in use out there...

My post was more concerned with the facts. What kinds of models we do have. Not what we should do or an universally accepted definition of ethical.

path: 0 25599769 25606382 25607596, hotness: undefined, score: 2, children: 1

jerkface 1 point 2 months ago

An inspectable dataset would allow users audit according to their own project's values and obligations. That's the only standard that matters. I don't yet see the need to bring a discussion of universal ethics into this.

Datasets containing unlicensed, stolen, and even privileged information have been used for major commercial models. This is a concrete problem for some people, and open source datasets address directly address it.

You tangentially bring up a bunch of other stuff but don't appear willing to defend a position on any of it. I don't really understand your point. You can speculate about what Jesus's objection might be, but if you're not willing to bring forward your own objections, I'm not sure what you're trying to accomplish at this point.

IMHO, the problem with AI isn't AI, it's that we are permitting states and obscenely wealthy individuals to engage in an extremely wasteful winner-takes-all arms race. IMHO, there is nothing wrong with private individuals exploring LLMs and other new AI technologies, and self-hosted open source models are the only way we get to participate and potentially protect ourselves.

path: 0 25599769 25606382 25607596 25611451, hotness: undefined, score: 1, children: 0

localllama

@sh.itjust.works

login for more options

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

Rules:

Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.

Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.

Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.

Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.

go to feed...

localllama

@sh.itjust.works

login for more options

Welcome to LocalLLaMA! Here we discuss running and developing machine learning models at home. Lets explore cutting edge open source neural network technology together.

Get support from the community! Ask questions, share prompts, discuss benchmarks, get hyped at the latest and greatest model releases! Enjoy talking about our awesome hobby.

As ambassadors of the self-hosting machine learning community, we strive to support each other and share our enthusiasm in a positive constructive way.

Rules:

Rule 1 - No harassment or personal character attacks of community members. I.E no namecalling, no generalizing entire groups of people that make up our community, no baseless personal insults.

Rule 2 - No comparing artificial intelligence/machine learning models to cryptocurrency. I.E no comparing the usefulness of models to that of NFTs, no comparing the resource usage required to train a model is anything close to maintaining a blockchain/ mining for crypto, no implying its just a fad/bubble that will leave people with nothing of value when it burst.

Rule 3 - No comparing artificial intelligence/machine learning to simple text prediction algorithms. I.E statements such as "llms are basically just simple text predictions like what your phone keyboard autocorrect uses, and they're still using the same algorithms since <over 10 years ago>.

Rule 4 - No implying that models are devoid of purpose or potential for enriching peoples lives.

go to feed...