GPT-4's details are leaked.

3 years ago by manitcor to c/chatgpt

@Yampeleg: GPT-4's details are leaked. It is over. Everything is here: twitter.com/i/web/status/1… Parameters count: GPT-4 is more than 10x the size of GPT-3. We believe it has a total of ~1.8 trillion parameters ac...…

cross-posted from: https://lemmy.intai.tech/post/72919

Parameters count:

GPT-4 is more than 10x the size of GPT-3. We believe it has a total of ~1.8 trillion parameters across 120 layers. Mixture Of Experts - Confirmed.

OpenAI was able to keep costs reasonable by utilizing a mixture of experts (MoE) model. They utilizes 16 experts within their model, each is about ~111B parameters for MLP. 2 of these experts are routed to per forward pass.

Related Article: https://lemmy.intai.tech/post/72922

load all comments

unstable_confusion 1 point 3 years ago

Is it just me, or are those links going to the wrong places?

save

path: 0 1115313, hotness: undefined, score: 1, children: 4

manitcor 1 point 3 years ago

They are the right ones. Should be a tweet archive and a blog post

save

path: 0 1115313 1122539, hotness: undefined, score: 1, children: 3

unstable_confusion 2 points 3 years ago

Well that's weird because the first takes me to a shitpost with a picture of cake, and the second a shitpost about sucking your dentist's fingers...

save

path: 0 1115313 1122539 1123813, hotness: undefined, score: 2, children: 2