> One can envision a world in which OpenAI pays chefs money to cook while ChatGPT watches—narrating their thought process, tasting the dishes, and describing the results. This information could be used for general-purpose training, but it might also be packaged as a “book”, “course”, or “partner” someone could ask for.

So we're speed running the idea of AI Facebook friends and creating a new para(ai)social relationship

whatever happened to the system prompt buffer? why did it not work out?

because it's a separate context window, it makes the model bigger, that space is not accessible to the "user". And the "language understanding" basically had to be done twice because it's a separate input to the transformer so you can't just toss a pile of text in there and say "figure it out".

so we are currently in the era of one giant context window.

Also it's not solving the problem at hand, which is that we need a separate "user" and "data" context.

Reminder that Anthropic's goal is to sell you more tokens...

Yes, exactly. That's why their harness has no incentive to help you save tokens. Just conflict of interest.

llama4 behemoth problems?

We pay people to create more high quality tokens (mercor, turing) which are then fed into data generating processes (synthetic data) to create even more tokens to train on

But does that really help, or do you get distortion? The frequency distribution of human generated content moves slowly over time as new subjects are discussed. What frequency distribution do those “data generating processes” use? And at root, aren’t those “data generating processes” basically just another LLM (I.e., generating tokens according to a probability distribution)? Thus, aren’t we just sort of feeding AI slop into the next training run and humoring ourselves by renaming the slop as “synthetic data?” Not trying to be argumentative. I’m far from being an AI expert, so maybe I’m missing it. Feel free to explain why I’m wrong.

That's the problem in a nutshell. There is an art to how you generate the synthdata so that you don't get crappy trained models (especially when mistakes cost XX million dollars).

It's also theoretically why facebook paid 14bn for alex wang and scale ai

Anyone done vibe testing at meta ai yet?

How are you different from all the other quant funds?

there is no human in the loop, there is no high frequency trading. We're trying to have AI mimic what fund managers do:

- lots of research - longer time horizons - zero humans in the loop, but explain every single thing you do.

what kind of tps slowdown would you realistically on an npu vs gpu?

Microsoft requires a 40 TOPS NPU for Copilot co-branding, which a RTX 3050 can beat.

fine tune an oss model and call it a groundbreaking innovation -- 20 points