The deers of Nara show that giving them food and protecting them is an easy way to achieve that.

I had never seen deers as aggressive as monkeys towards humans!

Does it mean that production reached the level where intermittence becomes problematic?

It is llama3-8B so it is not out of question but I am not sure how much memory you would need to really go to 1M context window. They use ring attention to achieve high context window, which I am unfamiliar with but that seems to lower greatly the memory requirements.

To actually read how they did it, here is there model page: https://huggingface.co/gradientai/Llama-3-8B-Instruct-Gradient-1048k

Approach:

• meta-llama/Meta-Llama-3-8B-Instruct as the base
• NTK-aware interpolation [1] to initialize an optimal schedule for RoPE theta, followed by empirical RoPE theta optimization
• Progressive training on increasing context lengths, similar to Large World Model [2] (See details below)

Infra

We build on top of the EasyContext Blockwise RingAttention library [3] to scalably and efficiently train on contexts up to 1048k tokens on Crusoe Energy high performance L40S cluster.

Notably, we layered parallelism on top of Ring Attention with a custom network topology to better leverage large GPU clusters in the face of network bottlenecks from passing many KV blocks between devices. This gave us a 33x speedup in model training (compare 524k and 1048k to 65k and 262k in the table below).

Data

For training data, we generate long contexts by augmenting SlimPajama. We also fine-tune on a chat dataset based on UltraChat [4], following a similar recipe for data augmentation to [2].

I know. But we know it is “just” an engineering problem which can be solved at a high cost.

Fusion is a field where you can’t have the “statup mindset”: investments are in hundreds of millions and take at best a decade (and most likely two) to pay off. That’s one field where it can’t go anywhere without public funding.

It is very possible that China gets there first, considering how ridiculous western fusion efforts have been.

“Theft” is actually legal. Sharing (what they call “piracy”) is not. How about getting the fucking copyright reform that we should have done two decades ago?

It would probably be more effective to put an explicit mention in the system prompt. “Your interlocutor is a <gendered term> and will be greatly offended to be refered to as a boy or a man.”

Let me guess: open source?

That’s according to a peer-reviewed study funded by the Ford Motor Company, a company that makes most of its profits from gas-powered vehicles.

If you want to see if a tech is part of a renewable future, it is direct emissions that should be counted. EVs are at zero. They don’t emit CO2 when running, when being produced or when being disposed of. They use electricity and transport, two things that we can provide without emitting CO2. They are a piece of the puzzle of a sustainable society, something thermal cars will never be, and something these graphs hide.

Of course we will be better off without cars and trucks, but the road towards them being totally gone is long, and it is time we don’t have.

OpenAI should be fine. They are leaders but there are plenty of competitors.

Microsoft is in a much more dominant situation and will have to argue that Google competes with them, which is true but may be hard to sell given the fact that I dont think Google offers its TPU services to any other company.

NVidia is in a situation of monopoly. For them it will be hard to argue otherwise. AMD is simply not there, no one using it.

And this is why research is going in another direction: smaller models which allow easier experiments.

I am pretty sure that there are ASIC being put in production as we speak with Whisper embeded. Expect a 4 dollars chip to add voice recognition and a basic LLM to any appliance.

Also, as a side effect, we just solve speech recognition. In a year or two, speaking to machines will be the default interface.

Your assumptions are far more numerous and offensive than that. From you thinking that I know nothing about discrimination at work or my driving habits, or even assuming that you are more to the left than I am or that I criticize your positions for being leftist rather than being wrong.

The cherry on the top of you laying down a dozen of wrong accusation is you calling my attitude patronizing and belittling.

There is a company-wide demotivation plague at Google. Don’t blame middle manager, it extends to the top.

it unusual for someone to get things this wrong this consistently.

At least we agree on something.

And once again your assumptions about my situation and my work ethics are hilariously wrong. I am cutting down my income in order to work non-profit on issues I do care about and turn down offers by unethical companies routinely. I am a freelance who changes client pretty often. My income does not depend on the acceptance of an ideology, I made sure of that and that was a reason for becoming independent.

I am sure I am not the first person you are antagonizing through your own projections. You should really be more careful about assuming things about the people who contradict you. Sometimes they just do it because you are wrong. Being more open to that possibility would make your life much better.

I use it almost daily.

It does produce good code. It does not reliably produce good code. I am a programmer, it makes my job 10x faster and I just have to fix a few bugs in the code it usually generates. Over time, I learned what it is good at (UI code, converting things, boilerplate) and what it struggles with (anything involving newer tech, algorithmic understanding, etc.)

I often refer to it as my intern: It acts like an academically trained, not particularly competent, but very motivated, fast typing intern.

But then I am also working on the field. Prompting it correctly is too often dismissed as a skill (I used to dismiss it too). It needs more understanding than people give it credit for.

I think that like many IT tech it will go from being a dev tool to everyday tool gradually.

All the pieces of the puzzle to be able to control a computer by voice using only natural language are there. You don’t realize how big it is. Companies haven’t assembled it yet because it is actually harder to monetize on it than code it. I think probably Apple is in the best position for it. Microsoft is going to attempt and will fail like usual and Google will probably put a half-assed attempt at it. I’ll personally go for the open source version of it.

First time I have someone complaining about me giving sources.

You are mostly arguing on things we agree (there needs to be policy efforts, there needs to be some change, the current transition is too slow) and we mostly disagree on what is a data-backed observation: renewables augment while fossils go down. Within fossils, gas displaces coal but fossils in general go down. If you refuse graphs and numbers about good sources about it, I am at a loss.

Damn I want to read it but it is from the only two accounts I muted (for different reasons)

EDIT: God the sewer when you unblock Musk’s account! I am never doing it again. Why do people talk over this stupidly noisy channel instead of having a threaded discussion like civilized great apes?