• 1 Post
  • 101 Comments
Joined 1 year ago
cake
Cake day: July 26th, 2023

help-circle







  • You were never into video games, right? The reason I ask, is because games use a lot of AI. One might see “AI” in the game settings, or if the game has some editing tool/level builder/ … one might see it there. If one takes an interest, one might pick up on people talking about the AI of one game or another.

    I am always surprised, when I hear people say that LLMs are too simple to be real AI, because I’m thinking that most people who grew up in the last ~20 years would have interacted a lot with these much simpler game AIs. I would have thought that this knowledge would diffuse to parents and peers.

    Non-rhetorical question: Any idea why that didn’t happen?




  • It’s not the definition in the paper. Here is the context:

    The idea of emergence was popularized by Nobel Prize-winning physicist P.W. Anderson’s “More Is Different”, which argues that as the complexity of a system increases, new properties may materialize that cannot be predicted even from a precise quantitative understanding of the system’s microscopic details.

    What this means is, that we cannot, for example, predict chemistry from physics. Physics studies how atoms interact, which yields important insights for chemistry, but physics cannot be used to predict, say, the table of elements. Each level has its own laws, which must be derived empirically.

    LLMs obviously show emergence. Knowing the mathematical, technological, and algorithmic foundation, tells you little about how to use (prompt, train, …) an AI model. Just like knowing cell biology will not help you interact with people, even if they are only colonies of cells working together.

    The paper talks specifically about “emergent abilities of LLMs”:

    The term “emergent abilities of LLMs” was recently and crisply defined as “abilities that are not present in smaller-scale models but are present in large-scale models; thus they cannot be predicted by simply extrapolating the performance improvements on smaller-scale models”

    The authors further clarify:

    In this paper, […] we specifically mean sharp and unpredictable changes in model outputs as a function of model scale on specific tasks.

    Bigger models perform better. An increase in the number of parameters correlates to an increase in the performance on tests. It had been alleged, that some abilities appear suddenly, for no apparent reason. These “emergent abilities of LLMs” are a very specific kind of emergence.





  • It’s noteworthy that patent law is 20 years to this day. It has survived with its core fairly intact, the main change being that you can no longer get a patent for bringing an invention into the country. Today that is called piracy (poor China).

    I believe that is because patents simply have to work for the whole country in encouraging progress. If cultural production is stifled, well… Who cares? The elites in the copyright industry benefit, and they have an outsize influence on public discourse.


  • This touches several difficult topics.

    I think my disagreement with you about AI copyright infringement is that you think that AI can create new things whereas I don’t think that.

    I don’t think that matters to copyright law, as it exists.

    Copyright law is all about substantial similarity in copyrightable elements. All portraits are similar by virtue of being portraits. Portraits are not copyrighted, nor can one copyright genres and such. A translation of a text has superficially no similarity with the original, but has to be authorized.

    What you are saying would mean, that similarity is no longer a requirement for an infringement. That’s a big change. It is copyright, after all.

    Furthermore it really wouldn’t take a huge change to copyright law, just clear differences between the rules that apply to sentient vs non-sentient sources.

    Non-sentient sources are not new. Take cameras, for example. Cameras have been improved over time so that less skill is necessary to operate one. It’s no longer necessary to manually focus, to set the exposure time, to develop the film, … This also means that photos today have less human creative input. In current smartphone cameras, neural AIs make many decisions and also “photoshop” the result.

    It doesn’t really make sense to me to treat modern cameras differently to old ones. Or: Someone poses and renders a figure in Blender. What difference does it make if they use an old-fashioned physical based render or a genAI?


    Nevertheless, the question whether AIs can create something new, can be answered. The formal definition of “information” is that it is a reduction in uncertainty. For example, take the sequence of letters: “creativit_”. You probably have a very clear idea what the last, missing letter is. So learning that it is “y” doesn’t give you much information.

    But take the sequence: “juubfpvoi_”. The missing letter could be any lower-case letter. You may not feel very informed when you learn that it is “f”, but it does represent a much bigger reduction in uncertainty.

    When we write texts, we use the same old words in the dictionary; just a few 10,000 at most. We string them together with the same old rules of grammar to tell the same old things. The sky is blue, things fall down, not up; people love and hate, and in the end the good guys win. You can probably think of exceptions to all these. They are exceptions. We create small variations on the same old themes. We rehash.

    If a story does not cater to expectations, then it’s not believable. People should behave as we know people to behave. The laws of nature should be consistent and familiar. Most of all: The conventions of the genre should be followed. As a human, you are supposed to lift ideas from previous works. New ideas may be appreciated, but are not required.

    The second string was, in fact, created by a machine; not an AI, but an RNG. Even with many GBs of output, it should be impossible to find any biases or patterns that allow one to guess at the next letter. I didn’t make one up myself because humans are not very random even when we try. And when we write, we do our best to reduce our randomness even further. We try not to invent new spellings; ie make spelling errors.

    AIs receive input from a pRNG, which means that they create new things. What they are supposed to do is to strip away all that novel information and create something largely predictable. They often fail and, say, create images of humans with an innovative number of fingers. LLMs make continuity errors, or straight start to spout gibberish. The problem is that AIs create too many new things, not that they don’t.