• Brkdncr@lemmy.world
    link
    fedilink
    arrow-up
    3
    ·
    13 hours ago

    Enterprise is different. Lots of business decision makers are prepping their workforce for AI, and do t want to put their data on someone’s cloud. Local AI will be a big deal.

    • Alphane Moon@lemmy.worldOPM
      link
      fedilink
      English
      arrow-up
      3
      ·
      13 hours ago

      Are the current crop of NPUs really suitable for this?

      I play around with video upscaling and local LLMs. I have a 3080 which is supposed to be 238 TOPS. It takes about 25 min to upscale a ~5 min SD video to HD (sometime longer depending on the source content). The “AI PC” NPUs are rated at around ~50 TOPs, so that would be a massive increase in upscale time (closer to 2 hours for ~5 min SD source).

      I also have a local LLM that I’ve been comparing against ChatGPT. For my limited use case (elaborate spelling/typo/style checking), the local LLM (llama) works comparable to ChatGPT, but I run it on a 3080. Is this true for local LLMs that run on NPUs? I would speculate that more complex use cases (programming support?), you would need even more throughput from your NPU.

      I have much more experience with upscaling though and my experiments/usage of local LLMs is somewhat limited compared to ChatGPT usage.