As an “infinity monitor” you can use anywhere, it is really great.
But it is not - as it was advertised - a “spatial computer”. I can’t even think of it as an AR device, because it is terrible at image recognition and tracking… something even an iPhone can do. I have no idea why that is the case, because the hardware is, theoretically, ridiculously powerful… but something is seriously, seriously wrong with the software right now… and it cripples the headset for what is supposed to be it’s one major use-case: spatial computing.
P.S. In case it wasn’t obvious, I bought one to develop for, and as a developer I’m pretty angry at how poorly it performs at basic AR tasks.
Yeah. You train it to recognize an object/image, and then you can respond to that however you want, programmatically. It’s really cool for things like tabletop games.
For the how: Apple has APIs to including lightweight ML models for that sort of thing. You have to train a model, but for something like a card game it’s super easy. For physical objects it requires more prep just because you have to take photos instead of using existing artwork… but it’s still relatively straightforward.
I have a card game (a physical one, not virtual) and I want to “replace” the real cards with digital, animated, “living” ones. Ideally, I could apply this technique to other types of tabletop game, later… but cards are the current project.
This works fine on iOS and even in the Vision Pro simulator, but on hardware, the image recognition is slow and unreliable, and it doesn’t track items through space in real-time. It’s laggy and “floaty”. Image recognition for unique, flat cards should be one of the simplest possible use-cases… and given how much more powerful the hardware is than a phone, and the fact that it works on the simulator, it sure seems like a software issue… but you can’t ship Apps with such severe problems, either.
Ouch. Don’t get me started on iPad “Pro”. There is nothing “Pro” about a device that is so heavily sandboxed and restricted to only installing things from a single, heavily curated app store. 😫
Such great hardware and UX, ruined because their app distribution monopoly is more profitable than moving devices.
I’m less bothered about the App Store than I am just allowing these devices with these very powerful chips to stretch their wings.
And it’s just simple things, too. iPadOS should allow you to have two different audio tracks going at once. It can handle that. But it’s too skewed to being a “bigger iPhone” than it is a “smaller MacBook.”
And I honestly love my iPad. Using it right this moment with a wireless KB and trackpad. And I’m happy I can extend the display relatively easily to a monitor now, too.
But it could definitely be so much more if the handcuffs were taken off.
As an “infinity monitor” you can use anywhere, it is really great.
But it is not - as it was advertised - a “spatial computer”. I can’t even think of it as an AR device, because it is terrible at image recognition and tracking… something even an iPhone can do. I have no idea why that is the case, because the hardware is, theoretically, ridiculously powerful… but something is seriously, seriously wrong with the software right now… and it cripples the headset for what is supposed to be it’s one major use-case: spatial computing.
P.S. In case it wasn’t obvious, I bought one to develop for, and as a developer I’m pretty angry at how poorly it performs at basic AR tasks.
What do you mean with image recognition?
Like show it an image of a cat and it should recognize what it is?
Yeah. You train it to recognize an object/image, and then you can respond to that however you want, programmatically. It’s really cool for things like tabletop games.
For the how: Apple has APIs to including lightweight ML models for that sort of thing. You have to train a model, but for something like a card game it’s super easy. For physical objects it requires more prep just because you have to take photos instead of using existing artwork… but it’s still relatively straightforward.
What’s the use-case you’re trying to build?
I have a card game (a physical one, not virtual) and I want to “replace” the real cards with digital, animated, “living” ones. Ideally, I could apply this technique to other types of tabletop game, later… but cards are the current project.
This works fine on iOS and even in the Vision Pro simulator, but on hardware, the image recognition is slow and unreliable, and it doesn’t track items through space in real-time. It’s laggy and “floaty”. Image recognition for unique, flat cards should be one of the simplest possible use-cases… and given how much more powerful the hardware is than a phone, and the fact that it works on the simulator, it sure seems like a software issue… but you can’t ship Apps with such severe problems, either.
It looks like you’re hitting what this person called out:
https://twitter.com/nathanwchan/status/1754624863762047162?s=46&t=uP3SZsZSJ5VGsmbyPE2Jfg
And that is a sucky limit that makes your use-case not usable-able. Which is a shame because it’s a cool use-case.
The tracking is great for everything except image tracking, too.
Yu-Gi-Oh is finally becoming a reality
So Apple hardware that’s super powerful but has bad software to support all that power?
“Where have I heard that one before?” he types on his iPad Pro…
Ouch. Don’t get me started on iPad “Pro”. There is nothing “Pro” about a device that is so heavily sandboxed and restricted to only installing things from a single, heavily curated app store. 😫
Such great hardware and UX, ruined because their app distribution monopoly is more profitable than moving devices.
I’m less bothered about the App Store than I am just allowing these devices with these very powerful chips to stretch their wings.
And it’s just simple things, too. iPadOS should allow you to have two different audio tracks going at once. It can handle that. But it’s too skewed to being a “bigger iPhone” than it is a “smaller MacBook.”
And I honestly love my iPad. Using it right this moment with a wireless KB and trackpad. And I’m happy I can extend the display relatively easily to a monitor now, too.
But it could definitely be so much more if the handcuffs were taken off.