-

2024-06-28 14:34:29

Robert Kingett backup via Alt Text Hall of Fame

weirdwriter@tweesecake.social

I would like sighted people to see this thread in particular. There are actually a lot of blind people that prefer LLM descriptions over human ones but I have seen some very well written alt text on here. That being said, sighted people should read this thread https://www.applevis.com/forum/other-apple-chat/alt-text-adventures-ai-vs-human-descriptions #AI #AltText #Accessibility

show all 7 comments

2024-06-28 17:02:01

Jupiter Rowland

jupiter_rowland@hub.netzgemeinde.eu

@Robert Kingett, blind I might be an extremely rare exception, and I probably am. But I seem to be one of the very few human image describers who beat any AI out there not only in accuracy, but also in level of details and informativity.

Granted, that's easy to do for me. For one, the AI only has the image whereas I can examine the real deal from up close. Besides, describing and explaining my images accurately require extreme niche knowledge, sometimes up-to-date by mere days. No AI has this knowledge.

Oh, and by the way, I have actually let LLaVA describe two of my images which I had manually described first. I've posted the AI description separately and then reviewed it.

First image:

My original post, incl. short visual description in alt-text (382 characters of description, 920 characters altogether) and full and detailed description in the post (25,271 characters)
The image posted again, incl. my own short visual description in alt-text (382 characters of description, 1,064 characters altogether), my own full and detailed description in the post (still 25,271 characters) and a description generated by LLaVA (558 characters).
In the same thread as the link above, my detailed review of LLaVA's description, pointing out the mistakes it has made

Second image:

My original post, incl. short visual description in the alt-text (1,402 characters of description, 1,500 characters altogether) and full detailed description in the post (60,553 characters)
In the same thread as the link above, detailed description of the same image generated by LLaVA (1,120 characters)
In the same thread as the two links above, my detailed review of LLaVA's description, pointing out the mistakes it has made

I guess it should be clear that no AI can do in 30 seconds what took me up to two days.

(1/3)

#Long #LongPost #CWLong #CWLongPost #FediMeta #FediverseMeta #CWFediMeta #CWFediverseMeta #AltText #AltTextMeta #CWAltTextMeta #ImageDescription #ImageDescriptions #ImageDescriptionMeta #CWImageDescriptionMeta #AI #LLaVA #AIVsHuman #HumanVsAI

2024-06-28 17:04:18

Jupiter Rowland

jupiter_rowland@hub.netzgemeinde.eu

@Robert Kingett, blind For convenience, here is a quote-post of my review of the first image description by LLaVA.

Jupiter Rowland schrieb den folgenden Beitrag Tue, 05 Mar 2024 20:28:12 +0100

(This is actually a comment. Find another post further up in this thread.)

Now let's pry LLaVA's image description apart, shall we?

The image appears to be a 3D rendering or a screenshot from a video game or a virtual environment.

Typical for an AI: It starts vague. That's because it isn't really sure what it's looking at.

This is not a video game. It's a 3-D virtual world.

At least, LLaVA didn't take this for a real-life photograph.

It shows a character

It's an avatar, not a character.

standing on a paved path with a brick-like texture.

This is the first time that the AI is accurate without being vague. However, there could be more details to this.

The character is facing away from the viewer,

And I can and do tell the audience in my own image description why my avatar is facing away from the viewer. Oh, and that it's the avatar of the creator of this picture, namely myself.

looking towards a sign or information board on the right side of the image.

Nope. Like the AI could see the eyeballs of my avatar from behind. The avatar is actually looking at the cliff in the background.

Also, it's clearly an advertising board.

The environment is forested with tall trees and a dense canopy, suggesting a natural, possibly park-like setting.

If I'm generous, I can let this pass as not exactly wrong. Only that there is no dense canopy, and this is not a park.

The lighting is subdued, with shadows cast by the trees, indicating either early morning or late afternoon.

Nope again. It's actually late morning. The AI doesn't know because it can't tell that the Sun is in the southeast, and because it has got no idea how tall the trees actually are, what with almost all treetops and half the shadow cast by the avatar being out of frame.

The overall atmosphere is calm and serene.

In a setting inspired by thrillers from the 1950s and 1960s. You're adorable, LLaVA. Then again, it was quiet because there was no other avatar present.

There's a whole lot in this image that LLaVA didn't mention at all. First of all, the most blatant shortcomings.

First of all, the colours. Or the lack of them. LLaVA doesn't say with a single world that everything is monochrome. What it's even less aware of is that the motive itself is monochrome, i.e. this whole virtual place is actually monochrome, and the avatar is monochrome, too.

Next, what does my avatar look like? Gender? Skin? Hair? Clothes?

Then there's that thing on the right. LLaVA doesn't even mention that this thing is there.

It doesn't mention the sign to the left, it doesn't mention the cliff at the end of the path, it doesn't mention the mountains in the background, and it's unaware of both the bit of sky near the top edge and the large building hidden behind the trees.

And it does not transcribe even one single bit of text in this image.

And now for what I think should really be in the description, but what no AI will ever be able to describe from looking at an image like this one.

A good image description should mention where an image was taken. AIs can currently only tell that when they're fed famous landmarks. AI won't be able to tell from looking at this image that it was taken at the central crossroads at Black White Castle, a sim in the OpenSim-based Pangea Grid anytime soon. And I'm not even talking about explaining OpenSim, grids and all that to people who don't know what it is.

Speaking of which, the object to the right. LLaVA completely ignores it. However, it should be able to not only correctly identify it as an OpenSimWorld beacon, but also describe what it looks like and explain to the reader what an OpenSimWorld beacon is, what OpenSimWorld is etc. because it should know that this can not be expected to be common knowledge. My own description does that in round about 5,000 characters.

And LLaVA should transcribe what's written on the touch screen which it should correctly identify as a touch screen. It should also mention the sign on the left and transcribe what's written on it.

In fact, all text anywhere within the borders of the picture should be transcribed 100% verbatim. Since there's no rule against transcribing text that's so small that it's illegible or that's so tiny that it's practically invisible or that's partially obscured or partially out of frame, a good AI should be capable of transcribing such text 100% verbatim in its entirety as well. Unless text is too small for me to read in-world, I can and do that.

And how about not only knowing that the advertising board is an advertising board, but also mentioning and describing what's on it? Technically speaking, there's actually a lot of text on that board, and in order to transcribe it, its context needs to be described. That is, I must admit I was sloppy myself and omitted a whole lot of transcriptions in my own description.

Still, AI has a very very long way to go. And it will never fully get there.

#Long #LongPost #CWLong #CWLongPost #AltText #ImageDescription #ImageDescriptions #ImageDescriptionMeta #CWImageDescriptionMeta #AI #LLaVA

(2/3)

#Long #LongPost #CWLong #CWLongPost #FediMeta #FediverseMeta #CWFediMeta #CWFediverseMeta #AltText #AltTextMeta #CWAltTextMeta #ImageDescription #ImageDescriptions #ImageDescriptionMeta #CWImageDescriptionMeta #AI #LLaVA #AIVsHuman #HumanVsAI

2024-06-28 17:05:56

Jupiter Rowland

jupiter_rowland@hub.netzgemeinde.eu

@Jupiter Rowland And here is a quote-post of my review of the second image description by LLaVA.

Jupiter Rowland schrieb den folgenden Beitrag Sat, 18 May 2024 00:24:46 +0200

It's almost hilarious how clueless the AI was again. And how wrong.

First of all, the roof isn't curved in the traditional sense. The end piece kind of is, but the roof behind it is more complex. Granted, unlike me, the AI can't look behind the roof end, so it doesn't know.

Next, the roof end isn't reflective. It isn't even glossy. And brushed stainless steel shouldn't really reflect anything.

The AI fails to count the columns that hold the roof end, and it claims they're evenly spaced. They're anything but.

There are three letters "M" on the emblem, but none of them is stand-alone.There is visible text on the logo that does provide additional context: "Universal Campus", "patefacio radix" and "MMXI". Maybe LLaVA would have been able to decipher at least the former, had I fed it the image at its original resolution of 2100x1400 pixels instead of the one I've uploaded with a resolution of 800x533 pixels. Decide for yourself which was or would have been cheating.

"Well-maintained lawn". Ha. The lawn is painted on, and the ground is so bumpy that I wouldn't call it well-maintained.

The entrance of the building is visible. In fact, three of the five entrances are. Four if you count the one that can be seen through the glass on the front. And the main entrance is marked with that huge structure around it.

The "few scattered clouds" are mostly one large cloud.

At least LLaVA is still capable of recognising a digital rendering and tells us how. Just you wait until PBR is out, LLaVA.

#Long #LongPost #CWLong #CWLongPost #FediMeta #FediverseMeta #CWFediMeta #CWFediverseMeta #AltText #AltTextMeta #CWAltTextMeta #ImageDescription #ImageDescriptions #ImageDescriptionMeta #CWImageDescriptionMeta #AI #LLaVA

(3/3)

#Long #LongPost #CWLong #CWLongPost #FediMeta #FediverseMeta #CWFediMeta #CWFediverseMeta #AltText #AltTextMeta #CWAltTextMeta #ImageDescription #ImageDescriptions #ImageDescriptionMeta #CWImageDescriptionMeta #AI #LLaVA #AIVsHuman #HumanVsAI

2024-06-28 17:09:49

Robert Kingett backup

weirdwriter@tweesecake.social

@jupiter_rowland Very similar, but not quite, I think you would like my thoughts on a similar kind of service. So-called AI audio description https://robertkingett.com/posts/6567/

2024-06-28 17:40:16

Jupiter Rowland

jupiter_rowland@hub.netzgemeinde.eu

@Robert Kingett, blind I don't trust anything generated. At least not with super-obscure niche content like what I post.

And audio descriptions in general are why I'll never publish videos in the Fediverse.

I'd have to go into similar detail as for my pictures, only for moving pictures plus sound plus voice-over now. My descriptions would have to be so detailed that the video would have to pause to let the audio description catch up with the visuals. In fact, the video would spend more time paused while the audio description is rambling than actually moving, and it would never spend more than a few seconds moving at a time.

For one, I would have to describe and explain what the video shows at the very same level of detailed as I describe my images. And at least once I've described one single image at such a level of detail that it'd probably take a screen reader one full hour to read the image description aloud.

Besides, I would have take into account that it's a video. Everything would need timestamps. And instead of only describing the camera position and the camera angle, I would have to describe the camera movements like so:

Seven minutes, eighteen point one three seconds. The camera quickly rotates to the left around a vertical axis through a point roughly two point four metres straight ahead of the avatar. It starts rotating from the direction in which the avatar is facing, roughly twelve degrees to the east of north. The barn which has first appeared at five minutes, fifty-two point two eight seconds comes into view again, including all decoration around it. The camera only rotates around this vertical axis and not around any horizontal axis. The avatar does not rotate with the camera.

Seven minutes, eighteen point six four seconds: The video pauses to let this description catch up.

Seven minutes, eighteen point seven one seconds: The video no longer pauses. The camera reaches a rotation angle of roughly twenty degrees to the south of west. The rotation speed of the camera slows down. It continues to rotate to the left.

Seven minutes, eighteen point nine three seconds: The video pauses to let this description catch up.

Seven minutes, nineteen point zero four seconds: The video no longer pauses. The camera stops rotating at an angle of roughly twenty-five degrees to the west of south.

That is, in order to cater to deaf-blind users, I would have to have two time codes. One, the time code of the original video, not taking the pauses into account. Two, the time code of the described video with catch-up pauses.

And the video with catch-up pauses would be dramatically longer than the original video. Ten minutes of video would take me weeks to describe, probably over a month. And it would end up many hours long, depending on how much there is to describe and explain.

So a time code in the Braille description for deaf-blind users might actually read, "Six minutes, thirty-seven point five five seconds in the original video, fourteen hours, three minutes, forty-nine point two one seconds in this described version of the video."

By the way, no, an AI can't do that.

#Long #LongPost #CWLong #CWLongPost #MediaDescription #MediaDescriptions #AudioDescription #AudioDescriptions

2024-06-29 04:36:54

Robert Kingett backup

weirdwriter@tweesecake.social

Long post

Artikel ansehen

2024-07-01 15:46:55

Jupiter Rowland

jupiter_rowland@hub.netzgemeinde.eu

@Robert Kingett, blind If at all, I would have to do it all myself with no technical aids involved. Nobody would be able to help me with it.

After all, I'm not talking about videos shot in real life with a video camera, especially not scripted ones.

I'm talking about unscripted, spontaneously produced video captures from very obscure 3-D virtual worlds. In order to describe these videos properly, extensive detail knowledge about this super-niche topic, its technology and its culture would be absolutely mandatory, and it would have to be up-to-date by hours at most. This detail knowledge is also necessary to be able to judge what has to be explained and described.

Also, it'd be impossible to properly describe these videos by watching these videos. They can only be described by logging into that world, teleporting to the place where the video starts and then looking at everything that's shown in the video from up close and sometimes even from different camera angles than in the video itself.

#Long #LongPost #CWLong #CWLongPost #MediaDescription #MediaDescriptions #AudioDescription #AudioDescriptions