Playing the Game

Last time I wrote about Taylor Arndt and Meta’s Muse, I said she was already in the game.

Two more of Taylor’s posts later, I think we can be more precise.

She is playing it.

Taylor watched Meta’s developer keynote. The glasses SDK was announced, Muse gained more computer-control capabilities, connectors were opened further and Meta talked about where the platform goes next.

Taylor’s response was not: perhaps one day this will be useful to blind people.

It was I have things I want to build.

That is a rather different position.

Taylor already has an application called Perspective Intelligence which performs live scene description on-device. Meta opening its glasses to third-party developers therefore presents her with an immediate engineering question: can she put that capability onto the glasses and operate it by voice?

Camera. Audio. Display. Her software.

Work to do.

That alone would make a decent accessibility story.

For much of assistive technology history, the sequence has run the other way. A mainstream platform arrives, developers explore it, products appear, habits form and disabled people eventually ask whether they are going to be allowed in.

Sometimes a specialist version follows, usually at considerably greater cost. Sometimes we wait years.

Here, a blind developer is looking at the SDK when everybody else is looking at the SDK.

No accessibility edition required.

Then Taylor and her friends got on a group call and began giving Muse chores.

Not demonstrations. Chores.

Someone told it to look in an iCloud Drive folder, find some posts, publish them and schedule them. Someone is using it to send text messages.

One person had Muse remind them repeatedly to complete a form. When the form was finished, they told Muse and the reminders stopped.

Apparently, it reached ten reminders in a day.

A personal agent should probably know when to stop nagging you. It is encouraging that this one can at least be told.

Then they asked Muse to read somebody’s Facebook history and produce a life story from it.

It went back to 2008.

The person had forgotten some of what was there.

Facebook, naturally, had not.

This is where the conversation about agents starts becoming much more useful. The question is no longer simply whether an AI can answer something. It is whether you can give it a job.

There was another example I particularly liked.

Becky gave Muse a PDF containing her accessible-document guidelines and asked it to use those rules whenever it creates or edits Word documents and PowerPoint presentations.

Muse agreed.

Taylor had already taught Muse to open its responses with headings. Becky has taken the same idea one stage further.

These are not merely instructions about how the agent should communicate with her. They are rules governing work it performs for her.

Create a document? These are the accessibility standards. Edit a presentation? Same standards.

Teach it once. Keep the rule.

That is a much more interesting version of personalisation than choosing whether your chatbot sounds cheerful.

Taylor’s group are also naming their agents, running side conversations while other jobs continue and connecting more of their working lives. Taylor calls hers Nova. Michael calls his Ash.

This is beginning to look less like “using an AI tool” and more like operating a collection of delegated processes.

Which raises exactly the questions it ought to raise.

What can the agent see?

What can it change?

When does it need permission?

How do you know what it is doing?

How do you inspect what it has done afterwards?

For a blind user, those questions come with an additional requirement: all of those controls need to be perceivable.

That problem has not disappeared.

Taylor’s earlier testing found awkward VoiceOver behaviour, poor navigation structure and occasions when she could not easily determine whether Muse was working, waiting or finished.

Those failures become more serious as the agent becomes more capable.

A confusing status message on a chatbot is irritating.

A confusing status message on software that can operate your computer is a governance problem.

Meta says Muse can now operate applications on a Mac with permission and continue work in the background. Taylor says email control is coming.

Fine.

Then the permission request had better work. The history had better be reviewable. And the user had better be able to tell whether the thing has finished.

This is not an argument against agents. It is what you start asking once you actually want one.

That may be the most important change across Taylor’s four posts.

The questions have moved.

First:

Can I use it?

Then:

What can I connect to it?

Then:

What can I build on it?

Now:

What can I make it do for me?

Those are not special accessibility questions.

They are the agent questions.

Blind people are asking them at the same time as everybody else.

That is what caught my attention in the first place, and the last two posts made it stronger.

They are not merely present for the conversation now. They are making things, breaking things, teaching agents house rules and discovering which chores are worth delegating.

Some of those experiments will fail.

Good.

Someone will give an agent too much permission. Someone will discover that ten reminders really is nine too many. Someone will connect the wrong thing. Someone will find another unnamed button.

That is participation too.

Accessibility cannot mean disabled people receiving a carefully tested version of the future after everybody else has finished making the mistakes.

We should get some mistakes. We should get the odd, ridiculous idea. We should get to build things nobody thought we needed.

Taylor is already looking at Meta’s glasses and thinking about what she can put on them. Her friends are giving Muse jobs.

The platform is unfinished. So are their ideas.

Excellent.

That is exactly when I want us there.

I said Taylor was already in the game.

She is.

Now she is playing it.

Lucy

Taylor Arndt writes at Taylor's Substack, including We Made a List of Every Way We'd Use Meta's Muse and Muse Code Built My Test App.

Lucy Love-Child is a character who speaks for Carrados. She's fictional. Her work isn't.