I've just been playing about with ChatGT's computer use features. Yes I know evil AI. but I thought it might be able to do things in a window that's inaccessible to me, namely Windows Komplete Kontrol's preferences dialogue.
turns out that no, it can't, and seemingly for the exact reasons I can't get to any of the controls with my screen reader, because all they both see is one big unnamed control. NVDA just says "Unknown".
Tried ChatGpT's Computer Use skill again , this time with NI's Reaktor. I asked it to enable midi input, which it innitially couldn't, but managed to do it when I asked it to sort itself out. It ended up controlling it via a PowerShell script.
Then I asked it as a test to create a basic Reaktor ensemble synth with a saw oscillator, a filter, an envelope and an amplifier with the modwheel controlling the filter cutoff. about 20 minutes later, I saw that it only went and did exactly what I asked and it works. Fucking hell. Mind blown.
I can't use Reaktor independently because the UI is inaccessible.
Just to add that this was with the Codex side of things.
Maybe it would've done it quicker if I asked it to make the ensemble from code or something, rather than manipulating the UI. but I wanted to see it actually use the UI and make sence of it.
@KaraLG84 huh. It should be able to do a lot more just by using visual screenshots instead of relying on accessibility APIs but I haven't tried that specific UI. Although come to think of it, don't a lot of controls in KK sorta kinda work with screen readers these days?
@zersiax they do but not in the preferences dialogue.
@zersiax This is what it said:
I can detect Komplete Kontrol and its Preferences dialog, but I can’t read the MIDI input setting. Screen capture times out, and the accessibility text exposes an empty Preferences pane.
A screenshot of the MIDI Inputs tab would let me identify it. I haven’t changed any settings.
@KaraLG84 ok so yeah the screen capture is failing for some reason. I'd poke it to figure out why that is happening and fix it
@zersiax I got it to do that and it ended up making a powershell script. I tried asking it to create a very basic reaktor ensemble and it only went and did it. My god.
@zersiax To clarify I gave up with the Komplete Kontrol app and got it to use Reaktor stand-alone.
@KaraLG84 @jaybird110127 AI is not evil. It is a tool that can be used for good or ill. It has amazing benefits for our community so don't give into the temptation to say "I know, AI is evil" hoping to mollify the anti-AI people on this platform.
@KaraLG84 haha did you actually say to it: "Sort yourself out?" LOL
@VE3RWJ Not literally. I don't want it sending drones at me. lol
@KaraLG84 Like, Oy! Mate! Check yoself before you wreck yoself!