Automate All the UIs!  ## Metadata - Author: [[Changelog Master Feed]] - Full Title: Automate All the UIs! - Category: #podcasts - URL: https://share.snipd.com/episode/19d6a4e2-4edc-4583-a289-55ebcb2b7536 ## Document Notes ## Highlights - **Visual UI Automation Approach** - askUI automates UI interaction by analyzing screenshots rather than scraping source code. - This visual approach allows automation across operating systems, including Windows, Mac, Linux, and Android. Transcript: Speaker 2 Is it more of a visual thing or is it something else? Speaker 1 Yeah, as a real on the visual part, for example, when you're talking about Selenium, Selenium is only working with web interfaces and you cannot use this, for example, on Android and So on. What we are doing, we're doing, we're doing, we do a screenshot of the system. So it means we can take a screenshot of the application, we can take a screenshot directly off the operating system. And then our AI model, what we trained can detect the user interfaces. That means we detect in buttons, we detect in text, we detect in the text fields, we detect in checkboxes, we detect in icons. And so we have already a trend model which can understand the visual representations of user interface. And then we connected our natural language part to it to match the intention, what do you have? For example, click on the lock in button to the really lock in button on the user to fight and then we are moving the mouth there. So this is a little bit different from the concept versus I connect directly to the application and then try to scrape the source code off the application to get my information. Speaker 3 Maybe this is the best way to differentiate. So you're kind of starting with a screenshot instead of doing web scraping. Correct. And then just doing classification on the screenshot itself to do that. Speaker 1 Correct. So we have object detection model in class, which really takes a screenshot and detect all the elements on it. Speaker 3 As a quick follow up, when you do the classification and identify what you have in the screenshot, how do you tie that in with the test desired? So you have a collection of tests that you're trying to automate. How are you like tying like, I got a screenshot. I got, I have a button, I have a text field. How does that tie into a particular like a unit test or something? Speaker 1 Currently, we have our textured application, which you can download, which is also available as an Ask the Eye campaign package, which you can use. And with this, you can directly start, install it and then generate a standard test. And then you have, for example, a check test test block, or Java JUnit test block, where you then can write, ask your eye.click.withtext.so on. And the background, what's happens, we have a controller then installed on your local system, which connects to his operating system, takes a screenshot and also have the ability To control the mouse. Then we're taking the screenshot, we are connecting the instruction, the click on button, for example, then this to our inference backend, then we get the result back and then we're Moving the mouse there. ([Time 0:08:45](https://share.snipd.com/snip/bc4c59a2-3009-4497-9699-92cbe863cfd7)) - **Expanding Automation Horizons** - Flexibility of AI lets askUI automate diverse UI workflows beyond just regression testing. - Future lies in automating transfer of unstructured data like PDFs into structured UI forms. Transcript: Speaker 2 Previously. Speaker 1 The QMLA, because we are using AI technology and with AI technology, you are a little bit more flexible what you can describe because you can learn it based on data and have not to explicit Describe it, which things you want to do. You use cases, what we started where we want to do regression tests for classical use interface tests, then we moved a little bit around in the direction of a robot processor to imagine. I think where we were landing in the future is to automatically transfer unstructured data to use interfaces. If you imagine you have a PDF, which is a formula, and then you want to say, hey, please, as normally, normal workers will do with this, which are copying, for example, the PDF or information From the PDF over to another formula. What you can do then with our technology is to say, hey, please automatically copy all the information over, which is on the left side to the right side. This is where we think, where the technology will go, so that you can have not defined the matching of the files. Everything, when you then connect also ([Time 0:12:33](https://share.snipd.com/snip/92d12441-334b-4e71-aff8-ad0d10a17038)) - **Getting Started with askUI** - Begin automating by uploading screenshots and simulating workflows on askUI's platform. - Use scheduling and Docker containers for running automated tests, simplifying maintenance and scaling. Transcript: Speaker 3 It, what does that picture look like? What does the customer go through as they start deploying or start utilizing the service? Speaker 1 What he's doing too? There's also a little bit history but what is this now? You're going now to our website, then lock in with your credentials and then you can directly upload to your first screenshot and then simulate on this. So do a simulation directly on the screenshot so that you see the effect. And then the next step what you want to do if you have created your workflow a little bit, then you want to automate this and you send and then schedule this, for example, in a Docker container In the background we have already. Also if you're on the weapon, then after time you get the result and you're happy that you have automated with really easy things, the workflow. What we are now doing is to reduce also the hurdle that you have to learn less as now because, you know, this is a thing. We're trying to bring all the problems from the user perspective away. The user has a really easy life to create automations, to maintain automations ([Time 0:20:40](https://share.snipd.com/snip/961c368c-5733-4a9e-80b6-1266fad563eb)) - **Machine Learning Startup Advice** - Use tools like Hugging Face and PyTorch Lightning to ease machine learning model building. - Start with a researcher and a software engineer working together for balance between research and engineering. Transcript: Speaker 3 Productive? OK. Speaker 1 Yes. First of all, what I would recommend directly introduce tools which support you. Use directly hugging face, try to build models based on hugging face so that you have libraries are based on PyTorch lightning because they're giving things for free. I would always say. Then the other thing I would directly introduce version control system for data. Beginning, I would recommend we are currently using clear. I would now recommend DVC at the beginning and then try to find, especially for the machine learning part, try to find one's researcher and one software engineer and bring them together And let them communicate. Because then you get the efficiency from software engineers or especially cloud software engineers and you get the research knowledge and bring them together. So they can learn from each other to exchange ideas, how to do this in the software manner and how to do research. So and this I would when I would start again, I would now say, hey, here you have one team, two people, one software guy, which DevOps background also visited the software development And DevOps spectrum a little bit. And then I'm bringing a machine learning researcher to it. And then I think they would benefit the best. Yeah. Speaker 2 And you talked a little bit ([Time 0:36:22](https://share.snipd.com/snip/e1c45807-8967-426d-b894-25f40fb9cab8)) - **AI Development's True Challenge** - The biggest challenge in AI product development is streamlining the process, not the technology itself. - Focusing teams around shared goals and efficient iteration speeds up progress significantly. Transcript: Speaker 2 Something completely different? Speaker 1 I would say more. It's not more about the technical challenge. What you could solve because technical challenge you can solve was with knowledge and a little bit of research. So normally if it's not physically impossible, you can solve things with a certain time. This is the nominally no problem. The main problem is what I see is to speed up the development process itself. So that the right things are researched, the right things are designed and the right things are started to develop so that you're bringing more the focus on one topic. And if you have a lot of people in your company or a lot of, I would say, interfaces, then you have to bring them one common understanding what you want to achieve, how you want to work, how You want to define requiring. This is, I would say, currently the main challenge when I would say so. And then from the technical challenge, we have to talk to customers, get the feedback, but it's a stuff as quick as possible and iterate in terms of what the business wants to have. Speaker 3 So as we kind of wind up here, and this is ([Time 0:38:27](https://share.snipd.com/snip/8c689c8a-c7d9-4f58-88db-9404d8ebb22b)) - **Vision for Universal UI Automation** - The ultimate vision is AI that understands any kind of data and automates UIs effortlessly for all users. - Combining LLMs and visual understanding enables automation from manuals and simple instructions without programming. Transcript: Speaker 3 Bed at night, you're like, that's the place I'm going eventually. What does that look like? Speaker 1 Maybe to look a little bit back in the history, how all this project is thought, I think I haven't told this before. When I was also at Siemens, then I also do a done-my-master sees about visual accretion answering, which should solve the task, what we are doing now, and separate tasks as an end-to-end Task. So what my dream is, is to take the now available technology, lab language models, also including the visual part at the natural part, combine everything. So that we can teach or bring every kind of data inside the model. What does this mean? Every kind of data. So this means we get, for example, manuals for software, which says, hey, please click on this button. So then of course, I'm coming to you and say, hey, please create me account in this one. You give this a manual and then it do it completely, automatically without any kinds of learning because it's learned how to interact with the operating system. This is one thing where we want to go to bring nowadays technology in, to make it accessible for the users, and also that everyone, really everyone can use it also your grandpa. ([Time 0:40:18](https://share.snipd.com/snip/d945541a-ccd9-42c6-8ef2-a622f523c397))