AI in the acid test: How good is GenAI really?
The project had a clear focus: understanding and analyzing how GenAI can support and accelerate the software development process.
Metrics were defined to measure the increase in efficiency and the development process was compared with and without AI support.
Four students from Rosenheim Technical University of Applied Sciences developed an IoT implementation for the management of wind farms in the cloud. The process consisted of two phases: Modeling and implementation. Development time, developer satisfaction and qualitative aspects of the software were evaluated.
Key findings
- GenAI significantly speeds up the development process, especially for standardized tasks. 94 % of students stated that AI has increased their productivity1.
- AI can also bring efficiency gains in the modeling phase, e.g. in the creation of documentation, log creation, formulation of tickets, creation of use case diagrams and UI mockups.
- In the implementation phase, efficiency gains are possible, particularly in code generation, testing & debugging, creating unit tests, improving code quality, refactoring and creating simple UIs.
- Efficiency can be increased by up to 50 % for many tasks. Significant time savings are particularly evident when creating unit tests and simple API integrations.
Challenges
Despite many advantages, the use of generative AI also poses challenges in software development. For example, AI sometimes generates insecure code, for example through the unencrypted storage of passwords or incorrect configurations in CI/CD pipelines. This can open up potential security vulnerabilities.
It also became apparent that the reliability of the AI is limited when it comes to complex tasks. In these cases, the AI occasionally produced inaccurate or incorrect results that required manual reworking by the team.
AI also reached its limits in the area of creativity. Although it could create basic UI designs, its support in the development of complex, structured documents and appealing user interfaces was limited.
In addition, there was the problem of a lack of contextualization: the AI had difficulties processing large tasks with many dependencies efficiently and grasping the necessary overall context, which led to suboptimal results.
AI: game changer or risky experiment?
Generative AI significantly optimizes the software development process, but does not replace human expertise. The quality of the results depends heavily on the precision of the prompts and the experience of the users. Efficient use of AI requires in-depth specialist knowledge in order to recognize and correct potential errors.
A key success factor is the continuous evaluation of results and the iterative improvement of AI prompts. Breaking down complex tasks into smaller, clearly defined steps is particularly beneficial in order to optimize the results in a targeted manner.
The cooperation with the TH Rosenheim has provided valuable insights. Not only was it possible to better understand the potential of GenAI in the software development process, but also to develop practical methods for integrating AI into existing workflows.
If you would like to find out more about how you can use Generative AI to make the software development process in your company more efficient, you can find more information on our AI Assistant Services page at doubleSlash.
