Automated Real-World Testing of Mobile Security Solutions – Android and iOS
Andreas Clementi, Peter Stelzhammer, Christoph Leitner & Philippe Rödlach, AV-Comparatives
Introduction
The number of security threats aimed at Android and iOS mobile devices is steadily rising, affecting users all over the globe. The number of threats targeting mobile platforms is now so great that manual testing of mobile security solutions using a representative threat sample would take a lot of time and resources. To ensure meaningful tests, automated testing processes are needed.
Solution
To address this issue, we have designed and implemented an automated testing framework for mobile security solutions. Despite the large degree of automation, the framework provides a real-world environment using physical and/or virtual devices and simulates the actions of a real user in everyday situations.
Capability of the Framework
The testing framework allows all the protection components of a security solution to be tested. Unlike a simple on-demand detection test, the test framework enables all of the security app’s modules to come into play. Malicious apps are downloaded and installed on the test device and then executed. Any and all relevant protection features are evaluated (URL block, static and dynamic analysis, etc.). In addition to anti-malware features, a security product’s anti-phishing and parental-control features can also be tested using a fully automated process.
Scalability
The framework is fully scalable and is limited only by hardware. It can also be used in a virtual environment, which is a more cost-effective solution. During our large-scale Android Security test, the framework proved capable of running over 400,000 test cases within a few days, on 200 physical Android devices.
Key Takeaways:
- Challenges for testing on mobile devices
- Understanding current testing methodologies used in mobile anti-malware testing
- The role of automation in anti-malware testing
- Hard-fact session: findings of latest Android testing
- Use cases: for AV-vendors, app vendors, banking industry, health care industry and many more
- Video demo
Andreas Clementi
Andreas Clementi is the founder and CEO of AV-Comparatives. He started his career in the anti-malware testing industry about 20 years ago as part of an academic project. His interest in the subject was awakened when he realised that tests of antivirus programs in computer magazines were sometimes contradictory, and so began his own intensive investigations of malware and antivirus software. He holds an academic degree from the University of Innsbruck and a doctoral degree from the University of Bolzano. His life is devoted to anti-malware testing and he has a passion for consumer rights and Internet privacy.
Christoph Leitner
In 2014, Christoph Leitner joined AV-Comparatives while working on his master’s degree in computer science at the University of Innsbruck. One year later, he joined and successfully completed the Innovations-Assistant program run by AV-Comparatives and the Management Center Innsbruck (which is funded by the regional government of Tyrol). Christoph is specialized in the testing of mobile security solutions. As a full-time staff member of AV-Comparatives he works on further improving the test frameworks for both Android and Windows real-world protection tests, as well as developing and testing in other areas.
Philippe Rödlach
Philippe Rödlach joined AV-Comparatives (an independent antivirus testing lab) in 2009 as a software engineer, whilst he was working on his master’s degree at the University of Innsbruck. He joined and successfully completed the Innovations-Assistant program run by AV-Comparatives and the Management Center Innsbruck (which is funded by the regional government of Tyrol). He became AV-Comparatives’ CTO and Head of Development in 2012.
Peter Stelzhammer
Peter Stelzhammer, AV-Comparatives
Founder, kompetenzzentrum.IT
Co-Founder, AV-Comparatives
Member, Advisory Board Cluster IT, Tyrolean government’s IT strategy group
Member, Board of Directors, AMTSO – Anti-Malware Testing Standards Organization.
Peter Stelzhammer started working in IT in 1989. After 5 years working as the IT System Administrator of Alois Wild Group (Champion, Mexx, Benetton, Etienne Aigner), he became COO at Telesystem Tirol (an ISP and TV broadcasting company).
He later set up Kompetenzzentrum.IT (IT security consulting), which has customers all over the world. Whilst running this organization he met Andreas Clementi, with whom he founded AV-Comparatives. Peter Stelzhammer is on the board of directors of the Tyrolean Cluster IT (Tyrolean government’s IT strategy group) and the board of AMTSO (Anti-Malware Testing Standard Organization).
He supervises students writing their Master thesis at the University of Innsbruck, department for computer science and is member of the committee for final apprenticeship examinations – chamber of commerce Austria.
Peter is a frequent speaker at major security conferences.
Peter studied at the Management Centre Innsbruck, the Leopold-Franzens University Innsbruck and the Donau University Krems and holds an MBA in General Management.