Automated Real-World Testing of Mobile Security Solutions – Android and iOS

Andreas Clementi, Peter Stelzhammer, Christoph Leitner & Philippe Rödlach, AV-Comparatives

Introduction
The number of security threats aimed at Android and iOS mobile devices is steadily rising, affecting users all over the globe. The number of threats targeting mobile platforms is now so great that manual testing of mobile security solutions using a representative threat sample would take a lot of time and resources. To ensure meaningful tests, automated testing processes are needed.

Solution
To address this issue, we have designed and implemented an automated testing framework for mobile security solutions. Despite the large degree of automation, the framework provides a real-world environment using physical and/or virtual devices and simulates the actions of a real user in everyday situations.

Capability of the Framework
The testing framework allows all the protection components of a security solution to be tested. Unlike a simple on-demand detection test, the test framework enables all of the security app’s modules to come into play. Malicious apps are downloaded and installed on the test device and then executed. Any and all relevant protection features are evaluated (URL block, static and dynamic analysis, etc.). In addition to anti-malware features, a security product’s anti-phishing and parental-control features can also be tested using a fully automated process.

Scalability
The framework is fully scalable and is limited only by hardware. It can also be used in a virtual environment, which is a more cost-effective solution. During our large-scale Android Security test, the framework proved capable of running over 400,000 test cases within a few days, on 200 physical Android devices.

Key Takeaways:

  • Challenges for testing on mobile devices
  • Understanding current testing methodologies used in mobile anti-malware testing
  • The role of automation in anti-malware testing
  • Hard-fact session: findings of latest Android testing
  • Use cases: for AV-vendors, app vendors, banking industry, health care industry and many more
  • Video demo

Andreas Clementi

Andreas Clementi is the founder and CEO of AV-Comparatives. He started his career in the anti-malware testing industry about 20 years ago as part of an academic project. His interest in the subject was awakened when he realised that tests of antivirus programs in computer magazines were sometimes contradictory, and so began his own intensive investigations of malware and antivirus software. He holds an academic degree from the University of Innsbruck and a doctoral degree from the University of Bolzano. His life is devoted to anti-malware testing and he has a passion for consumer rights and Internet privacy.

Peter Stelzhammer

Peter Stelzhammer, AV-Comparatives
Founder, kompetenzzentrum.IT
Co-Founder, AV-Comparatives
Member, Advisory Board Cluster IT, Tyrolean government’s IT strategy group
Member, Board of Directors, AMTSO – Anti-Malware Testing Standards Organization.

Peter Stelzhammer started working in IT in 1989. After 5 years working as the IT System Administrator of Alois Wild Group (Champion, Mexx, Benetton, Etienne Aigner), he became COO at Telesystem Tirol (an ISP and TV broadcasting company).

Christoph Leitner

Philippe Rödlach

The Dynamic Security Ecosystem
Other Topics