Testdroid: Best Practices in Mobile Game Testing

Software

bitbar
of 69
Description
Watch a live presentation at http://www2.testdroid.com/archive-learn-the-best-practices-in-mobile-game-testing

To get the best start for your mobile game - or just to improve the existing game's possibility to get in front of hundreds of millions of gamers - we'll be sharing our insights, best practices and lots of tips&tricks how to gain advantage in this race.
Text
  • 1. Learn the BestPractices in MobileGame Testing© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 12 December 2014Ville-Veikko HelppiTechnical Product Managerville-veikko.helppi@bitbar.comW E B I N A R
  • 2. • Different Types of Testing and How GameTesting Differs from Regular App Testing?• Frameworks, Tools and Technologies – What isImportant in Mobile Game Testing Today?• Top Priorities for Mobile Game Testing• Infrastructure for Successful Game Testing• Hands-on Example (Image Recognition)• Testdroid Update• Q&A© Copyrights by Bitbar Technologies Ltd. 2014AgendaAll rights reserved. 2
  • 3. Let's Define 'Testing'...© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 3
  • 4. Different Types of Testing?• Black-box testing vs. White-box testing• Methods used in Mobile Game Testing© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 4– Functional– Compatibility– Performance– Localization– Regression– Load– Smoke– Acceptance
  • 5. Defining Some Ways to TestFunctional Testing • Functionality of the game-play, features• Done always when new features/regressions are included Regression TestingCompatibility Testing • How game runs on different configurationsLocalization Testing • Different languages, geo-focused materials• Endurance test to determine if system canhandle the load Soak TestingStress Testing • Measures the capacity of the system• Simplest form of performance testing,measures how system handles certain loads Load Testing• Isolation of the environment (e.g. fromnetwork) to see how game works Hermetic Testing© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 5Feature-basedtestingEnd-usertestingPerformancetesting
  • 6. The Old-Fashion Thinking© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 6Deterministic systemsIdentifiable objectsTool support(s)User-driven actionsSimple inputs/outputsVirtualizationRandom setups/resultsNo handle on objectsNo standardsUser reacts to inputComplex inputs/outputsReal hardware<><><><><><>High-level of automation Low-level of automation
  • 7. Test Automation as a SolutionRandom setups/results -> Easy to generate random eventsNo handle on objects -> Variety of options (e.g. imagerecognition)No standards -> Variety of frameworks (cross-platformetc.)User reacts to input -> Test scripts can handle randomeventsComplex inputs/outputs -> Test scripts can generate input/contentTest Automation brings real valueReal hardware -> Availability of real mobile hardwarewith variety of options to mobile game testing© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 7
  • 8. Challenges that Test Automation Tackles• Proliferation of Devices, Brands, Operators,© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 8Platforms• Coupling the Testing and Development process(Continuous Integration/Deployment/Delivery)• Different Types of Testing (as automated)– Stress, Load, Performance, Functionality etc.• Test script & test asset usability• Cons of Manual testing– Tedious effort, Expensive human resources, Error-prone(as done by humans) & Not reusable (done once)
  • 9. Significant Benefits of Using AutomationOEMmods© Copyrights by Bitbar Technologies Ltd. 2014Relationto othersoftwareAll rights reserved. 9OSversionsChipsetsCPU + GPUTens ofOEMsMemoryDisplays(resolutions,physicalhw)Otherhardware(connectivity,calibration)
  • 10. Different 'Verticals' Require Different Practices© Copyrights by Bitbar Technologies Ltd. 2014Typically need tofully utilize HW(CPU+GPU)Resource (e.g.battery)consumptionOpenGL ES 2/3Correct behaviouracross platforms andbrowsersIntegration with webback-endsAll rights reserved. 10Functionalityand usabilityScreenorientations,connectivity,user profilesRobustnessRobustness andsecurity!BrandCompliances,verification with back-endsand data
  • 11. Importance ofMobile Game Testing© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 11
  • 12. Important Aspects in Game Testing• User Interface and Functionality– UI layouts/elements, Menus, Resolution,Orientation, Overall visuality• Graphics Performance• Usability and User Experience– Responsiveness, Real Occuring Events• Multi-player/User• Social Integrations• Security and Liabilities© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 12
  • 13. User Interface and Functional• Compared to regular app testing, mobile gametesting may seem to be more challenging– e.g. OpenGL ES content not action-identifiable• Graphics and UI – Identical look&feel across alldevices and variants• Usability controls in the context of game• Navigation/game-play, Progress, Delaysbetween "game-stages"• Animations, Fonts, Other graphics assets• Screen Resolution and Orientation© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 13
  • 14. Performance Testing• Performance, Load, Stress, Spike, Soak• Measure the realuser experience• Execute tests withall possible devices(SW&HW)• Discover the impactof server loads, appperformance(CPU/GPUutilization)© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 14
  • 15. Performance Makes +/- UX• Studies: Majority ofusers leave game duesluggish, generalslowness or other badperformance behavior• It's very important tounderstand how gamebehaves on differentdevices, underdifferent circumstances© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 15
  • 16. Building the User ExperienceMonetizationAre gamers willing to buy – invest – in gamePlaying ExperienceIntuition to follow the game and go forwardwith its instructions and game-playUsabilityThe main User Interface, menus, design andhow usable those areUnderstandingHow quickly gamer assimilates what thegame is about, how to fully play with it, whatare the functionalities.© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 16• Graphics performancecapability is the mostimportant thing for mobilegames when forming UX• Too many games fail towork properly with low-end/mid-range devices duehardware performance• The real understanding ofgame's performancecapability can be onlyfigured out by testing onreal devices
  • 17. • End-to-end between tests, AI ormore simple implementations• Optimize the data communicationbetween server and clients• Test simultaneously on several© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 17(different) devices• Check what happens if game getsinterrupted (platform)• Location-based features – how toproperly mock up those?• Hermetic environment – whathappens?Multi-Player/User Features
  • 18. • Important part of the after-engagement(how gamers© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 18share their results)• Many of OEM brandeddevices provide certainSoMe apps pre-installed(version?)• Connectivity, login anddata transferring aretested in context of game• Hermetic environment –what happens?Social Integrations
  • 19. Security and Liabilities© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 19
  • 20. 4 Top Priorities inMobile Game Testing© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 20
  • 21. © Copyrights by Bitbar Technologies Ltd. 2014LocalizationAll rights reserved. 21
  • 22. Device DiversityBy OpenSignal, Inc. 2014 - http://opensignal.com/reports/2014/android-fragmentation/© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 22
  • 23. Real Devices vs. EmulatorsWhy Real Devices are Must-to-Have?• Emulators/simulators won’t help you to test…– User Experience and Usability– Hardware– Software– Infrastructure0 % = the percentage of your app usersthat use emulator to run your app!© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 23
  • 24. Manual vs. AutomationManual Automation© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 24Smaller coverage, Moremoney burnt & timewasted, Error-proneLarge coverage,quicklycompleted, Lessmoney & timewasted, Exactresults.
  • 25. Infrastructure forMobile Game Testing© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 25
  • 26. High-Level Infrastructure© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 26
  • 27. Architecture / Infrastructure© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 27GAMEGameEngineAndroidiOS...API&servicesServerDatabaseApp Middleware Platform Connectivity/Network Back-end
  • 28. Isolated, Hermetic Testing© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 28GAMEGameEngineAndroidiOS...API&servicesServerDatabaseHermetic EnvironmentApp Middleware Platform Connectivity/Network Back-end
  • 29. Compatibility & Porting Testing© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 29GAMEGameEngineAndroidiOS...API&servicesServerDatabaseReal DevicesApp Middleware Platform Connectivity/Network Back-end
  • 30. Smoke/Reference Testing© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 30GAMEGameEngineAndroidiOS...API&servicesServerDatabaseReference EnvironmentApp Middleware Platform Connectivity/Network Back-end
  • 31. Back-End Performance Testing© Copyrights by Bitbar Technologies Ltd. 2014Massive no. of usersand significant loadAll rights reserved. 31GAMEGameEngineAndroidiOS...API&servicesServerDatabaseApp Middleware Platform Connectivity/Network Back-end
  • 32. Continuous Integration© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 32
  • 33. Game Testing Process© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 33
  • 34. Hands-on Example:Integrating Development withTesting Environment© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 34
  • 35. Setup• Continuous Integration (e.g. Jenkins)• Appium framework (for Android and iOS)• No need for devices – we'll use devices onTestdroid Cloud service• Test scripts / test cases for game© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 35
  • 36. Continuous Integration© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 36
  • 37. Family Tree: Android TestAutomation FrameworksAppium works for both, Android and iOS© Copyrights by Bitbar Technologies Ltd. 2014Appium IntroAll rights reserved. 37
  • 38. • Uses Selenium Webdriver (W3C standard) as a201438© Copyrights by Bitbar Technologies Ltd. 2013All rights reserved. 38scripting framework• Supports native Android, native iOS and mobileweb:– Android via uiautomator (API level >=16) andSelendroid (API level <16)– iOS via UI Automation– Mobile web as Selenium driver for Android and iOS• You can write your Appium scripts on almost anyprogramming language(Haskell/Go/Clojure/Java/Ruby)Appium Basics
  • 39. Desired Capabilities / Android© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved.39Test ScriptDesiredCapabilitiesTest Case{“device”: “Android”,“app”: “/Users/user/ApiDemos.apk”“app-package”: “com.example.android.apis”“app-activity”: “.ApiDemos”}Example
  • 40. © Copyrights by Bitbar Technologies Ltd. 2014All rights reserved.40Test ScriptDesiredCapabilitiesTest Case{“app”: “com.bitbar.testdroid.BitbarIOSSample”}ExampleDesired Capabilities / iOS
  • 41. Appium @ Localhost© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved.Example41Test ScriptWebDriverDesiredCapabilitiesTest Casehttp://localhost_4723/wd/hub4723Appium ServerDevice
  • 42. From Localhost to TestdroidCloud*Testdroid Caps© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved.Test ScriptWebDriverDesiredCapabilitiesTest Casehttp://localhost_4723/wd/hubhttp://appium.testdroid.com/wd/hubExample
  • 43. From Localhost to TestdroidCloudWebDriver {© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved.Test ScriptDesiredCapabilitiesTest Case“testdroid_username”: “user@domain.com”,“testdroid_password”: “p4s$w0rd”,“testdroid_project”: “My First Project”,“testdroid_testrun”: “Test 1”,“testdroid_device”: “iPad Mini 7.0.4 A1432”,“testdroid_app”: “http://domain.com/app_v1.ipa”..“app”: “com.bitbar.testdroid.BitbarIOSSample”}Example
  • 44. © Copyrights by Bitbar Technologies Ltd. 2014All rights reserved.Example44
  • 45. 350+ Android & 20+ iOS Devices© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 45
  • 46. © Copyrights by Bitbar Technologies Ltd. 2014All rights reserved.Example46
  • 47. © Copyrights by Bitbar Technologies Ltd. 2014All rights reserved.Example47
  • 48. © Copyrights by Bitbar Technologies Ltd. 2014All rights reserved.Example48
  • 49. © Copyrights by Bitbar Technologies Ltd. 2014All rights reserved.Example49
  • 50. © Copyrights by Bitbar Technologies Ltd. 2014All rights reserved.Example50
  • 51. © Copyrights by Bitbar Technologies Ltd. 2014All rights reserved.Example51
  • 52. Projects and Test Runs© Copyrights by Bitbar Technologies Ltd. 2014ExampleAll rights reserved. 52
  • 53. Behind the SceneAppiumBroker5 Appium Readysessionid© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved.53TestScriptDevice ClusterDevice1Device2Device31 WebDriver Session Request@ http://appium.testdroid.com/wd/hub/Desired Caps, .apk / .ipa2 Configure project3Wait for device tobecome available4 Start AppiumSessionMap6 WebDriver Session responsesessionid7 Test Runsessionid ProxyAppiumDevice1Example
  • 54. Image RecognitionThe basic example of Image Recognitionin Mobile Game Testing: TemplateMatching© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 54
  • 55. Code Example© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 55
  • 56. Code Example© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 56
  • 57. Code Example© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 57
  • 58. Code Example© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 58
  • 59. Example: Clash of Clans• Test Script contains the following 'steps':1. Identify the used platform (Android / iOS)2. Search for 'Goldmine' graphical element(using find_image to search for screen)3. Script runs test to go to shop4. Script runs test to buy a cannon5. Script runs test to place the cannon (allthese three are defined in .png files)6. Script runs test to start a battle!7. Game is brought down and test ends.© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 59
  • 60. Image Recognition with Real Android andiOS Devices in Local Environment© Copyrights by Bitbar Technologies Ltd. 2014VIDEOAll rights reserved. 60
  • 61. Testdroid Update© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 61
  • 62. New UI in Testdroid Cloud 2.0© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 62
  • 63. Testdroid Cloud's Interactive© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 63
  • 64. TestdroidPrivateCloud -Running user-selectedat any of ourhosting facilities© Copyrights by Bitbar Technologies Ltd. 2014devicesAll rights reserved. 64
  • 65. NEW HOSTING CENTER!Mountain View, CA© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 65San FranciscoandMountain Viewdatacenter
  • 66. Testdroid Deployment Options© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 66Option 1:Public Device Cloudon-demand devices(multitenant)Mobile app testing onover 400+ realAndroid and iOSdevices hosted byBitbarOption 2:Private Device Cloudreserved devicesHosted by Bitbar in theUS and/or EuropeDevices chosen by andreserved only for theCustomerOption 3:On-premise DeviceCloudAutomated mobileapp testing on realAndroid and iOS de-viceshosted by thecustomer, usually30-500 devices
  • 67. © Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 67
  • 68. THANK YOU!W E B I N A RVideo recording & slides of this webinar willbe soon available at Vimeo and Slideshare!Please visit at www.testdroid.com for moreinformation.© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 68
  • 69. We operate the largest global devicecloud with over 400 devicesinstantly available for developersDevices from the US, Europe, China, Japan and Korea– global devices for global customers© Copyrights by Bitbar Technologies Ltd. 2014All rights reserved. 69
  • Comments
    Top