Xamarin Test Cloud can help us identify applications’ functionality-related issues on real devices. It is a great source of application monitoring in terms of testing on different mobile devices and with different versions of operating systems.
Getting a detailed analysis of various applications’ functions is very important to make sure our application is running as expected on our target devices.
With that being said, it is also critical to the application to be able to run on different operating system versions, and to analyze how it performs and how much memory usage it has.
In this mobile DevOps tutorial, we will discuss how to use Xamarin Test Cloud and the analytics after running an application on different sets of devices.
This article is an excerpt from the book, Mobile DevOps, written by Rohin Tak and Jhalak Modi.
We will be using two different applications here to see the monitoring analytics and compare them, to get a better understanding of how this helps us identify various performance and functionality-related issues in our application.
Below are the applications we will be using:
- Xamarin Store
Let’s go through some steps to see how to monitor our PhoneCallApp:
- Go to https://testcloud.xamarin.com/.
- Click on the PhoneCallApp icon to get to the details of the test runs:
- On the next page, you’ll see a list of tests run for the application:
- Now, because we have only run one test so far, Test Cloud does not provide us with the graphical metrics shown in the preceding screenshot. In other examples we’ll see next, you’ll be able to see a more detailed comparison of different test runs.
- Click on the test run from the list to see its results:
- The test run listed is the one we ran earlier in previous chapters and uploaded from our machine to Xamarin Test Cloud using the command line.
- To get an idea of this interface, let’s have a look at different parts of Xamarin Test Cloud’s interface.
- Now, this is an overview screen that shows a summary of all the tests run for this application:
- This screen shows summary details, such as how many tests failed from the total number of tests run, how many times the app ran on a device, how many devices these tests were run on, and much more.
- This screen is very useful to get a brief idea when you want to get a report on how your application is doing on different devices and OS versions.
- The next thing you’ll see in the left pane is the list of UITests included in the test run:
- This screen basically has a list of all the Xamarin.UITests that you included in your project. You can click on these different tests to see their respective results on the right side of the screen.
- Let’s click on the test from the list in the preceding screen.
- This will take us to the next screen, which has detailed reports for the test run:
- Have a close look at the left pane on this screen.
- It gives us some steps of the test run on the device.
- These steps are only what we had written previously in the code to take a screenshot of every activity the test does.
- The steps are as mentioned (we are using the screens of the test code written in previous chapters here):
- App started: Take a screenshot when the app starts; this was written in the BeforeEachTest() method in the Tests.cs file:
- Call button pressed: This step is when the Xamarin.UITest presses the call button to make a call:
- Failed step (the assert): This is the last step and is shown to provide proof of the failed step, so you can see the outcome that we received and compare it with what was expected. This was the final assert that decides whether the test passes or not, based on the outcome in the Assert.IsTrue() condition.
- You can click on each of these steps in the left pane and analyze the screenshots taken to see exactly what went on during the test. This is a great way to see exactly what went wrong when the test failed.
- Now, sometimes the screenshots are not enough to identify the issue. For a more detailed analysis, Test Cloud also provides us with Device Log, as shown in the following screenshot:
- Device logs are a great way to see what’s going on under the hood and get more detailed information about the application’s behavior and how the device itself behaves when the application is run on it.
- This can help pinpoint the issues when a test fails on the device; logs are always a savior in that sort of scenario.
- Click on the Device Log and you can see step-by-step logs for each screenshot on the same screen:
- When a test fails, Test Cloud provides us with one more option, to see the Test Failures:
- It’s very useful for automated test developers to see the exception information when a test fails.
- Last but not least, there is also a Test Log option, which can be used to get a consolidated log of the entire test run:
Xamarin Store app
Now that we have seen different options provided by Test Cloud to monitor our application and its functionality using test runs, let’s see how the dashboard and tests look when we have multiple test runs on various physical devices with different OS versions.
This will give us a better idea of how comparative monitoring can be done on Test Cloud to analyze an application’s behavior on different devices, and compare them with one another.
The Xamarin Store application is a sample application provided by Test Cloud on its platform to help understand the platform and get an idea of the dashboard. Let’s go through the steps to understand how to monitor your application running on multiple devices, and how to compare different test runs:
- Go to the Test Cloud home page, just like in the previous example, and click on the Xamarin Store icon:
- On the next screen, you’ll see a graphical representation of different test runs and brief information about how many tests failed of the total tests run, what the application size is, and its peak memory usage information during different test runs:
- This gives us a nice comparative look at how our application is performing on different test runs. It is possible that the application was performing fine during the first run, and then some code changes made some functionality fail. So, this graph is very useful to monitor a timeline of changes that affected application functionality.
- You can further click on the graph or the test run to see an overview of it.
- Now, this screen gives us a great view of how an application running on different devices can be monitored. It’s a very nice way to keep track of the application on different devices and OS versions:
- Let’s click on one of the steps to see the results of the step on multiple devices:
- The red icon shows failed tests. This page shows all the devices you chose to run the test on; it shows all the devices the test passed on, and shows a red flag on failed devices.
- You can further click on each device to get device-specific screens and logs.
To summarize, we performed API monitoring efficiently using Xamarin Test Cloud.
If you found this post useful, do check out the book Mobile DevOps, to deliver continuous integration and delivery for Mobile applications.