Android platform testing

Android Open Source Project (AOSP) provides several tools and test suites for testing various parts of your implementation. Before using the pages in this section, you should be familiar with the following terms:

Android-compatible device
A device that can run any third-party app written by third-party developers using the Android SDK and NDK. Android-compatible devices must adhere to the requirements of the Compatibility Definition Document (CDD) and pass the Compatibility Test Suite (CTS). Android-compatible devices are eligible to participate in the Android ecosystem, which includes potential licensure of Google Play, potential licensure of the Google Mobile Services (GMS) suite of app and APIs, and use of the Android trademark. Anyone is welcome to use the Android source code, but to be considered part of the Android ecosystem, a device must be Android compatible.
artifact
A build-related log that enables local troubleshooting.
Compatibility Definition Document (CDD)
A document that enumerates the software and hardware requirements for an Android-compatible device.
Compatibility Test Suite (CTS)

A free, commercial-grade test suite, available for download as a binary or as source in AOSP. The CTS is a set of unit tests designed to be integrated into your daily workflow. The intent of CTS is to reveal incompatibilities, and ensure that the software remains compatible throughout the development process.

CTS and platform tests aren't mutually exclusive. Here are some general guidelines:

  • If a test is asserting correctness of framework API functions or behaviors, and the test should be enforced across OEM partners, it should be in CTS.
  • If a test is intended to catch regressions during platform development, and might require privileged permission to carry out, and might be dependent on implementation details (as released in AOSP), it should be a platform test.
Google Mobile Services (GMS)

A collection of Google apps and APIs that can be preinstalled on devices.

GoogleTest (GTest)

A C++ testing and mocking framework. GTest binaries typically access lower-level abstraction layers or perform raw IPC against various system services. The testing approach for GTest is usually tightly coupled with the service being tested. CTS contains the GTest framework.

instrumentation test

A special test execution environment launched by the am instrument command, where the targeted app process is restarted and initialized with basic app context, and an instrumentation thread is started inside the app process virtual machine. CTS contains instrumentation tests.

Logcat

A command-line tool that creates a log of system messages, including stack traces of when the device throws an error and messages that you have written from your app with the Log class.

logging

Using a log to keep track of computer system events, such as errors. Logging in Android is complex due to the mix of standards used that are combined in the Logcat tool.

postsubmit test

An Android test that is performed when a new patch is committed to a common kernel branch. By entering aosp_kernel as a partial branch name, you can see a list of kernel branches with results available. For example, results for android-mainline can be found at https://ci.android.com/builds/branches/aosp_kernel-common-android-mainline/grid.

presubmit test

A test used to prevent failures from being introduced into the common kernels.

Trade Federation

Also called Tradefed, a continuous test framework designed for running tests on Android devices. For example, Tradefed is used to run Compatibility Test Suite and Vendor Test Suite tests.

Vendor Test Suite (VTS)

A set of extensive capabilities for Android testing, promoting a test-driven development process, and automating hardware abstraction layer (HAL) and OS kernel testing.

Platform test types

A platform test typically interacts with one or more of the Android system services or HAL layers, exercises the functionalities of the subject under test, and asserts correctness of the testing outcome. A platform test might:

  • (Type 1) Exercise framework APIs using Android framework. Specific APIs being exercised can include:
    • Public APIs intended for third-party apps
    • Hidden APIs intended for privileged apps, namely system APIs or private APIs (@hide, or protected, package private)
  • (Type 2) Invoke Android system services using raw binder or IPC proxies directly.
  • (Type 3) Interact directly with HALs using low-level APIs or IPC interfaces.

Type 1 and 2 tests are typically instrumentation tests, while type 3 tests are usually GTests.

What's next?

Here is a list of documents that you can read for more detailed information: