Every API call to MOZARK platform is authorized using an api_access_token created by a user as part of login() API. After you create a user account(or MOZARK provisions administrator account for you), you configure API URL, user credentials, and client ID in the config file as mentioned in the SDK Configuration section below.
- Getting Started
- Table of contents
Configure API URL, user credentials, and client ID.
Create a configuration file with a name config under .mozark directory within your home folder. The template of the configuration file is as below:
[default]
MOZARK_APP_TESTING_URL={url}
MOZARK_APP_TESTING_USERNAME={username}
MOZARK_APP_TESTING_PASSWORD={password}
MOZARK_APP_TESTING_CLIENTID={client_id}
BASE_DOWNLOAD_DIR={local_download_base_dir}
The login() api reads the information mentioned in $HOME/.mozark/config and sets the api_access_token to be further used in subsequent APIs.
Example:
from mozark_sdk.client import Client
client = Client()
client.login()Example:
from mozark_sdk.client import Client
client = Client()
client.logout()User can organize the application files, test application/code files, test executions, test outcomes etc. within a project.
In order to create a project user should pass a unique project name along with an optional project description.
| Property | Description |
|---|---|
| project_name | Unique project name(mandatory) |
| project_description | A short note about a project(optional) |
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
project_name = "Android App Performance Testing"
project_description = "This project evaluate the quality of application experience in real user scenarios"
project_list = client.create_project(project_name=project_name, project_description=project_description)To get project information of already created project,
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
project_name = "Android App Performance Testing"
project_description = "This project evaluate the quality of application experience in real user scenarios"
project_info = client.get_project_info(project_name=project_name)
project_name = project_info["projectName"]
project_id = project_info["projectUUID"]get_project_info() returns unique project identifier projectUUID
from mozark_sdk.client import Client
client = Client()
client.login()
project_name = "Android App Performance Testing"
response = client.get_project_info(project_name=project_name)Returns a list of all projects.
from mozark_sdk.client import Client
client = Client()
client.login()
project_name = "Android App Performance Testing"
project_list = client.get_project_list()Upload android or ios application(.apk or .ipa) from given file path.
| Property | Description |
|---|---|
| file_category | File category (mandatory) |
| project_name | Unique project name(mandatory) |
| file_path | Relative or absolute file path(mandatory) |
List of supported values for a file_catogory
android-application-> For an android application packageios-application-> For an ios application package
upload_application() returns:
| Message | Description |
|---|---|
Success: File file_name uploaded successfully. |
Success |
Failure: File file_name not uploaded. |
Failure |
Error: File "file_name already exists. |
Failure |
NOTE
In order to avoid duplicate file uploads, MOZARK compares the md5 sum of already uploaded file with the file being uploaded now. Additionally, MOZARK uses file name to uniquely identify files, which can be referred by the user.
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
project_name = "Android App Performance Testing"
file_path = "./MyApplication-1.0.apk"
file_category = 'android-application'
message = client.upload_application(file_category=file_category,
project_name=project_name,
file_path=file_path)User can get information about the application file uploaded by passing the file_name.
get_application_info() returns a file information dictionary.
| Property | Description |
|---|---|
| fileName | Name of the file |
| fileCategory | android-application or ios-application |
| md5 | md5 checksum of a file |
| fileURL | http url of a file |
| fileUUID | unique identifier |
| packageName | Android package name(for android-application only) |
get_application_info() returns an error message if file with a given file name is not found.
| Message | Description |
|---|---|
Failure: File with name file_name not found. |
Failure |
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
file_name = "MyApplication-1.0.apk"
message = client.get_application_info(file_name=file_name)User can delete the application file uploaded by passing the file_name.
delete_application() returns:
| Message | Description |
|---|---|
Success: File file_name deleted successfully. |
Success |
Failure: File file_name not deleted. |
Failure |
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
file_name = "MyApplication-1.0.apk"
message = client.delete_application(file_name=file_name)To get list of all application by giving a file category and an optional project name.
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
file_category = "android-application"
project_name = "Android App Performance Testing"
application_file_info_list = client.get_application_list(file_category=file_category,
project_name=project_name)get_application_list() returns list of application info similar to get_application_info() as described in Application File Info
Upload native android or ios test application(.apk or .ipa) from given file path.
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
project_name = "Android App Performance Testing"
file_path = "./my-experience-test-1.0.apk"
file_category = 'android-test-application'
message = client.upload_native_test_application(file_category=file_category,
project_name=project_name,
file_path=file_path)upload_native_test_application() takes input parameter and returns the message same as mentioned in Upload an application package
List of supported values for a file_catogory
android-test-application- For an android application packageios-test-application- For an ios application package
User can get information about the application file uploaded by passing the file_name.
get_native_test_application_info() returns a file information dictionary as described in Get Application Package Information.
In addition to the common properties, get_native_test_application_info() also returns following information which is specific to the native test application. MOZARK derives these values by processing the uploaded .apk and .ipa files.
These properties are essential for running the test successfully, user must ensure the correctness of these values as per the information specified in their android or ios test code projects.
| Property | Description |
|---|---|
| testCodePackageName | Package name specified in the test code, present in android-test-applcation only |
| testRunnerName | Runner name specified in the test code, present in ios-test-applcation only |
| XCTestRunFileUrl | XCTestRun file bundled with test code, present in ios-test-applcation only |
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
file_name = "my-experience-test-1.0.apk"
message = client.get_native_test_application_info(file_name=file_name)User can delete the test application file uploaded by passing the file_name.
delete_native_test_application() works similar to delete_application() :
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
file_name = "my-experience-test-1.0.apk"
message = client.delete_native_test_application(file_name=file_name)To get list of all test application by giving a file category and an optional project name.
get_native_test_application_list() works similar to the get_application_list().
The properties returned in the file info dictionary are same as described in Get test application package information
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
file_category = "android-test-application"
project_name = "Android App Performance Testing"
test_application_file_info_list = client.get_native_test_application_list(file_category=file_category,
project_name=project_name)User can get list of real devices provisioned by specifying the platform.
List of supported platforms:
| Platform | Description |
|---|---|
| android | Devices installed with Android OS |
| ios | Devices installed with iOS |
| living-room | Living room devices like TVs, which has its own OS. |
get_device_list() returns a list of device information json objects. The properties of this object are described below:
| Property | Description |
|---|---|
| deviceSerial | Unique identifier provided by a device manufacturer |
| deviceBrand | Brand name of the device |
| deviceCity | City in which device is located |
| deviceCountry | Country in which device is located |
| deviceModelName | Model name of the device |
| deviceModelNumber | Model number of the device |
| devicePlatform | android or ios or living-room |
| deviceOSVersion | OS version installed on the device |
| deviceSDKVersion | List of supported SDK versions by device OS |
| deviceUUID | Unique identifier of the device(MOZARK) |
| deviceNetwork | Network to which the device is connected to |
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
device_list = client.get_device_list(platform='android')User can add devices in platform.
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
device_parameter = {
"brand": "Samsung",
"modelName": "Galaxy M20",
"serial": "3401868eba0a16fd",
"platform": "android",
"osVersion": "8.1.0",
"modelNumber": "",
"sdkVersion": [
"31",
"32"
],
"deviceParameters": {
"controllerId": "dubai-mac-1",
"city": "Dubai",
"country": "United Arab Emirates",
"network": "",
"liveLog": "https://{TenantName}}-logs.mozark.ai:10024",
"deviceStatus": "free",
"appiumPort": "4933",
"mjpegServerPort": "1047",
"systemPort": "8224",
"chromedriverPort": "8024"
},
"logParameters": {
"serverIpAddress": "{TenantName}}-logs.mozark.ai",
"serverUserName": "ubuntu",
"port": "10024",
"mitmPort": "8024"
}
}
response = client.add_device(device_parameter=device_parameter)NOTE
In order to execute test on the device, MOZARK APIs only need list of deviceSerial other properties are useful to pick and choose devices from provisioned list.
Device trays is useful feature to allow user to save the list of devices for future use.
User can create a device tray by passing platform, unique tray name, and a list of device serial numbers.
List of supported platforms is same as described in a Get list of all devices.
User can construct the list of device serial numbers from the output of get_device_list().
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
tray_name = "Android 10 Devices in Mumbai"
device_list = ["a1234a", "a1234b", "a1234c"] # use get_device_list() to construct this
message = client.create_tray(platform='android',
tray_name=tray_name,
device_list=device_list)User can get tray information for a given tray name.
get_tray_info() returns json object with a tray name, platform name, along with list of devices added in the tray.
get_tray_info() returns a list of devices under trayDevices key. The information of the devices is same as described in Get list of all devices
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
tray_name = "Android 10 Devices in Mumbai"
tray_info = client.get_tray_info(tray_name=tray_name)User can update the list of devices in a given tray. update_tray() replaces the existing list of devices with a newly passed device list.
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
tray_name = "Android 10 Devices in Mumbai"
updated_device_list = ["a1234x", "a1234y", "a1234z"] # use get_device_list() to construct this
message = client.update_tray(tray_name=tray_name,
device_list=updated_device_list)User can delete a tray by passing a tray name.
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
tray_name = "Android 10 Devices in Mumbai"
message = client.delete_tray(tray_name=tray_name)User can get a list of device trays.
get_tray_list() returns a list of objects containing tray name, tray platform, and tray id.
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
tray_list = client.get_tray_list()MOZARK enables various features to capture information while executing the application user scenario. The access to these features is made flexible by a test configuration and test parameters.
List of supported test configuration and how to set up test parameters is described in below sections.
List of supported test configurations:
| Feature | Key | Platform |
|---|---|---|
| Capture and store test code/application console logs | captureAutomationLogs | all |
| Capture and store test framework output files | captureTestFrameworkResult(*) | all |
| Capture and store device logs while user scenarios is being run | captureSystemDebugLogs | all |
| Capture and store device screenshots while user scenario is being run | captureDeviceScreenShots | all |
| Record and store device screen as a video while user scenario is being run | recordDeviceScreen | android, ios |
| Capture and store HTTP APIs request and response between application and its backend | captureHAR | android |
| Capture and store Device TCP traffic between application and its backend | captureDeviceNetworkPackets(*) | none |
| Capture and store CPU metrics while user scenario is being run | captureCPUMetrics(*) | android, ios |
| Capture and store Memory metrics while user scenario is being run | captureMemoryMetrics(*) | android, ios |
| Capture and store Battery metrics while user scenario is being run | captureBatteryMetrics(*) | android |
| Capture and store Graphics metrics while user scenario is being run | captureGraphicsMetrics(*) | android |
| Record and store TV screen as a video while user scenario is being run on device | recordLRDeviceScreen(*) | living-room |
| Start live streaming of test code/application console logs while user scenario is being run on device | captureLiveLogs | all |
NOTE
For the key marked with (*) and its respective feature is supported from SDK, but the platform support communicated as part of feature release.
The default value to all the above feature keys is set to false. User can set desired test configuration properties to true and pass it while Executing a test or Scheduling tests.
Example:
test_configuration = {
"captureHAR": True,
"captureDeviceScreenShots": True,
"recordDeviceScreen": True,
"captureSystemDebugLogs": True,
"captureLiveLogs": True
}To execute test on set of devices, User need to pass following parameters to start_test_execution() API.
| Parameter | Description |
|---|---|
| project_name | Name of the project |
| test_framework | Application automation testing framework |
| application_file_name | Name of application package file uploaded |
| test_application_file_name | Name of test application package file uploaded |
| devices | List of device serials |
| test_configuration | Test configuration see Test Configuration and Test Parameters |
| test_parameters | Test parameters see Test Configuration and Test Parameters |
List of supported test frameworks:
android-uiautomator- Android native test frameworkios-xcuitest- iOS native test frameworklr-android-uiautomator- Living room device automation uses Android native test framework to automate TV remote control app.
start_test_execution() returns a success message Success: Executed/Scheduled successfully along with the testId if the execution request is submitted successfully.
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
tray_list = client.get_tray_list()
project_name = "Android App Performance Testing"
test_framework = "android-uiautomator"
application_file_name = "MyApplication-1.0.apk"
test_application_file_name = "my-experience-test-1.0.apk"
devices = ["a1234a", "a1234b", "a1234c"]
test_configuration = {
"captureHAR": True,
"captureDeviceScreenShots": True,
"recordDeviceScreen": True,
"captureSystemDebugLogs": True,
"captureLiveLogs": True
}
test_parameters = {
"maxTestDuration": 10
}
response = client.start_test_execution(project_name=project_name,
test_framework="android-uiautomator",
application_file_name=application_file_name,
test_application_file_name=test_application_file_name,
devices=devices,
test_configuration=test_configuration,
test_parameters=test_parameters
)After the test execution request is submitted, user can monitor the test status by checking the test information.
In addition to the parameters passed to start_test_execution(), get_test_info() returns following information:
| Key | Description |
|---|---|
| testStartTime | Date and time when the test execution is actually started |
| testEndTime | Date and time when the test execution is test completed |
| testUUID | Unique ID of the test |
| testStatus | Status of the test |
| testStatusDescription | Sub-status or Error description of a test status |
List of test status and its description:
| Test Status | Description |
|---|---|
| SCHEDULED | Test execution request is submitted but test run is not yet started |
| STARTED | Test run has started |
| COMPLETED | Test run has completed |
| ABORTED | Test run is aborted before it has finished maxTestDuration |
| FAILED(*) | Test run has completed, but all the test cases in the test suite has failed |
NOTE
TODO: Include FAILED test status
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
test_id = "aa1234bb" # testUUID
test_info = client.get_test_info(test_id=test_id)User can abort test execution of a test which is running at the time of abort.
from mozark_sdk.client import Client
client = Client()
client.login()
test_id = "aa1234bb" # testUUID
test_info = client.abort_test_execution(test_id=test_id)User can get list of test runs. get_test_list() returns list of test information objects as described in Get test execution information.
NOTE
Filter options from_date_time and to_date_time will be implemented in next release.
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
test_id = "aa1234bb" # testUUID
test_info = client.get_test_list()Scheduling tests to execute in a recurrent manner could be useful in various scenarios. These scenarios may include but not limited to following:
- Evaluate user experience by running same user scenarios throughout day or at specific time on a day.
- Collect multiple test outcome - app performance metrics, user experience metrics etc. - to make data driven decision for product development.
- Multiple test runs planned within schedule can be deleted easily.
Scheduling recurrent tests is similar to executing test now. In addition to the parameters passed in start_test_execution() (see Execute a test now), following parameters are required.
| Parameter | Description |
|---|---|
| start_date_time | Date and time when schedule will start |
| end_date_time | Date and time when schedule will end |
| interval | Interval(in minutes) between two test runs in a scheduled execution |
interval should be greater than maxTestDuration in test_parameters.
schedule_test_execution() returns a status message along with scheduleUUID and list of test runs after the schedule test execution request is submitted successfully.
testStartDateTimementioned in a test object withintestRunsarray denote actual test start time when the test run is picked up for execution or the test run has completed execution. Alternatively, it denotes expected test start date and time if the test is yet to run(or the date time is in future).
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
tray_list = client.get_tray_list()
project_name = "Android App Performance Testing"
test_framework = "android-uiautomator"
application_file_name = "MyApplication-1.0.apk"
test_application_file_name = "my-experience-test-1.0.apk"
devices = ["a1234a", "a1234b", "a1234c"]
test_configuration = {
"captureHAR": True,
"captureDeviceScreenShots": True,
"recordDeviceScreen": True,
"captureSystemDebugLogs": True,
"captureLiveLogs": True
}
test_parameters = {
"maxTestDuration": 10
}
schedule_start_date_time = "2023-01-03T22:00:00.000+05:30"
schedule_end_date_time = "2023-01-05T23:00:00.000+05:30"
interval = 15
response = client.schedule_test_execution(project_name=project_name,
test_framework="android-uiautomator",
application_file_name=application_file_name,
test_application_file_name=test_application_file_name,
devices=devices,
test_configuration=test_configuration,
test_parameters=test_parameters,
start_date_time=schedule_start_date_time,
end_date_time=schedule_end_date_time,
interval=interval)Submitting request for scheduling test runs will fail if the device slots in existing schedules conflicts with the new request. Getting a schedule information could help choose the free device slots for new schedule.
User can get test execution schedule information by passing schedule_id to get_test_schedule_info(). This API returns common parameters used while scheduling the test execution and a test specific parameters as specified below:
Schedule information:
{
"scheduleUUID": "",
"scheduleStartTime": "",
"scheduleEndTime": "",
"testInterval": "",
"testConfiguration": {},
"testParameters": {},
"projectName" : "",
"testFramework": "android-uiautomator | ios-xcuitest | living-room-automate",
"applicationFileName": "",
"testApplicationFileName": ""
}Test specific information
{
"device": "",
"testStartTime": "",
"testEndTime": "",
"testUUID": "",
"testStatus": "SCHEDULED | STARTED | COMPLETED | ABORTED | FAILED",
"testStatusDescription": ""
}Example:
from mozark_sdk.client import Client
client = Client()
client.login()
schedule_id = "aa1234bb" # scheduleUUID
schedule_info = client.get_test_schedule_info(schedule_id=schedule_id)User can delete a test schedule by passing a schedule_id passed in delete_test_schedule().
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
schedule_id = "aa1234bb" # scheduleUUID
response = client.delete_test_schedule(schedule_id=schedule_id)Deleting a test execution schedule deletes all the test runs which are yet to start execution at the time of schedule deletion.
User can get list of all schedules, the get_test_schedule_list() returns list of schedule information same as described in Get test schedule information.
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
schedule_info = client.get_test_schedule_list()User can test parameter in all test present in schedule by passing a schedule_id and update_data passed in update_schedule().
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
schedule_id = "f0abd6ea-83db-46ca-99b3-d5bcb6931d6c" # scheduleUUID
update_data = {
"testParameters": {
"packageName": "some name",
"custom key": "custom value"
}
}
response = client.update_schedule(data=update_data,schedule_id=schedule_id)User can update test parameter in a single test by passing a test_id and update_data passed in update_test().
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
test_id = "a1c2e491-78be-47a1-aed8-7ca242b6a579" # testUUID
update_data = {
"testParameters": {
"packageName": "some name",
"custom key": "custom value"
}
}
response = client.update_test(data=update_data,test_id=test_id)NOTE
Filter options from_date_time and to_date_time will be implemented in next release.
Based on the test configuration, test outcome has various sections to display useful information about a test run.
Basic information about the test run includes:
| Key | Description |
|---|---|
| testUUID | Unique test ID |
| testStartDateTime | Start date time of a test run |
| testEndDateTime | Date time when the test run was completed |
| projectName | Project name |
| applicationFileName | Uploaded application file name |
| testApplicationFileName | Uploaded test application file name |
| deviceSerial | Device serial |
| deviceMake | Make of a device |
| deviceModel | Model of a device |
| deviceCity | City where device is provisioned |
| deviceCountry | Country in which device is provisioned |
| deviceNetwork | Network which is enabled during test execution |
| devicePlatform | android, ios, or living-room |
| deviceOSVersion | Version of OS installed in a device |
| deviceNetworkOperator | Name of a network operator chosen while executing a test |
| testStatus | SCHEDULED or STARTED or COMPLETED or ABORTED or FAILED |
| testStatusDescription | Test status description |
| testCasesTotal | Total number of test cases executed |
| testCasesPassed | Total number of test cases passed |
| testCasesFailed | Total number of test cases failed |
| Key | Description |
|---|---|
| testCaseName | test case name |
| testCaseResult | PASS or FAIL |
| testCaseStartDateTime(*) | Date and time when the test code flow entered the test case |
| testCaseEndDateTime(*) | Date and time when the test code flow exited the test case |
NOTE
testCaseStartDateTime and testCaseEndDateTime can be captured if the test framework support.
User experience KPIs are derived from the events captured during the automated user journey from the test code. These are captured within a scope of a test case.
userExperienceKpis section list all the KPIs which can be calculated based on events captured within a test case.
| Key | Description |
|---|---|
| kpiName | Name of the KPI |
| kpiValue | Value of KPI |
| testCaseName | Test case name within the events for this KPI were captured |
User can capture and send events during the user journey within test code. These events are further used to send notification or calculate user experience KPIs.
(see How to capture events from the test code)
This section lists all the events captured during the test run. It includes:
| Key | Description |
|---|---|
| eventName | Name of the event |
| eventDateTime | Date and time at which the event is captured |
| testCaseName | Test case name within this event is captured |
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
test_id = "aa1234bb" # testUUID
response = client.get_test_execution_info_full(test_id=test_id)User can get list of all test execution information by passing from_date_time and to_date_time to get_test_list(). This API returns list of test information same as described in Execute a test now.
Example:
from mozark_sdk.client import Client
import datetime
client = Client()
client.login()
from_date_time = datetime(2023, 1, 7, 0, 0, 0)
to_date_time = datetime(2023, 1, 8, 0, 0, 0)
test_info_list = client.get_test_list(from_date_time=from_date_time,to_date_time=to_date_time)| Section name | Description | Response/File Type |
|---|---|---|
| basic_test_info | same as Basic test information | json |
| test_configuration | Same as Test configuration | json |
| test_cases | Same as Test cases | json |
| events | Same as Events | json |
| kpis_user_experience | Same as User experience KPIs | json |
| kpis_api_performance_http | Same as User experience KPIs | json |
| files_device_screenshots | List of File URLs - same as File URL | png |
| files_device_screen_record | Same as File URL | mp4 |
| files_har | Same as File URL | har |
| files_device_cpu_metrics | Same as File URL | csv |
| files_device_memory_metrics | Same as File URL | csv |
| files_device_battery_metrics | Same as File URL | csv |
| files_device_graphics_metrics | Same as File URL | csv |
| files_device_network_packets | Same as File URL | pcap |
| files_device_debug_logs | Same as File URL | log |
| files_test_execution_output | Same as File URL | log |
| files_test_framework_output | Same as File URL | log or xml or json or html |
| kpis_system_performance_cpu_metrics | Same as User experience KPIs | json |
| kpis_system_performance_memory_metrics | Same as User experience KPIs | json |
| kpis_system_performance_battery_metrics | Same as User experience KPIs | json |
| kpis_app_performance_graphics_metrics | Same as User experience KPIs | json |
Structure is as below:
{
"fileURL" : ""
}Example:
from mozark_sdk.client import Client
client = Client()
client.login()
test_id = "aa1234bb" # testUUID
section = "basic_test_info"
response = client.get_test_execution_info_by_section(test_id=test_id,section=section)User can download test analytics information for various sections in the format specified in Section information
Example:
from mozark_sdk.client import Client
client = Client()
client.login()
test_id = "aa1234bb" # testUUID
section = "basic_test_info"
response = client.download_by_section(test_id=test_id,section=section)