1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
|
<html devsite>
<head>
<title>Camera HAL Testing Checklist</title>
<meta name="project_path" value="/_project.yaml" />
<meta name="book_path" value="/_book.yaml" />
</head>
<body>
<!--
Copyright 2017 The Android Open Source Project
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
{% include "_versions.html" %}
<p>This document lists all tests available for evaluating the Android camera
hardware abstraction layer (HAL). It is intended for original equipment
manufacturers (OEMs) and application
processor (AP) vendors so they may ensure proper implementation of the camera
HAL with minimum defects. Although this is a voluntary addition to the Android
Compatibility Test Suite (CTS), it greatly increases camera test coverage and
will certainly identify potential bugs.</p>
<p>By passing these tests, OEMs validate
whether they have properly integrated the Android
camera hardware
abstraction layer (HAL) 3 interfaces. When conforming with all items in the
checklist, a device implementation may be considered <em>full</em> with respect
to the Android Camera HAL interfaces. This will in turn enable a device to
properly support the
<a href="https://developer.android.com/reference/android/hardware/camera2/package-summary" class="external">
<code>android.hardware.camera2</code></a> package that
camera apps build upon.</p>
<h2 id=camera_hal_3_2_specification>Camera HAL3 specification</h2>
<p>The <a href="/devices/camera/camera3">Android Camera HAL3</a> specification
is the authoritative source of
information on what devices must satisfy; this page provides a summary
of all tests that can be used as a checklist. Camera HAL implementers (e.g. AP
vendors) should go through the Camera HAL3 specification line-by-line and ensure
their devices conform to it.</p>
<p>The current HAL specification is defined in these files within the Android
5.0 and later generic Android Platform Development Kit (PDK):</p>
<ul>
<li><em>Camera HAL 3.x interface and spec</em>: <code><a
href="https://android.googlesource.com/platform/hardware/libhardware/+/master/include/hardware/camera3.h"
class="external">hardware/libhardware/include/hardware/camera3.h</a></code>,
<code><a
href="https://android.googlesource.com/platform/hardware/libhardware/+/master/include/hardware/camera_common.h"
class="external">hardware/libhardware/include/hardware/camera_common.h</a></code>
<li><em>Camera HAL 3.x metadata spec</em>: <code><a
href="https://android.googlesource.com/platform/system/media/+/master/camera/docs/docs.html"
class="external">system/media/camera/docs/docs.html</a></code>
<li><em>HAL pixel format interface and spec</em>: <code><a
href="https://android.googlesource.com/platform/system/core/+/master/libsystem/include/system/graphics.h"
class="external">system/core/libsystem/include/system/graphics.h</a></code>
</ul>
<h2 id=camera_test_types>Camera test types</h2>
<p>Here are the primary types of tests available for the latest Android camera
along with references to associated instructions below:</p>
<ul>
<li><em><a href="#vendor_test_suite">Vendor Test Suite (VTS)</a>:</em> Tests that directly test the
camera HAL interface
<li><em><a href="#cts_tests">Compatibility Test Suite (CTS)</a></em>:
Standard, automated Android
tests to ensure device compatibility - see the
<a href="/compatibility/cts/index.html">CTS introduction</a> and the <a
href="/devices/tech/test_infra/tradefed/index.html">Trade Federation
Overview</a>
<li><em><a href="#its_tests">Image Test Suite (ITS)</a>:</em> Manually run
tests to ensure image correctness - see the top-level and test-specific
<code>README</code> files and <code>tutorial.py</code> for setup instructions
<li><em><a
href="#manual_testingcam_tests">Manual TestingCam tests</a>:</em>
Run from the source in <code>pdk/apps/TestingCamera/</code>
<li><em><a href="#manual_testingcam2_tests">Manual TestingCam2.1
tests</a>:</em> Run from the source in <code>pdk/apps/TestingCamera2/</code>
</ul>
<p>All of these test types are described in detail below. These tests are
presented in the chronological order in which OEMs are expected to execute
them.</p>
<p>For instance, if a device fails the native tests, it will assuredly fail the
subsequent Compatibility Test Suite (CTS) tests. And if a device fails CTS,
there is little use in proceeding to the Image Test Suite (ITS). We recommend
addressing failures in each test type before proceeding to the next set of
tests.</p>
<h2 id=vendor_test_suite>Vendor Test Suite (VTS) tests</h2>
<p>The Android Vendor Test Suite (VTS) is a testing suite that works on the
HIDL interface level. For more information on using VTS, see
<a href="/compatibility/vts/">Vendor Test Suite</a>.</p>
<h2 id=cts_tests>Compatibility Test Suite (CTS) tests</h2>
<p>Camera Android Compatibility Test Suite (CTS) tests focus upon device
compatibility. For more information on the recommended test environment for camera CTS, see <a href="/compatibility/cts/setup#camera">Set up CTS</a>.</p>
<p>The starting path for Camera CTS tests is: <code>platform/cts</code>.</p>
<p>When running Camera CTS for devices that support external cameras (such as
USB webcams), you must have a device plugged in when running CTS or the tests
will automatically fail. Examples of external cameras include: <a
href="https://www.logitech.com/en-us/product/hd-pro-webcam-c920"
class="external">Logitech HD Pro Webcam C920</a> and the <a
href="https://www.microsoft.com/accessories/en-us/products/webcams/lifecam-hd-3000/t3h-00011"
class="external">Microsoft LifeCam HD-3000</a>.</p>
<p>See the <a href="/compatibility/cts/index.html">CTS
introduction</a> and its subpages for general instructions on running CTS.</p>
<h3 id=cts_tests_for_the_android_hardware_camera_api>CTS tests for
the <code>android.hardware.Camera</code> API</h3>
<p>Find these camera tests under <code>cts/tests/tests/</code>:</p>
<ul>
<li><code>hardware/src/android/hardware/cts/CameraTest.java</code>
<li><code>hardware/src/android/hardware/cts/CameraGLTest.java</code>
<li><code>hardware/src/android/hardware/cts/Camera_SizeTest.java</code>
<li><code>permission/src/android/permission/cts/CameraPermissionTest.java</code>
</ul>
<h3 id=cts_tests_for_the_android_hardware_camera2_api>CTS tests for
the <code>android.hardware.camera2</code> API</h3>
<p>Find these camera tests under <code>cts/tests/tests/</code>:</p>
<ul>
<li><code>hardware/src/android/hardware/camera2/cts/*</code>
<li><code>permission/src/android/permission/cts/Camera2PermissionTest.java</code>
</ul>
<h3 id=cts_verifier_camera_tests>CTS Verifier camera tests</h3>
<p>Find these camera tests under:
<code>cts/apps/CtsVerifier/src/com/android/cts/verifier/camera/*</code></p>
<h2 id=its_tests>Image Test Suite (ITS) tests</h2>
<aside class="Note"><strong>Note:</strong>
<a href="/devices/camera/camera3">Camera HAL3</a> is strongly recommended for
devices running Android {{ androidPVersionNumber }} or higher as Camera HAL1
has been deprecated. This
also allows for a smooth phase out of the deprecated
<a href="https://developer.android.com/reference/android/hardware/Camera" class="external">
Camera API</a>.</aside>
<p>The Camera Image Test Suite (ITS) tests focus on image correctness.
To perform the tests, run the Python scripts on a workstation with the
Android device connected over USB.</p>
<p>The Camera ITS infrastructure and tests are located in the
<a href="https://android.googlesource.com/platform/cts/+/master/apps/CameraITS"
class="external">
<code>cts/apps/CameraITS</code></a> directory. Each test resides in a
<code>tests/scene<var>#</var></code> subdirectory.</p>
To set up the test environment, run:
<pre class="devsite-click-to-copy">
<code class="devsite-terminal">extract root/out/host/linux-x86/cts-verfier/android-cts-verifier.zip</code>
<code class="devsite-terminal">cd android-cts-verifier</code>
<code class="devsite-terminal">adb install -r CtsVerifier.apk</code>
<code class="devsite-terminal">cd CameraITS</code>
<code class="devsite-terminal">source build/envsetup.sh</code>
</pre>
<aside class="note"><strong>Note:</strong> Because ITS is a CTS Verifier subtest,
start CTS Verifier and the ITS subtest before running the Python scripts so they
have processes with which to communicate.</aside>
<p>For more information on how to set up and run the tests, see the
<code>CameraITS</code> PDF file in <code>cts/apps/CameraITS</code>. See
<a href="https://android.googlesource.com/platform/cts/+/master/apps/CameraITS/tests/tutorial.py"
class="external">
<code>tutorial.py</code></a> in the <code>tests</code> subdirectory for a
guide on how to use the scripts.</p>
<p>The ITS static tests (scenes 0-5) can run with any operating
system with the required Python 2.7 environment. However, the
<code>sensor_fusion</code>
test with the <a href="/compatibility/cts/sensor-fusion-quick-start">Sensor
Fusion Box</a> must be run with the
Linux operating system.
</p>
<p>The recommended setup for scenes 0-4
is described in <a
href="/compatibility/cts/camera-its-box">Camera ITS-in-a-box</a>. The
recommended setup for the sensor_fusion scene is described in <a
href="/compatibility/cts/sensor-fusion-quick-start">Sensor Fusion Box Quick
Start Guide</a>.</p>
<p>To run ITS manually, prepare a simple physical environment with a
specific, reusable target
such as a white wall, a grey card, and a desk lamp. Mount the Android device on
a tripod and run the scripts to test the camera functions. Most tests
are pass or fail but some offer metrics.</p>
<p>These scripts test scenarios that are not tested in CTS and are an important
component of the HAL 3.2 test plan.</p>
<p>ITS tests either pass or fail. All mandated
tests in each scene folder must pass. Tests that are not mandated can
fail and still count as a pass in
<code>CtsVerifier</code>.</p>
<h3 id="scene0_to_scene4_tests">Scene 0 to scene 4 tests</h3>
<p>These scenes represent a large part of ITS testing and are included as PDF
files in the <code>scene</code> folder. To automate these tests, use the <a
href="/compatibility/cts/camera-its-box">Camera ITS-in-a-box</a> system.</p>
<ul>
<li>Scene 0: Requires no setup.</li>
<li>Scene 1: Requires a grey card.</li>
<li>Scene 2: Requires a face scene.</li>
<li>Scene 3: ISO12233 chart.</li>
<li>Scene 4: Custom scene with a circle inside a square.</li>
</ul>
<h3 id="scene5_tests">Scene 5 tests</h3>
<p>Scene 5 tests require a diffuser to be placed on top of the camera.</p>
<h3 id="sensor_fusion_tests">Sensor fusion tests</h3>
<p>Sensor fusion tests require specific camera motion to test the timestamp
difference between the camera and the gyroscope for AR and VR applications. This
test is skipped if no gyroscope is included or if the <code>REALTIME</code>
parameter is not enabled. The <code>sensor_fusion</code> test can be automated
with the <a
href="/compatibility/cts/sensor-fusion-quick-start">Sensor Fusion Box</a>.</p>
<h2 id=media_framework_tests>Media Framework tests</h2>
<p>Pass all of the camera-related media tests in MediaFrameworkTest. Please
note, these tests require the mediaframeworktest.apk be installed on the Android
device. You will need to <code>make mediaframeworktest</code> and then use adb
to install the resulting .apk. Example commands are included below.</p>
<p>The starting path for Camera-related media framework tests is:
<code>platform/frameworks/base</code></p>
<p>Find the source code for the tests here:
<code>frameworks/base/media/tests/MediaFrameworkTest</code></p>
<p>To set up these tests:</p>
<pre class="devsite-click-to-copy">
<code class="devsite-terminal">make mediaframeworktest</code>
<code class="devsite-terminal">adb install out/target/product/<var>name</var>/data/app/mediaframeworktest.apk</code>
</pre>
<p>Where the <var>name</var> variable represents the
directory containing the vendor's product.</p>
<p>Find all of the tests in the following directory or its subdirectories:</p>
<pre class="devsite-click-to-copy">
frameworks/base/media/tests/MediaFrameworkTest/src/com/android/mediaframeworktest
</pre>
<p>Each subdirectory represents a class of tests:</p>
<ul>
<li><code>functional/</code>
<li><code>integration/</code>
<li><code>performance/</code>
<li><code>power/</code>
<li><code>stress/</code>
<li><code>unit/</code>
</ul>
<h3 id=running_media_framework_tests>Running Media Framework tests</h3>
<p>To see all of the available tests::</p>
<pre class="devsite-terminal devsite-click-to-copy">
adb shell pm list instrumentation
</pre>
<p>This will yield results resembling:</p>
<pre class="devsite-click-to-copy">
instrumentation:com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner
(target=com.android.mediaframeworktest)
instrumentation:com.android.mediaframeworktest/.MediaRecorderStressTestRunner
(target=com.android.mediaframeworktest)
instrumentation:com.android.mediaframeworktest/.MediaFrameworkPerfTestRunner
(target=com.android.mediaframeworktest)
instrumentation:com.android.mediaframeworktest/.MediaFrameworkPowerTestRunner
(target=com.android.mediaframeworktest)
</pre>
<p>Identify and extract the component (between <code>instrumentation:</code>
and <code>(target=com.android.mediaframeworktest) </code>from each test line.
The component is composed of the target package name
(<code>com.android.mediaframeworktest</code>) and the test runner name
(<code>MediaFramework<type>TestRunner</code>).</p>
<p>For instance:</p>
<pre class="devsite-click-to-copy">
com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner
com.android.mediaframeworktest/.MediaRecorderStressTestRunner
com.android.mediaframeworktest/.MediaFrameworkPerfTestRunner
com.android.mediaframeworktest/.MediaFrameworkPowerTestRunner
</pre>
<p>You may then pass each component to <code>adb shell am instrument</code> like
so:</p>
<pre class="devsite-terminal devsite-click-to-copy">
adb shell am instrument -w <var>component.name</var>
</pre>
<p>Where <code><var>component.name</var></code> equals the extracted value
above. For
example:</p>
<pre class="devsite-terminal devsite-click-to-copy">
adb shell am instrument -w com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner
</pre>
<p>Please note, while the class path is the Java package + class name, the
instrumentation package isn't necessarily the same as the Java package. Make
sure you use the AndroidManifest.xml package when concatenating the component
name, not the Java package in which the test runner class resides.</p>
<p>To run a single class of tests, pass the -e class <test-class> argument, like
so:</p>
<pre class="devsite-terminal devsite-click-to-copy">
adb shell am instrument -e class com.android.mediaframeworktest.integration.CameraBinderTest -w com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner
</pre>
<p>To run only a single method in a test class, append a pound (#) sign and the
method name (in this case, <code>testConnectPro</code>) to the class name, like so:</p>
<pre class="devsite-terminal devsite-click-to-copy">
adb shell am instrument -e class 'com.android.mediaframeworktest.integration.CameraBinderTest#testConnectPro' -w com.android.mediaframeworktest/.MediaFrameworkIntegrationTestRunner
</pre>
<h3 id=media_settings_functional_tests>Media settings functional tests</h3>
<p>Here is an example run of a functional test. This test verifies the basic
functionality of different combinations of camera settings. (ie, Flash,
exposure, WB, scene, picture size and geoTag)</p>
<p>Run the test command:</p>
<pre class="devsite-terminal devsite-click-to-copy">
adb shell am instrument -w -r -e delay_msec 15 -e log true -e class com.android.mediaframeworktest.functional.camera.CameraPairwiseTest com.android.mediaframeworktest/com.android.mediaframeworktest.CameraStressTestRunner
</pre>
<h3 id=media_integration_tests>Media integration tests</h3>
<p>Here is an example run of an integration test, in this case
mediaframeworktest/integration/CameraBinderTest.java and
mediaframeworktest/CameraStressTestRunner.java:</p>
<pre class="devsite-terminal devsite-click-to-copy">
adb shell am instrument -e class \ 'com.android.mediaframeworktest.<strong>integration</strong>.<strong>CameraBinderTest'</strong> -w \ 'com.android.mediaframeworktest/.<strong>CameraStressTestRunner'</strong>
</pre>
<p>If successful, this results in output resembling:</p>
<pre class="devsite-click-to-copy">
-----
com.android.mediaframeworktest.integration.CameraBinderTest:...........
Test results for CameraStressTestRunner=...........
Time: 3.328
OK (11 tests)
-----
</pre>
<h3 id=media_performance_tests>Media performance tests</h3>
<p>This preview memory test will open and release the camera preview for 200
times. In each 20 iterations, the snapshot of ps mediaserver will be recorded
and it will compare the memory usage different after 200 iterations. Test will
fail If the difference is greater than 150kM.</p>
<p>Run the test command:</p>
<pre class="devsite-terminal devsite-click-to-copy">
adb shell am instrument -w -r -e class com.android.mediaframeworktest.performance.MediaPlayerPerformance#testCameraPreviewMemoryUsage com.android.mediaframeworktest/.MediaFrameworkPerfTestRunner
</pre>
<p>More detailed output can be found in:
<code>/sdcard/mediaMemOutput.txt</code></p>
<h3 id=media_unit_tests>Media unit tests</h3>
<p>The commands to run unit tests are all similar. For example, for
CameraMetadataTest.java, the command would be:</p>
<pre class="devsite-terminal devsite-click-to-copy">
adb shell am instrument -e class 'com.android.mediaframeworktest.unit.CameraMetadataTest' -w 'com.android.mediaframeworktest/.CameraStressTestRunner'
</pre>
<h3 id=media_stress_tests>Media stress tests</h3>
<p>This test is to stress out the camera image capture and video recording.</p>
<p>Run the test command:</p>
<pre class="devsite-terminal devsite-click-to-copy">
adb shell am instrument -w com.google.android.camera.tests/com.android.camera.stress.CameraStressTestRunner
</pre>
<p>All tests should pass.</p>
<h2 id=manual_testingcam_tests>Manual TestingCam tests</h2>
<p>The TestingCam app should be run manually with the following checks performed.
The source for TestingCam is here: <code>pdk/apps/TestingCamera/</code></p>
<h3 id=infinity_focus_with_camera_tilt>Infinity focus with camera tilt</h3>
<p>Start TestingCam, turn on preview, and ensure that autofocus mode is set to
infinity. Using the <strong>Take picture</strong> button, capture shots of
distant subjects (at least 10m away) with the camera pointed horizontally,
upwards (close to vertical), and downwards (close to vertical); an example of
the upwards shot could be high leaves/branches of a tree from beneath and an
example of the downwards shot could be the street as seen from the roof of a
building. In all cases, the distant subject should be sharp and in focus. Save
and view the shots in the gallery view so that you can zoom in and inspect the
sharpness more easily.</p>
<p>Note that for a camera with a VCM actuator to pass this test, it will require
either a closed-loop AF control system, or it will need some sort of SW
correction based on using accelerometer data to determine camera orientation.
Reliable factory calibration of the lens infinity position will also be needed.</p>
<h2 id=manual_testingcam2_tests>Manual TestingCam2 tests</h2>
<p>The TestingCam2 app should be run manually, with the following checks
performed. The source for TestingCam2 is here: <code>pdk/apps/TestingCamera2/</code></p>
<h3 id=9_1_jpeg_capture>JPEG capture</h3>
<p>Start TestingCam2, and press the <strong>JPEG</strong> button. The image
that appears to the right of the viewfinder image should appear the same as the
viewfinder, including having the same orientation.</p>
</body>
</html>
|
