1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
|
<html devsite>
<head>
<title>End-to-End Test Example</title>
<meta name="project_path" value="/_project.yaml" />
<meta name="book_path" value="/_book.yaml" />
</head>
<body>
<!--
Copyright 2017 The Android Open Source Project
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
-->
<p>This tutorial guides you through creating a "hello world" Trade Federation
(TF) test configuration and gives you a hands-on introduction to the TF
framework. Starting from a development environment, you will create a simple
configuration and add features.</p>
<p>The tutorial presents the test development process as a set of exercises,
each consisting of several steps, that demonstrate how to build and gradually
refine your configuration. All sample code you need to complete the test
configuration is provided, and the title of each exercise is annotated with a
letter describing the roles involved in that step:</p>
<ul>
<li><strong>D</strong> for Developer</li>
<li><strong>I</strong> for Integrator</li>
<li><strong>R</strong> for Test Runner</li>
</ul>
<p>After completing the tutorial, you will have a functioning TF configuration
and understand many important concepts in the TF framework.</p>
<h2 id="setup">Setting up the Trade Federation development environment</h2>
<p>For details on seting up the TF development environment, see
<a href="/devices/tech/test_infra/tradefed/fundamentals/machine_setup.html">Machine
Setup</a>. The rest of this tutorial assumes you have a shell open that has been
initialized to the TF environment.</p>
<p>For simplicity, this tutorial illustrates adding a configuration and its
classes to the TF framework core library. This can be extended to developing
modules outside the source tree by compiling the tradefed JAR, then compiling
your modules against that JAR.</p>
<h2 id="testclass">Creating a test class (D)</h2>
<p>Lets create a hello world test that just dumps a message to stdout. A
tradefed test generally implements the
<a href="/reference/com/android/tradefed/testtype/IRemoteTest.html">IRemoteTest</a>
interface. Here's an implementation for the HelloWorldTest:</p>
<pre class="prettyprint">
package com.android.tradefed.example;
import com.android.tradefed.device.DeviceNotAvailableException;
import com.android.tradefed.result.ITestInvocationListener;
import com.android.tradefed.testtype.IRemoteTest;
public class HelloWorldTest implements IRemoteTest {
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
System.out.println("Hello, TF World!");
}
}
</pre>
<p>Save this sample code to
<code><tree>/tools/tradefederation/core/prod-tests/src/com/android/tradefed/example/HelloWorldTest.java</code>
and rebuild tradefed from your shell:</p>
<pre class="devsite-terminal devsite-click-to-copy">
m -jN
</pre>
<p>Note that <code>System.out</code> in the example above may not actually
direct output to the console. While this is acceptable for this test example,
you should establish logging in Trade Federation as described in <a
href="#logging">Logging (D, I, R)</a>.</p>
<p>If the build does not succeed, consult
<a href="/devices/tech/test_infra/tradefed/fundamentals/machine_setup.html">Machine
Setup</a> to ensure you didn't miss a step.</p>
<h2 id="createconfig">Creating a Configuration (I)</h2>
<p>Trade Federation tests are made executable by creating a
<strong>Configuration</strong>, an XML file that instructs tradefed on which
test (or tests) to run, as well as which other modules to execute and in what
order.</p>
<p>Lets create a new Configuration for our HelloWorldTest (note the full class
name of the HelloWorldTest):</p>
<pre class="prettyprint">
<configuration description="Runs the hello world test">
<test class="com.android.tradefed.example.HelloWorldTest" />
</configuration></pre>
<p>Save this data to a <code>helloworld.xml</code> file anywhere on your local
filesystem (e.g. <code>/tmp/helloworld.xml</code>). TF will parse the
Configuration XML file (aka <b>config</b>), load the specified class using
reflection, instantiate it, cast it to a <code>IRemoteTest</code>, and call its
<code>run</code> method.</p>
<h2 id="runconfig">Running the config (R)</h2>
<p>From your shell, launch the tradefed console:</p>
<pre class="devsite-terminal devsite-click-to-copy">
tradefed.sh
</pre>
<p>Ensure a device is connected to the host machine and is visible to tradefed:</p>
<pre class="devsite-click-to-copy">
tf> list devices
Serial State Product Variant Build Battery
004ad9880810a548 Available mako mako JDQ39 100
</pre>
<p>Configurations can be executed using the <code>run <config></code>
console command. Try:</p>
<pre class="devsite-click-to-copy">
tf> run /tmp/helloworld.xml
05-12 13:19:36 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World!
</pre>
<p>You should see "Hello, TF World!" output on the terminal.</p>
<h2 id="addconfig">Adding the config to the Classpath (D, I, R)</h2>
<p>For convenience of deployment, you can also bundle configs into the tradefed
JARs themselves. Tradefed automatically recognizes all configurations placed in
<em>config</em> folders on the classpath.</p>
<p>To illustrate, move the <code>helloworld.xml</code> file into the tradefed
core library
(<code><tree>/tools/tradefederation/core/prod-tests/res/config/example/helloworld.xml</code>).
Rebuild tradefed, restart the tradefed console, then ask tradefed to display the
list of configurations from the classpath:</p>
<pre class="devsite-click-to-copy">
tf> list configs
[…]
example/helloworld: Runs the hello world test
</pre>
<p>You can now run the helloworld config using:</p>
<pre class="devsite-click-to-copy">
tf> run example/helloworld
05-12 13:21:21 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World!
</pre>
<h2 id="deviceinteract">Interacting with a device (D, R)</h2>
<p>So far, our HelloWorldTest isn't doing anything interesting. Tradefed's
specialty is running tests using Android devices, so lets add an Android device
to the test.</p>
<p>Tests can get a reference to an Android device by implementing the
<a href="/reference/com/android/tradefed/testtype/IDeviceTest.html">IDeviceTest</a>
interface. Here's a sample implementation of what this looks like:</p>
<pre class="prettyprint">
public class HelloWorldTest implements IRemoteTest, IDeviceTest {
private ITestDevice mDevice;
@Override
public void setDevice(ITestDevice device) {
mDevice = device;
}
@Override
public ITestDevice getDevice() {
return mDevice;
}
…
}
</pre>
<p>The Trade Federation framework will inject the <code>ITestDevice</code>
reference into your test via the <code>IDeviceTest#setDevice</code> method,
before the <code>IRemoteTest#run</code> method is called.</p>
<p>Let's modify the HelloWorldTest print message to display the serial number of
the device:</p>
<pre class="prettyprint">
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
System.out.println("Hello, TF World! I have device " + getDevice().getSerialNumber());
}
</pre>
<p>Now rebuild tradefed and check the list of devices:</p>
<pre class="devsite-terminal devsite-click-to-copy">
tradefed.sh
</pre>
<pre class="devsite-click-to-copy">
tf> list devices
Serial State Product Variant Build Battery
004ad9880810a548 Available mako mako JDQ39 100
</pre>
<p>Take note of the serial number listed as <strong>Available</strong>; that is
the device that should be allocated to HelloWorld:</p>
<pre class="devsite-click-to-copy">
tf> run example/helloworld
05-12 13:26:18 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World! I have device 004ad9880810a548
</pre>
<p>You should see the new print message displaying the serial number of the
device.</p>
<h2 id="sendresults">Sending test results (D)</h2>
<p><code>IRemoteTest</code> reports results by calling methods on the
<a href="/reference/com/android/tradefed/result/ITestInvocationListener.html">ITestInvocationListener</a>
instance provided to the <code>#run</code> method. The TF framework itself is
responsible for reporting the start (via
<a href="/reference/com/android/tradefed/result/ITestInvocationListener.html#invocationStarted(com.android.tradefed.build.IBuildInfo)">ITestInvocationListener#invocationStarted</a>)
and end (via
<a href="/reference/com/android/tradefed/result/ITestInvocationListener.html#invocationEnded(long)"
>ITestInvocationListener#invocationEnded</a>)
of each Invocation.</p>
<p>A <b>test run</b> is a logical collection of tests. To report test results,
<code>IRemoteTest</code> is responsible for reporting the start of a test run,
the start and end of each test, and the end of the test run.</p>
<p>Here's what the HelloWorldTest implementation might look like with a single
failed test result.</p>
<pre class="prettyprint">
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
System.out.println("Hello, TF World! I have device " + getDevice().getSerialNumber());
TestIdentifier testId = new TestIdentifier("com.example.TestClassName", "sampleTest");
listener.testRunStarted("helloworldrun", 1);
listener.testStarted(testId);
listener.testFailed(testId, "oh noes, test failed");
listener.testEnded(testId, Collections.emptyMap());
listener.testRunEnded(0, Collections.emptyMap());
}
</pre>
<p>TF includes several <code>IRemoteTest</code> implementations you can reuse
instead of writing your own from scratch. For example,
<a href="/reference/com/android/tradefed/testtype/InstrumentationTest.html">InstrumentationTest</a>
can run an Android application's tests remotely on an Android device, parse the
results, and forward those results to the <code>ITestInvocationListener</code>).
For details, see
<a href="/reference/com/android/tradefed/testtype/package-summary.html">Test
Types</a>.</p>
<h2 id="storeresults">Storing test results (I)</h2>
<p>The default test listener implementation for a TF config is
<a href="/reference/com/android/tradefed/result/TextResultReporter.html">TextResultReporter</a>,
which dumps the results of an invocation to stdout. To illustrate, run the
HelloWorldTest config from the previous section:</p>
<pre class="devsite-terminal devsite-click-to-copy">
./tradefed.sh
</pre>
<pre class="devsite-click-to-copy">
tf> run example/helloworld
05-16 20:03:15 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World! I have device 004ad9880810a548
05-16 20:03:15 I/InvocationToJUnitResultForwarder: run helloworldrun started: 1 tests
Test FAILURE: com.example.TestClassName#sampleTest
stack: oh noes, test failed
05-16 20:03:15 I/InvocationToJUnitResultForwarder: run ended 0 ms
</pre>
<p>To store the results of an invocation elsewhere, such as in a file, specify a
custom <code>ITestInvocationListener</code> implementation using the
<code>result_reporter</code> tag in your configuration.</p>
<p>TF also includes the
<a href="/reference/com/android/tradefed/result/XmlResultReporter.html">XmlResultReporter</a>
listener, which writes test results to an XML file in a format similar to that
used by the <em>ant</em> JUnit XML writer. To specify the result_reporter in the
configuration, edit the <code>…/res/config/example/helloworld.xml</code>
config:</p>
<pre class="prettyprint">
<configuration description="Runs the hello world test">
<test class="com.android.tradefed.example.HelloWorldTest" />
<result_reporter class="com.android.tradefed.result.XmlResultReporter" />
</configuration>
</pre>
<p>Now rebuild tradefed and re-run the hello world sample:</p>
<pre class="devsite-click-to-copy">
tf> run example/helloworld
05-16 21:07:07 I/TestInvocation: Starting invocation for target stub on build 0 on device 004ad9880810a548
Hello, TF World! I have device 004ad9880810a548
05-16 21:07:07 I/XmlResultReporter: Saved device_logcat log to /tmp/0/inv_2991649128735283633/device_logcat_6999997036887173857.txt
05-16 21:07:07 I/XmlResultReporter: Saved host_log log to /tmp/0/inv_2991649128735283633/host_log_6307746032218561704.txt
05-16 21:07:07 I/XmlResultReporter: XML test result file generated at /tmp/0/inv_2991649128735283633/test_result_536358148261684076.xml. Total tests 1, Failed 1, Error 0
</pre>
<p>Notice the log message stating that an XML file has been generated; the
generated file should look like this:</p>
<pre class="prettyprint">
<?xml version='1.0' encoding='UTF-8' ?>
<testsuite name="stub" tests="1" failures="1" errors="0" time="9" timestamp="2011-05-17T04:07:07" hostname="localhost">
<properties />
<testcase name="sampleTest" classname="com.example.TestClassName" time="0">
<failure>oh noes, test failed
</failure>
</testcase>
</testsuite>
</pre>
<p>You can also write your own custom invocation listeners—they simply
need to implement the
<a href="/reference/com/android/tradefed/result/ITestInvocationListener.html">ITestInvocationListener</a>
interface.</p>
<p>Tradefed supports multiple invocation listeners, so you can send test results
to multiple independent destinations. To do this, just specify multiple
<code><result_reporter></code> tags in your config.</p>
<h2 id="logging">Logging (D, I, R)</h2>
<p>TF's logging facilities include the ability to:</p>
<ol>
<li>Capture logs from the device (aka device logcat)</li>
<li>Record logs from the TradeFederation framework running on the host machine
(aka host log)</li>
</ol>
<p>The TF framework automatically captures the logcat from the allocated device
and sends it to the invocation listener for processing.
<code>XmlResultReporter</code> then saves the captured device logcat as a file.
</p>
<p>TF host logs are reported using the
<a href="/reference/com/android/tradefed/log/LogUtil.CLog.html">CLog wrapper</a>
for the ddmlib Log class. Let's convert the
previous <code>System.out.println</code> call in HelloWorldTest to a
<code>CLog</code> call:</p>
<pre class="prettyprint">
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
CLog.i("Hello, TF World! I have device %s", getDevice().getSerialNumber());
</pre>
<p><code>CLog</code> handles string interpolation directly, similar to
<code>String.format</code>. When you rebuild and rerun TF, you should see the
log message on stdout:</p>
<pre class="devsite-click-to-copy">
tf> run example/helloworld
…
05-16 21:30:46 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548
…
</pre>
<p>By default, tradefed
<a href="/reference/com/android/tradefed/log/StdoutLogger.html">outputs host log
messages to stdout</a>. TF also includes a log implementation that writes
messages to a file:
<a href="/reference/com/android/tradefed/log/FileLogger.html">FileLogger</a>.
To add file logging, add a <code>logger</code> tag to the config, specifying the
full class name of <code>FileLogger</code>:</p>
<pre class="prettyprint">
<configuration description="Runs the hello world test">
<test class="com.android.tradefed.example.HelloWorldTest" />
<result_reporter class="com.android.tradefed.result.XmlResultReporter" />
<logger class="com.android.tradefed.log.FileLogger" />
</configuration>
</pre>
<p>Now, rebuild and run the helloworld example again:</p>
<pre class="devsite-click-to-copy">
tf >run example/helloworld
…
05-16 21:38:21 I/XmlResultReporter: Saved device_logcat log to /tmp/0/inv_6390011618174565918/device_logcat_1302097394309452308.txt
05-16 21:38:21 I/XmlResultReporter: Saved host_log log to /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt
…
</pre>
<p>The log message indicates the path of the host log, which, when viewed,
should contain your HelloWorldTest log message:</p>
<pre class="devsite-terminal devsite-click-to-copy">
more /tmp/0/inv_6390011618174565918/host_log_4255420317120216614.txt</pre>
<p>Example output:</p>
<pre class="devsite-click-to-copy">
…
05-16 21:38:21 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548
</pre>
<h2 id="optionhandling">Handling options (D, I, R)</h2>
<p>Objects loaded from a TF Configuration (aka <b>Configuration objects</b>)
can also receive data from command line arguments through the use of the
<code>@Option</code> annotation.<p>
<p>To participate, a Configuration object class applies the <code>@Option</code>
annotation to a member field and provides it a unique name. This enables that
member field value to be populated via a command line option (and also
automatically adds that option to the configuration help system).</p>
<p class="note"><strong>Note:</strong> Not all field types are supported. For a
description of supported types, see
<a href="/reference/com/android/tradefed/config/OptionSetter.html">OptionSetter</a>.
</p>
<p>Let's add an <code>@Option</code> to HelloWorldTest:</p>
<pre class="prettyprint">
@Option(name="my_option",
shortName='m',
description="this is the option's help text",
// always display this option in the default help text
importance=Importance.ALWAYS)
private String mMyOption = "thisisthedefault";
</pre>
<p>Next, let's add a log message to display the value of the option in
HelloWorldTest so we can demonstrate it was received correctly:</p>
<pre class="prettyprint">
@Override
public void run(ITestInvocationListener listener) throws DeviceNotAvailableException {
…
CLog.logAndDisplay(LogLevel.INFO, "I received option '%s'", mMyOption);
</pre>
<p>Finally, rebuild TF and run helloworld; you should see a log message with the
<code>my_option</code> default value:</p>
<pre class="devsite-click-to-copy">
tf> run example/helloworld
…
05-24 18:30:05 I/HelloWorldTest: I received option 'thisisthedefault'
</pre>
<h3 id="passclivalues">Passing values from the command line</h3>
<p>Pass in a value for <code>my_option</code>; you should see
<code>my_option</code> populated with that value:</p>
<pre class="devsite-click-to-copy">
tf> run example/helloworld --my_option foo
…
05-24 18:33:44 I/HelloWorldTest: I received option 'foo'
</pre>
<p>TF configurations also include a help system, which automatically displays
help text for <code>@Option</code> fields. Try it now, and you should see the
help text for <code>my_option</code>:</p>
<pre class="devsite-click-to-copy">
tf> run example/helloworld --help
Printing help for only the important options. To see help for all options, use the --help-all flag
cmd_options options:
--[no-]help display the help text for the most important/critical options. Default: false.
--[no-]help-all display the full help text for all options. Default: false.
--[no-]loop keep running continuously. Default: false.
test options:
-m, --my_option this is the option's help text Default: thisisthedefault.
'file' logger options:
--log-level-display the minimum log level to display on stdout. Must be one of verbose, debug, info, warn, error, assert. Default: error.
</pre>
<p>Note the message about "printing only the important options." To reduce
option help clutter, TF uses the <code>Option#importance</code> attribute to
determine whether to show a particular <code>@Option</code> field help text when
<code>--help</code> is specified. <code>--help-all</code> always shows help for
all <code>@Option</code> fields, regardless of importance. For details, see
<a href="/reference/com/android/tradefed/config/Option.Importance.html">Option.Importance</a>.
</p>
<h3 id="passconfvalues">Passing values from a configuration</h3>
<p>You can also specify an Option value within the config by adding a
<code><option name="" value=""></code> element. Test it using
<code>helloworld.xml</code>:</p>
<pre class="prettyprint">
<test class="com.android.tradefed.example.HelloWorldTest" >
<option name="my_option" value="fromxml" />
</test>
</pre>
<p>Re-building and running helloworld should now produce this output:</p>
<pre class="devsite-click-to-copy">
05-24 20:38:25 I/HelloWorldTest: I received option 'fromxml'
</pre>
<p>The configuration help should also update to indicate the default value of
<code>my_option</code>:</p>
<pre class="devsite-click-to-copy">
tf> run example/helloworld --help
test options:
-m, --my_option this is the option's help text Default: fromxml.
</pre>
<p>Other configuration objects included in the helloworld config, such as
<code>FileLogger</code>, also accept options. The option
<code>--log-level-display</code> is interesting because it filters the logs that
show up on stdout. Earlier in the tutorial, you may have noticed the "Hello, TF
World! I have device …' log message stopped being displayed on stdout after we
switched to using <code>FileLogger</code>. You can increase the verbosity of
logging to stdout by passing in the <code>--log-level-display</code> arg.</p>
<p>Try this now, and you should see the 'I have device' log message reappear on
stdout, in addition to being logged to a file:</p>
<pre class="devsite-click-to-copy">
tf> run example/helloworld --log-level-display info
…
05-24 18:53:50 I/HelloWorldTest: Hello, TF World! I have device 004ad9880810a548
</pre>
<h2 id="conclusion">That's all, folks!</h2>
<p>As a reminder, if you're stuck on something, the
<a href="https://android.googlesource.com/platform/tools/tradefederation/+/master">Trade
Federation source code</a> has a lot of useful information that isn't exposed in
the documentation. If all else fails, try asking on the
<a href="/setup/community.html">android-platform</a> Google Group,
with "Trade Federation" in the message subject.</p>
</body>
</html>
|
