| Age | Commit message (Collapse) | Author |
|---|
|
* changes:
lstat generated files
Shell-escape $in and $out
Disallow missing depfiles
Implement --frontend-file
Add multithreaded parsing
|
|
Adding this sleep did seem to drastically reduce the flakiness, but
we recently got another flake. Increase the length of the sleep to
hopefully hammer out the last bit of flakeyness.
Test: Cargo test
Change-Id: I3b0ef31efc92357f90e49d04876f17397fa507e5
|
|
This is to match a patch that we added to the android fork of ninja,
where generated files are lstat'd instead of stat'd. Without this,
n2 always reruns rules that generate dangling symlinks, because it
thinks those files don't exist.
Bug: 318434287
Test: Manually
Change-Id: I3ced97c9195adb3b8d94e3103940f197097c278c
|
|
Excape in cases where they need to be read by n2 directly.
E.g. `rspfile = $in` is not escaped because n2 will just open the file
directly, but `rspfile_content = $in` will be escaped because rspfiles
are often treated as command line arguments.
Bug: 318434287
Test: cargo test
Change-Id: I68a8649eabbf3b0ca459e0cac28160a834eebd62
|
|
We always run ninja with -w missingdepfile=err. N2 doesn't have that
flag, but we can just make it always enforce that dep files exist.
Bug: 318434287
Test: cargo test
Change-Id: I5e7c195efca6779db631e65017f14a10d3f48d15
|
|
This is what soong uses to take control of ninja's output. Ninja will
just be quiet but write protobuf-encoded status messages to a file that
soong_ui reads.
Not all fields of the protobuf are implemented in this cl, but it's
enough to display a functional UI in an android build. Missing things
include:
- A bunch of fields related to how much time/memory an action took
- A critical path time estimate used in displaying an ETA, but that
requires the infrastructure about the time of prior builds used
to schedule slower actions with higher priority in regular ninja
- The inputs that changed that caused an action to rerun, used in
some telemetry
Bug: 318434287
Test: cargo test
Change-Id: I79432cdab965ff5fc7e0d5b42c645488b2e9093b
|
|
This check seems to flake sometimes, make it more lenient.
Bug: 318434287
Test: cargo test
Change-Id: I21c46360ae60c9f1a1e0ba488f5f674bc4691615
|
|
This PR ports over most of the enhancements that are in Android's fork
of ninja to n2. N2 goes from taking ~16 seconds to parse the AOSP ninja
files, to ~1 second. (at least on my work computer where
available_parallelism returns 128) It still seems to be about ~0.1
seconds slower than the android fork of ninja, but it's at least in the
range where it's a little difficult to measure.
A general list of the changes in this PR:
- Add support for subninja, with scoped variables.
- mmap input files instead of reading them into memory.
- Remove the FileIds and the Vec of files, replacing them with Arc.
Having to maintain a separate vec mapping FileIds to files is too
expensive.
- Use the DashMap crate to map from filename to File object, so that
id_from_canonical can be called in parallel.
- Use rayon for multithreading.
- Split input files into chunks and parse the chunks in parallel
- Parse subninjas in parallel. Subninja support was implemented because
regular includes cannot be processed in parallel.
- Revamp the evaluation of variable assignments. Now, every statement
in the ninja file gets a "scope position". When evaluating global
variables, they are evaluated relative to a certain scope position,
so you can evaluate the variable at a particular point in the file,
even if it is reassigned later.
- Introduce "clumps" of parse results, as returning all the parsed
statements in a flat vector causes too many large vector
concatenations.
- Box Build objects, as they are quite large, and Vecs of unboxed
builds are too large and slow to concatenate.
- Don't evaluate all the build bindings at load time. Only evaluate the
input/output files, and defer evaluation of other bindings until the
build is actually run.
- Evalstrings are now just regular strings instead of Vecs of strings.
This means when they're evaluated they're re-parsed. Doing it this
way saves of the Vec allocations, and in the case of owned
evalstrings, does one big string allocation instead of a bunch of
little ones. There are a lot more owned evalstrings now due to
deferred evaluation of build bindings, so it makes a bit of a
difference.
One thing that the android fork has that I didn't include was
precomputed hashes. The android fork has a HashedString and
HashedStrView type that it uses to ensure hashes aren't recalculated.
This is somewhat difficult to do in rust, you have to either make
concessions about being able to look up map values by their borrowed
representation, use the raw entry map apis on nightly rust, or use
hashbrown's RawTable. And even when I was experimenting with RawTable it
didn't seem to provide a noticeable speedup.
Some of these optimizations are memory optimizations. This turned out to
be necessary because apparently the exit_group syscall that quits the
process spends time cleaning up memory. The exit_group syscall was
taking over a second to run before some of these optimizations.
Bug: 318434287
Test: OUT_DIR=out prebuilts/build-tools/build-prebuilts.sh
Change-Id: I2dd9e5dae37c2904ecf01db2cd771857e5f7aae8
|
|
Sleep before/after running n2, because it seems like occasionally
n2 is not seeing updating timestamps. Maybe the tests run too quickly
and the timestamps are the same?
Bug: 338259955
Test: Presubmits
Change-Id: Ib695815317499c95514ccf0653c55ed19e2fbb96
|
|
This test is flakey, disable it until we can do a proper fix.
Bug: 338259955
Test: Presubmits
Change-Id: I8218fb08524f17c2332738e353da2a744e08f762
|
|
Bug: 318434287
Test: OUT_DIR=out prebuilts/build-tools/build-prebuilts.sh
Change-Id: I8cad9c5951d71b719e5a87907a5099be4b73468b
|
|
Import b/328273370
|
|
Test: m n2
Third-Party Import of: https://github.com/evmar/n2
Request Document: go/android3p
For CL Reviewers: go/android3p#reviewing-a-cl
For Build Team: go/ab-third-party-imports
Bug: http://b/328273370
Original import of the code can be found at: https://googleplex-android.googlesource.com/platform/external/n2/+/refs/heads/third-party-review.
Security Questionnaire: http://b/328273370#comment1
Change-Id: Iab199b146d606cb1e8212f5c38d8612a5845b6bf
|
|
|
|
This is needed for android. It just treats all the deps of all targets
in the depfile as deps for the Build.
In task.rs there is a TODO to verify that the targets refer to the
outputs of the Build, but adding that verification breaks the android
build. In android's case, there are some depfiles that contain
`foo.o: path/to/some/dep.asm` where it should instead be
`path/to/foo.o: path/to/some/dep.asm`.
|
|
FxHashMap has a faster hashing algorithm,
at the expense of not being resistent to DOS
attacks.
|
|
Use HashMap.entry() instead of a lookup + insert.
|
|
So that we can do one memory allocation for them.
|
|
id_from_canonical ideally takes owned strings
instead of references to avoid a copy.
|
|
Scanner expects its input to be nul terminated, which I implemented as
buf = std::fs::read(file)
buf.push(0)
However, std::fs::read carefully sizes its buffer to be exactly the file's size,
which means the push() must grow the buffer. This means for e.g. a 40mb input
file, we'd read it into a 40mb buffer, then grow the buffer, copying it into an
80mb buffer, just to add one byte!
Fix this by using a buffer of the appropriate size.
I attempted to measure the perf impact here and it wasn't measureable.
Possibly this buffer growth really is that fast? It seems it will at least
have memory impact at least.
|
|
This fixes an incompatibility with ninja.
I've also moved a bunch of variable evaluations out of the
parser and into the loader, in preparation for parsing the
build file in multiple threads, and then only doing the
evaluations after all the chunks of the file have been
parsed.
Fixes #91 and #39.
|
|
I'm making some changes that affect the performance
of the parser and the loader, and would like to
benchmark them together.
Unfortunately this required making a bunch of things
public (#[cfg(test)] doesn't seem to apply for
benchmarks), which isn't great.
|
|
Regular ninja also has this validation.
|
|
It's a needless burden for people who don't care about this snapshot.
I switched to Google Drive for now, shruggie.
Fixes #105.
|
|
The test is supposed to ensure that out builds before validation_input.
|
|
The test wanted to verify that in a file containing
build foo: ... |@ bar
it's possible for foo to finish before bar does.
This changes to the test to make the bar step wait (at runtime) for the
foo step to finish, enforcing that that ordering is possible.
Fixes #102.
|
|
CMake has files that are outputs of a phony rule but still used as inputs.
build ...: ... some_file
build some_file: phony ...
Before this change, n2 would just mark all phony outputs as missing, which
breaks the above. This change instead makes phony a little closer to how
non-phony rules work which fixes #40 and which I hope is a benefit in general.
Fixes #40.
|
|
|
|
|
|
See bug report context in #98.
From a patch from Tobias Hieta <tobias.hieta@ubisoft.com>.
|
|
CMake can sometimes write depfile directives to build.ninja
even when no depfile is generated. This was handled by "ninja"
by just ignoring the missing file and going on with the build,
see the code here:
https://github.com/ninja-build/ninja/blob/master/src/build.cc#L894
This fixes #80
|
|
|
|
|
|
|
|
It's possible to have a ninja dep on '.', meaning the directory that
ninja was run from. Previously, canon_path_fast would canonicalize
this to an empty string, which would cause stats and other opterations
to fail.
This was found when running the android codebase with n2.
|
|
|
|
|
|
Add docs for how to use it too.
|
|
This is in the Criterion FAQ.
|
|
|
|
|
|
I had been too clever here with a statically-sized buffer, when really I ought
to profile first and optimize second.
I think it's still correct (without profiling) to avoid calling the underlying
write() syscall too many times, so buffering at least a record at a time still
makes sense.
Fixes #92.
|
|
The pointers passed in a PROC_THREAD_ATTRIBUTE_LIST must
outlive the struct, so add some lifetime hackery.
|
|
|
|
These were added to regular ninja in:
https://github.com/ninja-build/ninja/commit/04c410b15b70fb321928ffba19d697db15cb0121
Android uses them, so they're needed to port android to n2. (which is
something I'm exploring but not committing to)
|
|
|
|
The Ninja rules here aren't super clear, like I think this might be legal
build $
foo$
:$
bar
baz$
=$
blah
and at least that kind of thing appears to be possible output of ninja_syntax.py.
Fixes #89.
|
|
|
|
I'm ambivalent about this behavior, but it's come up twice so I think
people depend on it.
Fixes #88 and #53.
|
|
|
