I'm a little annoyed at myself that I only realized this _after_ #392030
got merged. But I realized that if something else is using AF_VSOCK or
you simply have another interactive test running (e.g. by another user
on a larger builder), starting up VMs in the driver fails with
qemu-system-x86_64: -device vhost-vsock-pci,guest-cid=3: vhost-vsock: unable to set guest cid: Address already in use
Multi-user setups are broken anyways because you usually don't have
permissions to remove the VM state from another user and thus starting
the driver fails with
PermissionError: [Errno 13] Permission denied: PosixPath('/tmp/vm-state-machine')
but this is something you can work around at least.
I was considering to generate random offsets, but that's not feasible
given we need to know the numbers at eval time to inject them into the
QEMU args. Also, while we could do this via the test-driver, we should
also probe if the vsock numbers are unused making the code even more
complex for a use-case I consider rather uncommon.
Hence the solution is to do
sshBackdoor.vsockOffset = 23542;
when encountering conflicts.
Closes#180089
I realized that the previous commits relying on `sys.exit` for dealing
with `MachineError`/`RequestedAssertionFailed` exit the interactive
session which is kinda bad.
This patch uses the ipython driver: it seems to have equivalent features
such as auto-completion and doesn't stop on SystemExit being raised.
This also fixes other places where this happened such as things calling
`log.error` on the CompositeLogger.
After a discussion with tfc, we agreed that we need a distinction
between errors where the user isn't at fault (e.g. OCR failing - now
called `MachineError`) and errors where the test actually failed (now
called `RequestedAssertionFailed`).
Both get special treatment from the error handler, i.e. a `!!!` prefix
to make it easier to spot visually.
However, only `RequestedAssertionFailed` gets the shortening of the
traceback, `MachineError` exceptions may be something to report and
maintainers usually want to see the full trace.
Suggested-by: Jacek Galowicz <jacek@galowicz.de>
When doing `machine.succeed(...)` or something similar, it's now clear
that the command `...` was issued on `machine`.
Essentially, this results in the following diff in the log:
-(finished: waiting for unit default.target, in 13.47 seconds)
+machine: (finished: waiting for unit default.target, in 13.47 seconds)
(finished: subtest: foobar text lorem ipsum, in 13.47 seconds)
I think it's reasonable to also have this kind of visual distinction
here between test failures and actual errors from the test framework.
A failing `machine.require_unit_state` now lookgs like this for
instance:
!!! Traceback (most recent call last):
!!! File "<string>", line 3, in <module>
!!! machine.require_unit_state("postgresql","active")
!!!
!!! RequestedAssertionFailed: Expected unit 'postgresql' to to be in state 'active' but it is in state 'inactive'
Co-authored-by: Benoit de Chezelles <bew@users.noreply.github.com>
Replaces / Closes#345948
I tried to integrate `pytest` assertions because I like the reporting,
but I only managed to get the very basic thing and even that was messing
around a lot with its internals.
The approach in #345948 shifts too much maintenance effort to us, so
it's not really desirable either.
After discussing with Benoit on Ocean Sprint about this, we decided that
it's probably the best compromise to integrate `unittest`: it also
provides good diffs when needed, but the downside is that existing tests
don't benefit from it.
This patch essentially does the following things:
* Add a new global `t` that is an instance of a `unittest.TestCase`
class. I decided to just go for `t` given that e.g.
`tester.assertEqual` (or any other longer name) seems quite verbose.
* Use a special class for errors that get special treatment:
* The traceback is minimized to only include frames from the
testScript: in this case I don't really care about anything else and
IMHO that's just visual noise.
This is not the case for other exceptions since these may indicate a
bug and then people should be able to send the full traceback to the
maintainers.
* Display the error, but with `!!!` as prefix to make sure it's
easier to spot in between other logs.
This looks e.g. like
!!! Traceback (most recent call last):
!!! File "<string>", line 7, in <module>
!!! foo()
!!! File "<string>", line 5, in foo
!!! t.assertEqual({"foo":[1,2,{"foo":"bar"}]},{"foo":[1,2,{"bar":"foo"}],"bar":[1,2,3,4,"foo"]})
!!!
!!! NixOSAssertionError: {'foo': [1, 2, {'foo': 'bar'}]} != {'foo': [1, 2, {'bar': 'foo'}], 'bar': [1, 2, 3, 4, 'foo']}
!!! - {'foo': [1, 2, {'foo': 'bar'}]}
!!! + {'bar': [1, 2, 3, 4, 'foo'], 'foo': [1, 2, {'bar': 'foo'}]}
cleanup
kill machine (pid 9)
qemu-system-x86_64: terminating on signal 15 from pid 6 (/nix/store/wz0j2zi02rvnjiz37nn28h3gfdq61svz-python3-3.12.9/bin/python3.12)
kill vlan (pid 7)
(finished: cleanup, in 0.00 seconds)
Co-authored-by: bew <bew@users.noreply.github.com>
The current setup is really weird and definitely wrong for many cases
because it inverts the colors of the image, which is never a good idea
for GUIs. So, try to OCR three different times: once on the source image,
once with processing, and once with processing but no negation.
This should hopefully make things work at least somewhat better for GUIs.
Specifies the "last try" parameter in all methods called by `retry`.
Doing this clarifies its presence, and makes it easier to use it in the
future if needed.
I don't know why it doesn't want to do TIFF now, but there's also
absolutely no reason for it to be TIFF anyway, so let's just use
an image format that is actually sane.
