test(skew): version-skew smoke — dev Studio against the released hub
Some checks failed
Security / Security check (push) Failing after 2s

Walks every sidebar page against an older, released chain binary
(hermetic fixture hub) and fails when any page claims 'unreachable'
although the hub answers, or renders raw error text. Run via the
platform repo's scripts/skew-smoke.sh as a Studio release gate.

Verified against the real pair hub 0.21.0 vs workspace 0.22.0: all
invariants hold. Two timing notes are handled explicitly: real-async
settles between navigations and an app teardown before the fixture
teardown keep grpc-dart's data-after-cancel race (unguarded
_responses.add, call.dart:395) from failing the walk spuriously;
the final fake-time pump drains the channel's 5-minute idle timer.

Signed-off-by: flemming-it <stefan.a.flemming@googlemail.com>
This commit is contained in:
flemming-it 2026-07-19 02:08:48 +02:00
parent 66b26304fd
commit bfd58baa75
2 changed files with 162 additions and 0 deletions

View file

@ -6,6 +6,31 @@ lockstep.
## Unreleased
### Hardening round (test strategy, 2026-07-19)
- **Hermetic widget suites.** `HubService.instance` is injectable;
a scriptable fake (`test/support/fake_hub.dart`) answers every
member the pages touch and scripts per-RPC failures. The a11y and
responsive sweeps run against it — results no longer depend on
whether a hub happens to listen on the operator's machine (which
was the root of the historic a11y flake).
- **State-matrix sweep** (`state_matrix_test.dart`): every sidebar
page × hub condition (healthy / gone / too old / feature off) with
app-wide invariants — no "unreachable" claims while the hub
answers, no raw error text in the UI, classified states only.
- **Shared load-error view.** Store, Doctor, Audit, Approvals and
Federation no longer fold every load failure into "hub not
reachable"; `HubLoadErrorView` classifies into unreachable /
needs-newer-hub / load-failed-with-copyable-detail (new DE+EN
strings). Also fixed: the approvals page's hidden tab surfaced
load failures as uncaught async errors; the audit status bar
rendered the raw gRPC error wall instead of the friendly headline.
- **Version-skew gate.** `integration_test/skew_smoke_test.dart`
(run via the platform repo's `scripts/skew-smoke.sh`) boots the
released hub binary and walks every page with the same
invariants — the dev-Studio-against-old-hub combination is now a
release gate instead of a live-bug generator.
### Added (0.74.0)
- **Doctor findings deep-link to their page.** Summary tiles and

View file

@ -0,0 +1,137 @@
// Version-skew smoke the DEV Studio against an OLDER, released
// hub binary. The two worst live bugs of the project were skew
// bugs (ABI 1.0/1.1, a 0.21 hub without the ListInvocations RPC
// rendered as "hub not reachable"), and no test ever ran the
// combination operators actually have: a Studio that is newer
// than its hub.
//
// Run through the platform repo's scripts/skew-smoke.sh, which
// resolves the old binary and isolates $HOME:
//
// CHAIN_BIN=<old released chain> \
// flutter test integration_test/skew_smoke_test.dart -d macos
//
// Invariants per page (the hub IS running and healthy):
// * never "nicht erreichbar" a reachable hub must never be
// rendered as unreachable, whatever RPCs it is missing
// * never raw error text (gRPC walls, UnimplementedError, ...)
// missing RPCs must surface as classified, localised states
// ("braucht eine neuere Hub-Version", feature-off, ...)
import 'dart:io';
import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';
import 'package:shared_preferences/shared_preferences.dart';
import 'package:chain_client_sdk/chain_client_sdk.dart' show HubEndpoint;
import 'package:chain_studio/data/hub.dart';
import 'package:chain_studio/main.dart';
import '../test/integration/hub_fixture.dart';
const _destinations = <String>[
'welcome',
'store',
'doctor',
'flows',
'audit',
'approvals',
'runs',
'federation',
];
const _rawErrorMarkers = <String>[
'gRPC Error',
'UnimplementedError',
'SocketException',
'Instance of',
'StackTrace',
];
Future<void> _pumpFrames(WidgetTester tester, [int frames = 20]) async {
for (var i = 0; i < frames; i++) {
await tester.pump(const Duration(milliseconds: 100));
}
}
String _renderedText(WidgetTester tester) {
final buf = StringBuffer();
for (final w in tester.widgetList<Text>(find.byType(Text))) {
buf.writeln(w.data ?? w.textSpan?.toPlainText() ?? '');
}
for (final w in tester.widgetList<SelectableText>(
find.byType(SelectableText),
)) {
buf.writeln(w.data ?? '');
}
return buf.toString();
}
void main() {
IntegrationTestWidgetsFlutterBinding.ensureInitialized();
testWidgets('dev Studio against the released hub: reachable is never '
'rendered unreachable, no raw errors', (tester) async {
final fixture = await HubFixture.start(skipIfBinaryMissing: false);
addTearDown(fixture!.dispose);
final binary = (await HubFixture.binaryPath())!;
final version = await Process.run(binary, ['--version']);
// Surfaced in the runner output so the log states which skew
// pair was actually exercised.
// ignore: avoid_print
print('skew-smoke: hub binary = ${(version.stdout as String).trim()}');
SharedPreferences.setMockInitialValues({});
await tester.pumpWidget(
const StudioApp(
initialThemeMode: ThemeModeValue.dark,
initialLocale: Locale('de'),
),
);
await HubService.instance.reconnect(
HubEndpoint(host: '127.0.0.1', port: fixture.port),
authToken: null,
persist: false,
);
await _pumpFrames(tester, 30);
final shell = tester.state<StudioShellState>(find.byType(StudioShell));
final violations = <String>[];
for (final id in _destinations) {
shell.navigateTo(id);
await _pumpFrames(tester, 25);
// Let in-flight gRPC frames land in REAL async before the next
// navigation disposes the page: grpc-dart 4.2 adds incoming
// data to an already-closed controller when a cancel races the
// response (call.dart:395, unguarded `_responses.add`) an
// upstream bug that would fail the walk spuriously.
await tester.runAsync(
() => Future<void>.delayed(const Duration(milliseconds: 150)),
);
final text = _renderedText(tester);
if (text.contains('nicht erreichbar')) {
violations.add(
'page "$id" claims the hub is unreachable although the '
'fixture hub is healthy',
);
}
for (final marker in _rawErrorMarkers) {
if (text.contains(marker)) {
violations.add('page "$id" renders raw error text "$marker"');
}
}
}
// Tear the app down BEFORE the fixture teardown closes the hub:
// page subscriptions cancel while the channel is still alive, so
// the shutdown cannot race live streams into add-after-close.
await tester.pumpWidget(const SizedBox.shrink());
await tester.runAsync(
() => Future<void>.delayed(const Duration(milliseconds: 200)),
);
await tester.pump(const Duration(minutes: 6));
expect(violations, isEmpty, reason: violations.join('\n'));
});
}