fai/chain-studio
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

feat(test): hermetic hub fake + state-matrix sweep across all pages

Browse source 
The widget suites used to talk to whatever listens on the real
endpoint — results depended on the operator's machine (a running
hub fed real data into a11y/responsive runs and its gRPC channel
timers caused the historic flake). Hardening round:

- HubService.instance is now injectable (debugSetInstance);
  FakeHubService (test/support/fake_hub.dart) answers every member
  the pages touch with healthy-empty defaults and scripts per-RPC
  failures (UNIMPLEMENTED / UNAVAILABLE / detached gate) through a
  GrpcError-shaped fake. Unimplemented members are recorded and
  fail the sweep with the exact list.
- state_matrix_test.dart pins the app-wide invariants for every
  sidebar page x hub condition: healthy => no unreachable claims
  and no raw error text; hub gone => honest unreachable states;
  UNIMPLEMENTED => never 'not reachable' while the sidebar shows
  connected; detached gate => plain-language feature-off state.
- a11y + responsive sweeps now inject the fake (hermetic); the
  6-minute idle-timer drain workaround is gone with the cause.

Real bugs the new sweep caught immediately:
- every data page (store, doctor, audit, approvals, federation)
  folded ANY load failure into 'hub not reachable' — the runs-page
  bug class; they now share HubLoadErrorView, which classifies
  into unreachable / needs-newer-hub / load-failed-with-copyable-
  detail (new generic DE+EN strings)
- the approvals page's hidden tab had no future listener: a load
  failure there surfaced as an uncaught async error
- the audit status bar rendered the raw gRPC error wall verbatim;
  it now shows the classified friendly headline (still selectable)

Signed-off-by: flemming-it <stefan.a.flemming@googlemail.com>
This commit is contained in:
flemming-it 2026-07-19 01:02:51 +02:00
parent adc5fc2311
commit 66b26304fd
19 changed files with 733 additions and 97 deletions
Download patch file Download diff file Expand all files Collapse all files

20
test/a11y_test.dart
View file

@ -24,6 +24,8 @@ import 'package:shared_preferences/shared_preferences.dart';
import 'package:chain_studio/data/hub.dart';
import 'package:chain_studio/main.dart';
import 'support/fake_hub.dart';
const _destinations = <String>[
'welcome',
'store',
@ -40,6 +42,11 @@ void main() {
testWidgets('all pages meet a11y guidelines — ${mode.name}',
(tester) async {
SharedPreferences.setMockInitialValues({});
// Hermetic: pages render against the scriptable fake, never a
// hub that happens to listen on the operator's machine (the
// suite's historic pending-timer flake came from real gRPC
// channels arming their 5-minute idle timer mid-test).
installFakeHub();
// Layout correctness across sizes is responsive_test.dart's
// job; this suite audits colors and labels at a normal size.
tester.view.physicalSize = const Size(1280, 800);
@ -88,19 +95,6 @@ void main() {
// !timersPending after the test body.
await tester.pumpWidget(const SizedBox.shrink());
await tester.pump(const Duration(minutes: 1));
// The suite's long-standing pending-timer flake, finally
// caught with a creation stack: when the last gRPC stream
// closes, Http2ClientConnection._handleActiveStateChanged
// arms the channel's 5-minute idleTimeout timer — even on a
// shut-down connection — so a 1-minute drain never covered
// it. Close the channel in real-async space (lets in-flight
// socket callbacks land), then pump PAST the idle timeout so
// the timer fires inside the test body.
await tester.runAsync(() async {
HubService.instance.debugResetChannel();
await Future<void>.delayed(const Duration(milliseconds: 100));
});
await tester.pump(const Duration(minutes: 6));
expect(
violations,
isEmpty,

5
test/responsive_test.dart
View file

@ -13,6 +13,8 @@ import 'package:flutter_test/flutter_test.dart';
import 'package:chain_studio/data/hub.dart';
import 'package:chain_studio/main.dart';
import 'support/fake_hub.dart';
const _destinations = <String>[
'welcome',
'store',
@ -35,6 +37,9 @@ void main() {
for (final entry in _sizes.entries) {
testWidgets('all pages lay out without overflow — ${entry.key}',
(tester) async {
// Hermetic: pages render against the scriptable fake, never a
// hub that happens to listen on the operator's machine.
installFakeHub();
tester.view.physicalSize = entry.value;
tester.view.devicePixelRatio = 1.0;
addTearDown(tester.view.reset);

202
test/state_matrix_test.dart Normal file
View file

@ -0,0 +1,202 @@
// State-matrix sweep — every sidebar page against every hub
// condition, with the app-wide invariants that single-page tests
// keep missing:
//
// 1. healthy-empty -> no page may claim "hub not reachable",
// no raw error text anywhere, and the fake
// must cover every HubService member the
// pages touch (hermeticity guarantee).
// 2. hub gone -> the data pages show the honest
// unreachable state; still no raw errors.
// 3. hub too old -> RPCs answer UNIMPLEMENTED: NO page may
// say "not reachable" while the sidebar
// shows connected (the runs-page bug class,
// now pinned for every page).
// 4. feature off -> the detached gate error renders the
// plain-language feature-off empty state.
//
// The suite runs against the scriptable FakeHubService — never a
// real hub — so results cannot depend on the operator's machine.
import 'package:flutter/material.dart';
import 'package:flutter_test/flutter_test.dart';
import 'package:shared_preferences/shared_preferences.dart';
import 'package:chain_studio/data/hub.dart';
import 'package:chain_studio/main.dart';
import 'support/fake_hub.dart';
const _destinations = <String>[
'welcome',
'store',
'doctor',
'flows',
'audit',
'approvals',
'runs',
'federation',
];
/// Substrings that must NEVER appear in rendered text — they mean
/// a raw thrown object reached the UI instead of a classified,
/// localised state.
const _rawErrorMarkers = <String>[
'gRPC Error',
'UnimplementedError',
'SocketException',
'Instance of',
'StackTrace',
];
/// Page-primary read RPCs — the ones an older hub would be
/// missing. Scenario 3 fails exactly these with UNIMPLEMENTED.
const _pageReads = <String>{
'doctor',
'searchStore',
'recentEvents',
'streamEvents',
'pendingApprovals',
'listApprovalsRecords',
'listDetachedRuns',
'listSatellites',
};
String _renderedText(WidgetTester tester) {
final buf = StringBuffer();
for (final w in tester.widgetList<Text>(find.byType(Text))) {
buf.writeln(w.data ?? w.textSpan?.toPlainText() ?? '');
}
for (final w in tester.widgetList<SelectableText>(
find.byType(SelectableText),
)) {
buf.writeln(w.data ?? '');
}
return buf.toString();
}
Future<void> _boot(WidgetTester tester) async {
SharedPreferences.setMockInitialValues({});
tester.view.physicalSize = const Size(1280, 800);
tester.view.devicePixelRatio = 1.0;
addTearDown(tester.view.reset);
await tester.pumpWidget(
const StudioApp(
initialThemeMode: ThemeModeValue.dark,
initialLocale: Locale('de'),
),
);
await tester.pump(const Duration(milliseconds: 100));
}
Future<void> _goto(WidgetTester tester, String id) async {
await tester.tap(find.byKey(ValueKey('sidebar-item-$id')));
await tester.pump(const Duration(milliseconds: 400));
await tester.pump(const Duration(milliseconds: 400));
}
Future<void> _teardownApp(WidgetTester tester) async {
await tester.pumpWidget(const SizedBox.shrink());
await tester.pump(const Duration(minutes: 1));
}
void main() {
testWidgets('healthy hub: no unreachable claims, no raw errors, '
'fake covers all members', (tester) async {
final fake = installFakeHub();
await _boot(tester);
final violations = <String>[];
for (final id in _destinations) {
await _goto(tester, id);
final text = _renderedText(tester);
if (text.contains('nicht erreichbar')) {
violations.add('page "$id" claims unreachable on a healthy hub');
}
for (final marker in _rawErrorMarkers) {
if (text.contains(marker)) {
violations.add('page "$id" renders raw error text "$marker"');
}
}
}
await _teardownApp(tester);
expect(
fake.missing,
isEmpty,
reason:
'pages touched HubService members the fake does not '
'implement — add healthy defaults for: ${fake.missing}',
);
expect(violations, isEmpty, reason: violations.join('\n'));
});
testWidgets('hub gone: data pages say unreachable, never raw errors', (
tester,
) async {
final fake = installFakeHub();
fake.failWith(kUnavailable);
await _boot(tester);
final violations = <String>[];
// Pages that render a full-page load state from the hub.
for (final id in ['store', 'doctor', 'audit', 'approvals', 'runs',
'federation']) {
await _goto(tester, id);
final text = _renderedText(tester);
if (!text.contains('nicht erreichbar')) {
violations.add('page "$id" hides that the hub is unreachable');
}
for (final marker in _rawErrorMarkers) {
if (text.contains(marker)) {
violations.add('page "$id" renders raw error text "$marker"');
}
}
}
await _teardownApp(tester);
expect(violations, isEmpty, reason: violations.join('\n'));
});
testWidgets('hub too old (UNIMPLEMENTED): connected is never shown '
'as unreachable', (tester) async {
final fake = installFakeHub();
fake.failWith(kUnimplemented, only: _pageReads);
await _boot(tester);
final violations = <String>[];
for (final id in _destinations) {
await _goto(tester, id);
final text = _renderedText(tester);
// THE invariant: the hub answered, so no page may
// contradict the sidebar's connected state.
if (text.contains('nicht erreichbar')) {
violations.add(
'page "$id" claims unreachable although the hub answered '
'(UNIMPLEMENTED)',
);
}
for (final marker in _rawErrorMarkers) {
if (text.contains(marker)) {
violations.add('page "$id" renders raw error text "$marker"');
}
}
}
// Spot-check the classified rendition on an affected page.
await _goto(tester, 'doctor');
expect(
find.textContaining('neuere Hub-Version'),
findsWidgets,
reason: 'the too-old state must say so in plain language',
);
await _teardownApp(tester);
expect(violations, isEmpty, reason: violations.join('\n'));
});
testWidgets('detached gate error renders the feature-off state', (
tester,
) async {
final fake = installFakeHub();
fake.failWith(kDetachedOff, only: {'listDetachedRuns'});
await _boot(tester);
await _goto(tester, 'runs');
expect(find.text('Keine Läufe im Hintergrund'), findsOneWidget);
expect(find.textContaining('nicht erreichbar'), findsNothing);
await _teardownApp(tester);
});
}

297
test/support/fake_hub.dart Normal file
View file

@ -0,0 +1,297 @@
// Scriptable HubService fake — the hermeticity backbone of the
// widget suites. Injected via HubService.debugSetInstance so the
// pages under test never open a real gRPC channel: test results
// must not depend on whether a hub happens to listen on the
// operator's machine (the a11y suite's old flake came exactly
// from that coupling).
//
// Behaviour model:
// * Default: a healthy, empty hub — every read succeeds with
// zero modules / flows / events / approvals / runs.
// * `failWith(error, only: {...})` scripts per-RPC failures —
// e.g. UNIMPLEMENTED for one method to simulate an older hub,
// or `failAll` for a hub that is gone entirely.
// * `FakeGrpcError` mirrors package:grpc's GrpcError shape
// (`.code` / `.message`) because production code duck-types
// on those fields (friendly_error.dart, runs classification).
//
// Any HubService member the fake does not implement lands in
// noSuchMethod, which records the name in [missing] and throws —
// so a sweep test can fail with the exact list of members that
// still need a default instead of silently rendering error states.
import 'dart:ui';
import 'package:flutter_test/flutter_test.dart';
import 'package:chain_client_sdk/chain_client_sdk.dart'
show HubEndpoint, HubProbeResult, StoreSource;
import 'package:chain_studio/data/hub.dart';
/// Duck-typed stand-in for `GrpcError`: production code reads
/// `.code` (int) and `.message` (String?) off thrown objects.
class FakeGrpcError implements Exception {
final int code;
final String? message;
const FakeGrpcError(this.code, [this.message]);
@override
String toString() => 'gRPC Error (code: $code, message: $message)';
}
/// Shorthands for the codes the scenarios use.
const kUnimplemented = FakeGrpcError(12, 'method not implemented');
const kUnavailable = FakeGrpcError(14, 'connection refused');
const kDetachedOff = FakeGrpcError(
9,
'detached invocations are not enabled — set detached.enabled: true '
'in the operator config',
);
class FakeHubService extends Fake implements HubService {
/// Error thrown by every RPC-backed method when set.
Object? failAll;
/// Per-method errors, keyed by method name (e.g.
/// 'listDetachedRuns'). Wins over [failAll] absence; [failAll]
/// wins overall when both are set? No — per-method is checked
/// first so a scenario can single out one RPC.
final Map<String, Object> failures = {};
/// Members hit without an implementation (see header).
final Set<String> missing = {};
/// Scriptable bits of otherwise-default answers.
bool detachedEnabled = false;
bool probeServing = true;
void failWith(Object error, {Set<String>? only}) {
if (only == null) {
failAll = error;
} else {
for (final m in only) {
failures[m] = error;
}
}
}
T _guard<T>(String method, T Function() value) {
final err = failures[method] ?? failAll;
if (err != null) throw err;
return value();
}
Future<T> _async<T>(String method, T Function() value) async =>
_guard(method, value);
@override
dynamic noSuchMethod(Invocation invocation) {
missing.add(invocation.memberName.toString());
return super.noSuchMethod(invocation);
}
// ── Connection / endpoint plumbing (never fails) ─────────────
@override
String get endpointLabel => 'http://127.0.0.1:65535';
@override
HubEndpoint get currentEndpoint =>
HubEndpoint(host: '127.0.0.1', port: 65535, secure: false);
@override
Future<void> loadPersistedEndpoint() async {}
@override
Future<void> reconnect(
HubEndpoint endpoint, {
Object? authToken = const Object(),
bool persist = true,
}) async {}
@override
Future<void> reloadAuthToken() async {}
@override
Future<bool> reloadAuthTokenIfChanged() async => false;
@override
void debugResetChannel() {}
@override
Future<ThemeModeValue> loadThemeMode() async => ThemeModeValue.system;
@override
Future<void> saveThemeMode(ThemeModeValue mode) async {}
@override
Future<Locale> loadLocale() async => const Locale('de');
@override
Future<void> saveLocale(Locale locale) async {}
@override
String? get connectedChannelName => 'local';
// ── Health ───────────────────────────────────────────────────
@override
Future<bool> healthy() =>
_async('healthy', () => probeServing && failAll == null);
@override
Future<HubProbeResult> probeHealth() async {
if (failAll != null) return HubProbeResult.unreachable;
return probeServing ? HubProbeResult.serving : HubProbeResult.notServing;
}
// ── Read paths the pages hit (healthy-empty defaults) ────────
@override
Future<String> hubVersion() => _async('hubVersion', () => '0.0.0-fake');
@override
Future<List<ModuleSummary>> listModules() =>
_async('listModules', () => const []);
@override
Future<List<CapabilityInfo>> allCapabilities() =>
_async('allCapabilities', () => const []);
@override
Future<List<StoreSource>> listStores() =>
_async('listStores', () => const []);
@override
Future<List<StoreItem>> searchStore({
String query = '',
String category = '',
String tag = '',
String status = '',
int limit = 50,
}) => _async('searchStore', () => const []);
@override
Future<List<String>> installedVersions(String moduleName) =>
_async('installedVersions', () => const []);
@override
Future<List<SavedFlow>> listFlows() => _async('listFlows', () => const []);
@override
Future<List<ProjectRef>> listProjects() =>
_async('listProjects', () => const []);
@override
Stream<AuditEvent> streamEvents({
int backfill = 0,
List<String> types = const [],
String project = '',
}) {
final err = failures['streamEvents'] ?? failAll;
if (err != null) return Stream.error(err);
return const Stream.empty();
}
@override
Future<List<AuditEvent>> recentEvents({
int limit = 50,
List<String> types = const [],
String project = '',
}) => _async('recentEvents', () => const []);
@override
Future<List<PendingApproval>> pendingApprovals({String project = ''}) =>
_async('pendingApprovals', () => const []);
@override
Future<List<ApprovalRecord>> listApprovalsRecords({
List<String> statuses = const [],
int limit = 200,
String project = '',
}) => _async('listApprovalsRecords', () => const []);
@override
Future<({List<DetachedRun> runs, bool enabled})> listDetachedRuns({
String project = '',
}) => _async('listDetachedRuns', () => (runs: const <DetachedRun>[], enabled: detachedEnabled));
@override
Future<List<Satellite>> listSatellites() =>
_async('listSatellites', () => const []);
@override
Future<SystemAiStatus> systemAiStatus() => _async(
'systemAiStatus',
() => const SystemAiStatus(
enabled: false,
provider: '',
endpoint: '',
model: '',
privacyMode: 'off',
apiKeyEnv: '',
),
);
@override
Future<ChannelStatusSnapshot> channelStatus() => _async(
'channelStatus',
() => const ChannelStatusSnapshot(
active: 'local',
channels: [
ChannelInfo(
name: 'local',
port: 65535,
running: true,
endpoint: 'http://127.0.0.1:65535',
),
],
),
);
@override
Future<List<McpClientInfo>> listMcpClients() =>
_async('listMcpClients', () => const []);
@override
Future<List<N8nEndpointInfo>> listN8nEndpoints() =>
_async('listN8nEndpoints', () => const []);
@override
Future<DoctorSnapshot> doctor() => _async(
'doctor',
() => const DoctorSnapshot(
moduleCount: 0,
capabilityCount: 0,
pendingApprovals: 0,
eventChainTotal: 0,
eventChainVerified: 0,
eventChainTamperedAt: null,
services: [],
update: UpdateStatus(
channel: 'stable',
localVersion: '0.0.0-fake',
latestVersion: '0.0.0-fake',
updateAvailable: false,
manifestReachable: true,
),
paths: DaemonPathsSnapshot(
logPath: '',
dbPath: '',
modulesDir: '',
flowsDir: '',
configPath: '',
pidPath: '',
),
),
);
}
/// Install a [FakeHubService] for the duration of the current
/// test and restore a real instance afterwards.
FakeHubService installFakeHub() {
final fake = FakeHubService();
HubService.debugSetInstance(fake);
addTearDown(() => HubService.debugSetInstance(null));
return fake;
}