test: deterministic operator-guide screenshot harness

integration_test/guide_shots_test.dart boots a hermetic hub
(HubFixture), seeds demo projects (open Bürgeramt; sealed
Ratsinformation and a setup.applied audit event only when the
runner confirms an isolated $HOME), launches the app in German +
dark mode, walks every sidebar page in Cmd order with
content-aware waits, opens the workspace switcher and the setup
wizard, and writes the guide PNGs via a driverless RepaintBoundary
capture. Driven by the platform repo's scripts/regen-studio-guide.sh.

Also fixes the hub fixture's binary resolution, dead since the
rename (it looked for 'fai' and ../chain_platform/): now $CHAIN_BIN,
'chain' on PATH, then ../fai_chain/target/{release,debug}/chain —
the integration tests actually run again instead of silently
skipping.

Signed-off-by: flemming-it <stefan.a.flemming@googlemail.com>
This commit is contained in:
flemming-it 2026-07-13 00:47:41 +02:00
parent ed2c5ace26
commit cf4024a4e2
6 changed files with 346 additions and 16 deletions

View file

@ -5,6 +5,43 @@ version + `kStudioVersion` in `lib/main.dart` stay in lockstep.
## Unreleased ## Unreleased
### Added (guided setup on grade-1 — steps A4/A5/B1 + doc automation)
- **Clickable next steps.** After apply, the wizard renders real
Studio actions instead of CLI text: start-hub button (polls until
the daemon answers), per-module install buttons with progress/done
states (capability-name install via the hub's store index), and an
open-the-starter-flow button navigating to Flows.
- **Signature dead end resolved.** Regulated plans explain in plain
language that modules come from a signed source; the preview offers
"allow installing from the public store" as one deliberate,
reversible switch (`allow_unsigned_modules`) that re-assembles the
plan. Air-gapped plans point to the offline-bundle path instead.
- **Fresh-install auto-open + honest framing.** On a fresh hub (no
config, no `setup-plan.yaml`) the wizard opens by itself, once per
run; the welcome CTA is now "In 3 Fragen loslegen" / "Get started
in 3 questions". After the wizard closes, the onboarding checklist
re-probes and states what the assistant already covered (applied
profile from `setup-plan.yaml`).
- **Free-text AI path (phase 1.2).** "Or just describe what you want
to do": the goal goes to the configured system AI, the reply is
validated against strict enum whitelists and comes back as an
editable "this is how I read your task" reflection feeding the same
preview/apply. Privacy line states local vs. provider processing;
without a system AI the menu path stands alone.
- **Nav manifest guard.** `test/nav_manifest_test.dart` generates
`docs/nav.generated.json` from the sidebar truth (ids, order = Cmd
numbers, DE+EN labels); the platform repo checks the operator guide
against the mirrored copy.
- **Guide screenshot harness.** `integration_test/guide_shots_test.dart`
boots a hermetic hub, seeds demo projects, walks every nav page,
the workspace switcher and the setup wizard, and writes the guide
PNGs — driven by the platform repo's `scripts/regen-studio-guide.sh`.
- **Fixed:** the integration-test hub fixture still looked for the
pre-rename `fai` binary and `chain_platform/` path, so its tests
silently skipped since the rename; it now resolves `$CHAIN_BIN`,
`chain` on PATH, and `../fai_chain/target/{release,debug}/chain`.
### Added (multi-project, stage ③ — sealed areas) ### Added (multi-project, stage ③ — sealed areas)
- **Sealed-area connection switch.** The workspace switcher now lists - **Sealed-area connection switch.** The workspace switcher now lists

View file

@ -0,0 +1,242 @@
// Deterministic operator-guide screenshot harness (one command
// instead of the fragile cliclick/screencapture procedure).
//
// Boots a hermetic hub (HubFixture: temp data dir, free port),
// seeds deterministic demo data via the CLI (an open "Bürgeramt"
// project; optionally a sealed "Ratsinformation" when the runner
// confirms an isolated $HOME), launches the real app in German +
// dark mode against that hub, walks every sidebar destination in
// nav order, opens the workspace switcher and the guided-setup
// wizard, and writes the guide PNGs via a RepaintBoundary capture
// (driverless works headed on macOS with plain `flutter test`).
//
// Run through the platform repo's scripts/regen-studio-guide.sh,
// which isolates $HOME, builds the hub binary, and copies the
// PNGs into fai_chain/docs/studio/images/. Direct invocation:
//
// flutter test integration_test/guide_shots_test.dart -d macos
//
// Output lands in build/guide-shots/ unless GUIDE_SHOTS_OUT is
// set in the environment.
//
// Hermeticity notes:
// - SharedPreferences is mocked empty the operator's real Studio
// prefs are neither read nor written.
// - The app reconnects to the fixture hub with persist: false.
// - The sealed seed writes to $HOME/.chain/sealed/ by design, so
// it only runs when CHAIN_GUIDE_SHOTS_SEALED=1 confirms the
// caller isolated $HOME (the regen script does).
import 'dart:io';
import 'dart:ui' as ui;
import 'package:flutter/material.dart';
import 'package:flutter/rendering.dart';
import 'package:flutter/services.dart' show LogicalKeyboardKey;
import 'package:flutter_test/flutter_test.dart';
import 'package:integration_test/integration_test.dart';
import 'package:shared_preferences/shared_preferences.dart';
import 'package:chain_client_sdk/chain_client_sdk.dart' show HubEndpoint;
import 'package:chain_studio/data/hub.dart';
import 'package:chain_studio/main.dart';
import 'package:chain_studio/widgets/widgets.dart';
import '../test/integration/hub_fixture.dart';
final GlobalKey _shotKey = GlobalKey();
/// Pump a fixed number of frames instead of pumpAndSettle: the
/// sidebar clock ticks every second, so the tree never settles and
/// pumpAndSettle degenerates into its multi-minute timeout.
Future<void> _pumpFrames(WidgetTester tester, [int frames = 20]) async {
for (var i = 0; i < frames; i++) {
await tester.pump(const Duration(milliseconds: 100));
}
}
/// Pump until [finder] matches (page content arrived) or ~10 s
/// pass pages load their data from the hub asynchronously, and a
/// blind fixed wait raced the slower ones (the flows editor).
Future<void> _pumpUntil(
WidgetTester tester,
Finder finder, {
int maxFrames = 100,
}) async {
for (var i = 0; i < maxFrames; i++) {
if (finder.evaluate().isNotEmpty) return;
await tester.pump(const Duration(milliseconds: 100));
}
}
String get _outDir =>
Platform.environment['GUIDE_SHOTS_OUT'] ?? 'build/guide-shots';
/// Rasterize the app's RepaintBoundary into `<outDir>/<name>.png`.
Future<void> _shot(WidgetTester tester, String name) async {
// Two extra frames so ripples/route transitions settle visually.
await tester.pump(const Duration(milliseconds: 120));
await tester.pump(const Duration(milliseconds: 120));
final boundary =
_shotKey.currentContext!.findRenderObject() as RenderRepaintBoundary;
final image = await boundary.toImage(pixelRatio: 2.0);
final bytes = await image.toByteData(format: ui.ImageByteFormat.png);
image.dispose();
final file = File('$_outDir/$name.png');
file.parent.createSync(recursive: true);
file.writeAsBytesSync(bytes!.buffer.asUint8List());
// ignore: avoid_print
print('guide-shot: ${file.path}');
}
Future<void> _cli(
String binary,
HubFixture fixture,
List<String> args,
) async {
final r = await Process.run(
binary,
args,
environment: {
...Platform.environment,
'CHAIN_DATA_DIR': fixture.tempDir.path,
'CHAIN_MODULES_DIR': '${fixture.tempDir.path}/modules',
},
);
if (r.exitCode != 0) {
fail('seed command `chain ${args.join(' ')}` failed: ${r.stderr}');
}
}
/// The guide's image set, in sidebar (= Cmd number) order. Must
/// stay aligned with docs/nav.generated.json the nav guard in
/// the platform repo keeps the guide itself honest.
const _pageShots = [
('welcome', '01-willkommen'),
('store', '02-store'),
('doctor', '03-diagnose'),
('flows', '04-flows'),
('audit', '05-protokoll'),
('approvals', '06-freigaben'),
('runs', '07-laeufe'),
('federation', '08-foederation'),
];
void main() {
IntegrationTestWidgetsFlutterBinding.ensureInitialized();
testWidgets('capture the operator-guide screenshots', (tester) async {
final fixture = await HubFixture.start(skipIfBinaryMissing: false);
addTearDown(fixture!.dispose);
final binary = (await HubFixture.binaryPath())!;
// Deterministic demo data. The slug is explicit the derived
// one would drop the umlaut ("brgeramt").
await _cli(binary, fixture, [
'project',
'create',
'Bürgeramt',
'--slug',
'buergeramt',
'--color',
'#2e8f9e',
]);
// The sealed area writes a manifest under $HOME/.chain/sealed/,
// so it needs the runner's confirmation that $HOME is isolated.
final sealedSeed =
Platform.environment['CHAIN_GUIDE_SHOTS_SEALED'] == '1';
if (sealedSeed) {
await _cli(binary, fixture, [
'project',
'create',
'Ratsinformation',
'--slug',
'ratsinformation',
'--isolation',
'sealed',
]);
// Seed one audit event the guide can show (setup.applied) by
// applying a minimal guided setup against the fixture's data
// dir. Writes $HOME/.chain/config.yaml, hence the isolated-
// $HOME guard: only the regen script's throwaway HOME is
// ever touched.
final answers = File('${fixture.tempDir.path}/seed-answers.yaml');
answers.writeAsStringSync(
'scenario: trying-out\nintent: hello-world\ntarget: this-laptop\n',
);
await _cli(binary, fixture, [
'init',
'--answers',
answers.path,
'--apply',
'--force',
]);
}
// Hermetic prefs: never read or write the operator's real ones.
SharedPreferences.setMockInitialValues({});
await tester.pumpWidget(
RepaintBoundary(
key: _shotKey,
child: const StudioApp(
initialThemeMode: ThemeModeValue.dark,
initialLocale: Locale('de'),
),
),
);
await HubService.instance.reconnect(
HubEndpoint(host: '127.0.0.1', port: fixture.port),
authToken: null,
persist: false,
);
await _pumpFrames(tester, 30);
final shell =
tester.state<StudioShellState>(find.byType(StudioShell));
// 0108: every sidebar destination, in Cmd order. Where a page
// loads slower content from the hub, wait for a sentinel that
// proves the content arrived before capturing.
for (final (id, shotName) in _pageShots) {
shell.navigateTo(id);
await _pumpFrames(tester);
if (id == 'flows') {
// The imported sample flows populate the file list.
await _pumpUntil(tester, find.textContaining('extract-summarize'));
} else if (id == 'audit') {
// The seeded setup.applied event (isolated-$HOME runs).
await _pumpUntil(tester, find.textContaining('setup.applied'));
}
await _shot(tester, shotName);
}
// 09: the workspace switcher, opened (shows the demo projects;
// with the sealed seed also the shield entry).
shell.navigateTo('audit');
await _pumpFrames(tester);
final switcher = find.byType(ChainWorkspaceSwitcher);
if (switcher.evaluate().isNotEmpty) {
await tester.tap(switcher.first);
await _pumpFrames(tester);
await _shot(tester, '09-projekt-umschalter');
// Close the menu again before moving on.
await tester.sendKeyEvent(LogicalKeyboardKey.escape);
await _pumpFrames(tester, 5);
}
// 11: the guided-setup wizard, step 1 (localized option cards).
// 10 (inside a sealed area) still needs the manual procedure
// it requires switching to a second, isolated hub instance.
shell.navigateTo('welcome');
await _pumpFrames(tester);
final cta = find.text('In 3 Fragen loslegen');
if (cta.evaluate().isNotEmpty) {
await tester.tap(cta.first);
await _pumpFrames(tester);
await _shot(tester, '11-einrichtung');
await tester.sendKeyEvent(LogicalKeyboardKey.escape);
await _pumpFrames(tester, 5);
}
});
}

View file

@ -13,4 +13,4 @@ SPEC CHECKSUMS:
PODFILE CHECKSUM: 54d867c82ac51cbd61b565781b9fada492027009 PODFILE CHECKSUM: 54d867c82ac51cbd61b565781b9fada492027009
COCOAPODS: 1.16.2 COCOAPODS: 1.17.0

View file

@ -172,6 +172,11 @@ packages:
url: "https://pub.dev" url: "https://pub.dev"
source: hosted source: hosted
version: "0.3.5" version: "0.3.5"
flutter_driver:
dependency: transitive
description: flutter
source: sdk
version: "0.0.0"
flutter_highlight: flutter_highlight:
dependency: transitive dependency: transitive
description: description:
@ -219,6 +224,11 @@ packages:
description: flutter description: flutter
source: sdk source: sdk
version: "0.0.0" version: "0.0.0"
fuchsia_remote_debug_protocol:
dependency: transitive
description: flutter
source: sdk
version: "0.0.0"
google_cloud: google_cloud:
dependency: transitive dependency: transitive
description: description:
@ -291,6 +301,11 @@ packages:
url: "https://pub.dev" url: "https://pub.dev"
source: hosted source: hosted
version: "4.1.2" version: "4.1.2"
integration_test:
dependency: "direct dev"
description: flutter
source: sdk
version: "0.0.0"
intl: intl:
dependency: "direct main" dependency: "direct main"
description: description:
@ -435,6 +450,14 @@ packages:
url: "https://pub.dev" url: "https://pub.dev"
source: hosted source: hosted
version: "2.1.8" version: "2.1.8"
process:
dependency: transitive
description:
name: process
sha256: c6248e4526673988586e8c00bb22a49210c258dc91df5227d5da9748ecf79744
url: "https://pub.dev"
source: hosted
version: "5.0.5"
protobuf: protobuf:
dependency: transitive dependency: transitive
description: description:
@ -544,6 +567,14 @@ packages:
url: "https://pub.dev" url: "https://pub.dev"
source: hosted source: hosted
version: "1.4.1" version: "1.4.1"
sync_http:
dependency: transitive
description:
name: sync_http
sha256: "7f0cd72eca000d2e026bcd6f990b81d0ca06022ef4e32fb257b30d3d1014a961"
url: "https://pub.dev"
source: hosted
version: "0.3.1"
term_glyph: term_glyph:
dependency: transitive dependency: transitive
description: description:
@ -664,6 +695,14 @@ packages:
url: "https://pub.dev" url: "https://pub.dev"
source: hosted source: hosted
version: "1.1.1" version: "1.1.1"
webdriver:
dependency: transitive
description:
name: webdriver
sha256: "2f3a14ca026957870cfd9c635b83507e0e51d8091568e90129fbf805aba7cade"
url: "https://pub.dev"
source: hosted
version: "3.1.0"
win32: win32:
dependency: transitive dependency: transitive
description: description:

View file

@ -50,6 +50,8 @@ dependencies:
dev_dependencies: dev_dependencies:
flutter_test: flutter_test:
sdk: flutter sdk: flutter
integration_test:
sdk: flutter
flutter_lints: ^6.0.0 flutter_lints: ^6.0.0
# Local-development override: while editing the swappable flow # Local-development override: while editing the swappable flow

View file

@ -71,13 +71,13 @@ class HubFixture {
if (binary == null) { if (binary == null) {
if (skipIfBinaryMissing) { if (skipIfBinaryMissing) {
markTestSkipped( markTestSkipped(
'Hub fixture skipped: no `fai` binary found on PATH or in ' 'Hub fixture skipped: no `chain` binary found via \$CHAIN_BIN, '
'../chain_platform/target/release/fai. Build it with ' 'PATH, or ../fai_chain/target/{release,debug}/chain. Build it '
'`cargo build --release` and re-run.', 'with `cargo build` in the platform repo and re-run.',
); );
return null; return null;
} }
throw StateError('fai binary not found'); throw StateError('chain binary not found');
} }
final tempDir = await Directory.systemTemp.createTemp('chain_studio_test_'); final tempDir = await Directory.systemTemp.createTemp('chain_studio_test_');
@ -157,27 +157,37 @@ class HubFixture {
} }
} }
/// Look for `fai` first on PATH (production-ish), then in /// Resolved hub binary path, for tests that also drive the CLI
/// the standard `target/release/` build output relative to /// against the fixture's data dir (e.g. seeding projects).
/// the platform checkout (developer flow). Returns null when static Future<String?> binaryPath() => _resolveBinary();
/// neither path resolves.
/// Look for the hub binary: $CHAIN_BIN override first, then
/// `chain` on PATH (production-ish), then the release/debug
/// build outputs relative to the platform checkout (developer
/// flow, repo dir `fai_chain`). Returns null when nothing
/// resolves.
static Future<String?> _resolveBinary() async { static Future<String?> _resolveBinary() async {
final fromEnv = Platform.environment['CHAIN_BIN'];
if (fromEnv != null && fromEnv.isNotEmpty && File(fromEnv).existsSync()) {
return fromEnv;
}
final pathResult = await Process.run( final pathResult = await Process.run(
Platform.isWindows ? 'where' : 'which', Platform.isWindows ? 'where' : 'which',
['fai'], [Platform.isWindows ? 'chain.exe' : 'chain'],
); );
if (pathResult.exitCode == 0) { if (pathResult.exitCode == 0) {
final s = (pathResult.stdout as String).trim(); final s = (pathResult.stdout as String).trim();
if (s.isNotEmpty) return s.split('\n').first.trim(); if (s.isNotEmpty) return s.split('\n').first.trim();
} }
final candidate = Platform.isWindows final exe = Platform.isWindows ? 'chain.exe' : 'chain';
? '../chain_platform/target/release/fai.exe' for (final dir in ['release', 'debug']) {
: '../chain_platform/target/release/fai'; final f = File('../fai_chain/target/$dir/$exe');
final f = File(candidate);
if (await f.exists()) { if (await f.exists()) {
return f.absolute.path; return f.absolute.path;
} }
}
return null; return null;
} }