Skip to content

Commit

Permalink
Fix Flutter test golden threshold
Browse files Browse the repository at this point in the history
  • Loading branch information
Afroz-Shaikh committed May 6, 2024
1 parent 71bd731 commit 6c27998
Show file tree
Hide file tree
Showing 258 changed files with 66 additions and 20 deletions.
2 changes: 1 addition & 1 deletion example/pubspec.lock
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ packages:
path: ".."
relative: true
source: path
version: "1.0.2"
version: "1.0.3"
leak_tracker:
dependency: transitive
description:
Expand Down
1 change: 1 addition & 0 deletions test/custom_curve_test/custom_curve_test.dart
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ class MyCustomCurveGauge extends StatelessWidget {
Widget build(BuildContext context) {
return MaterialApp(
theme: ThemeData(
useMaterial3: false,
fontFamily: 'Roboto',
),
home: Scaffold(
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified test/custom_curve_test/goldens/custom-curve-midpoint-50-top.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
1 change: 1 addition & 0 deletions test/custom_labels_test/custom_labels_test.dart
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ class MyPointerTestLinearGauge extends StatelessWidget {
Widget build(BuildContext context) {
return MaterialApp(
theme: ThemeData(
useMaterial3: false,
fontFamily: 'Roboto',
),
home: Scaffold(
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified test/custom_labels_test/goldens/extend-linear-gauge-25.png
Binary file modified test/custom_labels_test/goldens/long-labels-Horizontal.png
Binary file modified test/custom_labels_test/goldens/long-labels-Vertical.png
Binary file modified test/custom_labels_test/goldens/multiple-labels-vertical.png
Binary file modified test/custom_labels_test/goldens/no-labels-horizontal.png
Binary file modified test/custom_labels_test/goldens/no-labels-vertical.png
Binary file modified test/custom_labels_test/goldens/ruler-position-bottom.png
Binary file modified test/custom_labels_test/goldens/ruler-position-left.png
Binary file modified test/custom_labels_test/goldens/ruler-position-right.png
Binary file modified test/custom_labels_test/goldens/ruler-position-top.png
4 changes: 1 addition & 3 deletions test/extend_linear_gauge_test/extend_linear_gauge_test.dart
Original file line number Diff line number Diff line change
Expand Up @@ -43,9 +43,7 @@ class MyExtendLinearGaugeTest extends StatelessWidget {
@override
Widget build(BuildContext context) {
return MaterialApp(
theme: ThemeData(
fontFamily: 'Roboto',
),
theme: ThemeData(fontFamily: 'Roboto', useMaterial3: false),
home: Scaffold(
appBar: AppBar(
title: const Text("Default LinearGauge"),
Expand Down
Binary file modified test/extend_linear_gauge_test/goldens/extend-linear-gauge-0.png
Binary file modified test/extend_linear_gauge_test/goldens/extend-linear-gauge-50.png
73 changes: 57 additions & 16 deletions test/flutter_test_config.dart
Original file line number Diff line number Diff line change
@@ -1,22 +1,63 @@
import 'dart:async';
import 'dart:io';

import 'package:golden_toolkit/golden_toolkit.dart';
import 'package:flutter/foundation.dart';
import 'package:flutter_test/flutter_test.dart';

// All tests in this folder and its subfolders will have the configuration defined here
const _kGoldenTestsThreshold = 10 / 100; // 1% tolerance

Future<void> testExecutable(FutureOr<void> Function() testMain) async {
return GoldenToolkit.runWithConfiguration(
() async {
await loadAppFonts();
await testMain();
},
config: GoldenToolkitConfiguration(
// Currently, goldens are not generated/validated in CI for this repo. We have settled on the goldens for this package
// being captured/validated by developers running on MacOSX. We may revisit this in the future if there is a reason to invest
// in more sophistication
skipGoldenAssertion: () => !Platform.isMacOS,
enableRealShadows: true,
),
);
if (goldenFileComparator is LocalFileComparator) {
final testUrl = (goldenFileComparator as LocalFileComparator).basedir;

goldenFileComparator = LocalFileComparatorWithThreshold(
Uri.parse('$testUrl/test. dart'),
_kGoldenTestsThreshold,
);
} else {
throw Exception(
'Expected goldenFileComparator to be of type'
'LocalFileComparator'
'but it is of type ${goldenFileComparator.runtimeType}`',
);
}
await testMain();
}

/// Works just like [LocalFileComparator] but includes a [threshold] that, when
/// exceeded, marks the test as a failure.
class LocalFileComparatorWithThreshold extends LocalFileComparator {
/// Threshold above which tests will be marked as failing.
/// Ranges from 0 to 1, both inclusive.
final double threshold;

LocalFileComparatorWithThreshold(Uri testFile, this.threshold)
: assert(threshold >= 0 && threshold <= 1),
super(testFile);

/// Copy of [LocalFileComparator]'s [compare] method, except for the fact that
/// it checks if the [ComparisonResult.diffPercent] is not greater than
/// [threshold] to decide whether this test is successful or a failure.
@override
Future<bool> compare(Uint8List imageBytes, Uri golden) async {
final result = await GoldenFileComparator.compareLists(
imageBytes,
await getGoldenBytes(golden),
);

if (!result.passed && result.diffPercent <= threshold) {
print(
'A difference of ${result.diffPercent * 100}% was found, but it is '
'acceptable since it is not greater than the threshold of '
'${threshold * 100}%',
);

return true;
}

if (!result.passed) {
final error = await generateFailureOutput(result, golden, basedir);
throw FlutterError(error);
}
return result.passed;
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ class MyLinearGaugeContainer extends StatelessWidget {
return MaterialApp(
theme: ThemeData(
fontFamily: 'Roboto',
useMaterial3: false,
),
home: Scaffold(
appBar: AppBar(
Expand Down
Binary file modified test/pointers_test/goldens/align-start-bottom-diamond-value-30.png
Binary file modified test/pointers_test/goldens/bottom-circle-value-30-inverse.png
Binary file modified test/pointers_test/goldens/bottom-diamond-value-30-inverse.png
Binary file modified test/pointers_test/goldens/bottom-rectangle-value-30-inverse.png
Binary file modified test/pointers_test/goldens/bottom-triangle-value-30-inverse.png
Binary file modified test/pointers_test/goldens/center-circle-value-30-inverse.png
Binary file modified test/pointers_test/goldens/center-diamond-value-30-inverse.png
Binary file modified test/pointers_test/goldens/center-rectangle-value-30-inverse.png
Binary file modified test/pointers_test/goldens/center-triangle-value-30-inverse.png
Binary file modified test/pointers_test/goldens/circle-color-blue.png
Binary file modified test/pointers_test/goldens/circle-value-0-width-150.png
Binary file modified test/pointers_test/goldens/circle-value-0.png
Binary file modified test/pointers_test/goldens/circle-value-100.png
Binary file modified test/pointers_test/goldens/circle-value-50.png
Binary file modified test/pointers_test/goldens/diamond-color-blue.png
Binary file modified test/pointers_test/goldens/diamond-value-0-height-150.png
Binary file modified test/pointers_test/goldens/diamond-value-0-width-150-vertical.png
Binary file modified test/pointers_test/goldens/diamond-value-0-width-150.png
Binary file modified test/pointers_test/goldens/diamond-value-0.png
Binary file modified test/pointers_test/goldens/diamond-value-100.png
Binary file modified test/pointers_test/goldens/diamond-value-50.png
Binary file modified test/pointers_test/goldens/rectangle-color-blue.png
Binary file modified test/pointers_test/goldens/rectangle-value-0-height-150.png
Binary file modified test/pointers_test/goldens/rectangle-value-0-width-150.png
Binary file modified test/pointers_test/goldens/rectangle-value-0.png
Binary file modified test/pointers_test/goldens/rectangle-value-100.png
Binary file modified test/pointers_test/goldens/rectangle-value-50.png
Binary file modified test/pointers_test/goldens/showLabel-false-circle-value-0.png
Binary file modified test/pointers_test/goldens/showLabel-false-circle-value-100.png
Binary file modified test/pointers_test/goldens/showLabel-false-diamond-value-0.png
Binary file modified test/pointers_test/goldens/showLabel-false-diamond-value-100.png
Binary file modified test/pointers_test/goldens/showLabel-false-rectangle-value-0.png
Binary file modified test/pointers_test/goldens/showLabel-false-rectangle-value-100.png
Binary file modified test/pointers_test/goldens/showLabel-false-triangle-value-0.png
Binary file modified test/pointers_test/goldens/showLabel-false-triangle-value-100.png
Binary file modified test/pointers_test/goldens/top-circle-value-30-inverse.png
Binary file modified test/pointers_test/goldens/top-diamond-value-30-inverse.png
Binary file modified test/pointers_test/goldens/top-rectangle-value-30-inverse.png
Binary file modified test/pointers_test/goldens/top-triangle-value-30-inverse.png
Binary file modified test/pointers_test/goldens/triangle-color-blue.png
Binary file modified test/pointers_test/goldens/triangle-value-0-height-150.png
Binary file modified test/pointers_test/goldens/triangle-value-0-width-150.png
Binary file modified test/pointers_test/goldens/triangle-value-0.png
Binary file modified test/pointers_test/goldens/triangle-value-100.png
Binary file modified test/pointers_test/goldens/triangle-value-50.png
Binary file modified test/pointers_test/goldens/value-0-label-triangle.png
Binary file modified test/pointers_test/goldens/value-100-label-triangle-hw-100.png
Binary file modified test/pointers_test/goldens/value-100-label-triangle.png
Binary file modified test/pointers_test/goldens/value-50-label-triangle.png
1 change: 1 addition & 0 deletions test/pointers_test/pointers_test.dart
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ class MyPointerTestLinearGauge extends StatelessWidget {
return MaterialApp(
theme: ThemeData(
fontFamily: 'Roboto',
useMaterial3: false,
),
home: Scaffold(
appBar: AppBar(
Expand Down
Binary file modified test/range_linear_gauge_test/goldens/range-linear-gauge-extend.png
1 change: 1 addition & 0 deletions test/range_linear_gauge_test/range_linear_test.dart
Original file line number Diff line number Diff line change
Expand Up @@ -65,6 +65,7 @@ class MyRangeLinearGauge extends StatelessWidget {
return MaterialApp(
theme: ThemeData(
fontFamily: 'Roboto',
useMaterial3: false,
),
home: Scaffold(
appBar: AppBar(
Expand Down
1 change: 1 addition & 0 deletions test/shader_paint_test/shader_paint_test.dart
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ class MyValuBarTestLinearGauge extends StatelessWidget {
return MaterialApp(
theme: ThemeData(
fontFamily: 'Roboto',
useMaterial3: false,
),
home: Scaffold(
appBar: AppBar(
Expand Down
Binary file modified test/value_bar_test/goldens/both-edge-border-radius-value-bar.png
Binary file modified test/value_bar_test/goldens/end-edge-border-radius-value-bar.png
Binary file modified test/value_bar_test/goldens/multi-value-bar-value-100.png
Binary file modified test/value_bar_test/goldens/multi-value-bar-value-50-60-100.png
Binary file modified test/value_bar_test/goldens/start-edge-border-radius-value-bar.png
Binary file modified test/value_bar_test/goldens/value-bar-value-0-vertical.png
Binary file modified test/value_bar_test/goldens/value-bar-value-0.png
Binary file modified test/value_bar_test/goldens/value-bar-value-100-vertical.png
Binary file modified test/value_bar_test/goldens/value-bar-value-100.png
Binary file modified test/value_bar_test/goldens/value-bar-value-35-inverse.png
Binary file modified test/value_bar_test/goldens/value-bar-value-35-vertical.png
Binary file modified test/value_bar_test/goldens/value-bar-value-35.png
Binary file modified test/value_bar_test/goldens/value-bar-value-50-vertical.png
Binary file modified test/value_bar_test/goldens/value-bar-value-50.png
1 change: 1 addition & 0 deletions test/value_bar_test/value_bar_test.dart
Original file line number Diff line number Diff line change
Expand Up @@ -52,6 +52,7 @@ class MyValuBarTestLinearGauge extends StatelessWidget {
return MaterialApp(
theme: ThemeData(
fontFamily: 'Roboto',
useMaterial3: false,
),
home: Scaffold(
appBar: AppBar(
Expand Down

0 comments on commit 6c27998

Please sign in to comment.