Skip to content

Commit

Permalink
Javadoc changes
Browse files Browse the repository at this point in the history
  • Loading branch information
stanbrub committed Oct 21, 2024
1 parent 69e9078 commit 1d70bc6
Show file tree
Hide file tree
Showing 30 changed files with 82 additions and 46 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,12 @@
* A wrapper for the Bench api that allows running tests for the purpose of comparing Deephaven to other products that
* perform similar operations. It allows running Deephaven operations or using Deephaven as an agent to run command line
* python tests in the same environment (e.g. Docker).
* <p/>
* <p>
* One of two initializers must be called to set up which type of tests is desired; {@code initDeephaven()} or
* {@code initPython()}. Deephaven tests run queries inside of Deephaven like the standard benchmarks. Python tests use
* Deephaven as an agent to run python scripts from the command line by first installing required pip modules in a
* python virtual environment and then running each test from there.
* <p/>
* <p>
* Note: This runner requires test ordering, so it follows that tests in a single test class are meant to be run as a
* group. This violates the standard Benchmark convention that every test be able to be run by itself. This is done for
* practical purposes, though it is not ideal.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@
* Product comparison tests for the average by group operation. Tests read the same parquet data. To avoid an unfair
* advantage where some products may partition or group data during the read, parquet read time is included in the
* benchmark results.
* <p/>
* <p>
* Each test calculates two new average columns and groups by a string and an integer.
* <p/>
* <p>
* Data generation only happens in the first tests, the Deephaven test. Tests can be run individually, but only after
* the desired data has been generated.
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@
* Product comparison tests for the distinct (or select distinct) group operation. Tests read the same parquet data. To
* avoid an unfair advantage where some products may partition or group data during the read, parquet read time is
* included in the benchmark results.
* <p/>
* <p>
* Each test produces a table result that contains rows unique according to a string and an integer.
* <p/>
* <p>
* Data generation only happens in the first tests, the Deephaven test. Tests can be run individually, but only after
* the desired data has been generated.
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,10 +10,10 @@
* Product comparison tests for filter (where) operations. Tests read the same parquet data. To avoid an unfair
* advantage where some products may partition or group data during the read, parquet read time is included in the
* benchmark results.
* <p/>
* <p>
* Each test produces a table result filtered by three criteria; value is an exact string, value > an integer, value <
* an integer.
* <p/>
* <p>
* Data generation only happens in the first tests, the Deephaven test. Tests can be run individually, but only after
* the desired data has been generated.
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@
* Product comparison tests for inner join operations. Tests read the same parquet data. To avoid an unfair advantage
* where some products may partition or group data during the read, parquet read time is included in the benchmark
* results.
* <p/>
* <p>
* Each test produces a table that is the result of two tables intersected by a string and an integer.
* <p/>
* <p>
* Data generation only happens in the first tests, the Deephaven test. Tests can be run individually, but only after
* the desired data has been generated.
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@
/**
* Product comparison tests for sort operations. Tests read the same parquet data. To avoid an unfair advantage where
* some products may partition or group data during the read, parquet read time is included in the benchmark results.
* <p/>
* <p>
* Each test sorts a table by a string and an integer.
* <p/>
* <p>
* Data generation only happens in the first tests, the Deephaven test. Tests can be run individually, but only after
* the desired data has been generated.
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
* boilerplate logic like imports, parquet reads, time measurement logic, etc. Each <code>test</code> runs two tests;
* one reading from a static parquet, and the other exercising ticking tables through the
* <code>AutotuningIncrementalReleaseFilter</code>.
* <p/>
* <p>
* Note: This class is for running tests in the <code>experimental</code> package. It will change as new experiments are
* added and may require external setup (i.e. parquet files) to work.
*/
Expand Down Expand Up @@ -112,7 +112,7 @@ public void table(String name, long rowCount) {
* Run the benchmark test according to the operation and the columns loaded from the source table. The name will
* show in the benchmark result output. The expected row count, since tests can scale, is an upper bound what result
* row count is expected.
* <p/>
* <p>
* This method assembles and runs two queries according to the settings provided previously: static and incremental
* release. Both runs are expected to produce the same resulting row count.
*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
* This tests a sort operation based on generated rows of data. The expected row count is achieved by fully generating
* the data to a parquet file or partially generating the data and merging that by <code>tableFactor</code> to get row
* count.
* <p/>
* <p>
* Note: For best results, use base and row counts that are highly divisible and clear like 1,000,000 so that the
* <code>tableFactor</code> is a whole number.
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ public void setup() {
* </ol>
* This test is identical to {@link #joinTwoTablesFromKafkaStream_Shorthand} except without the use of
* <code>bench_api_</code> functions for Kafka consumers and table waiting.
* <p/>
* <p>
* Properties (e.g. ${kafka.consumer.addr}) are automatically filled in during query execution.
*/
@Test
Expand Down Expand Up @@ -114,7 +114,7 @@ with exclusive_lock(table):
* </ol>
* This test is identical to {@link #joinTwoTablesFromKafkaStream_Longhand} except without the use of
* <code>bench_api_</code> functions for Kafka consumers and table waiting.
* <p/>
* <p>
* Properties (e.g. ${kafka.consumer.addr}) are automatically filled in during query execution.
*/
@Test
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ public void test(String name, String operation, String... loadColumns) {
* <li>If static test duration <code>&lt; scale.elapsed.time.target</code>, scale row count and do it again</li>
* <li>Run test with auto increment release filter according to the previously determined row count</li>
* <li>Assert that both static and incremental result tables have the same number of rows</li>
* <p/>
* <p>
*
* @param name the name of the test as it will show in the result file
* @param expectedRowCount the max row count expected from the operation regardless of scale, or zero if the count
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
/**
* Standard tests for the groupBy table operation. ungroups column content. It is the inverse of the group_by method.
* Ungroup unwraps columns containing either Deephaven arrays or java arrays.
* <p/>
* <p>
* Note: These test do group then ungroup, since the data generator does not support arrays
*/
public class UngroupTest {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,10 @@

/**
* Standard tests for running user-defined functions. These tests are meant to be compared, and so use the same data.
* <p/>
* <p>
* Note: When scaling row count, vector size should not get bigger. That would cause more than one axis change and
* invalidate any expected comparisons.
* <p/>
* <p>
* Note: The "NoHints" tests do have return hints to make them equivalent to the hints tests, otherwise the return value
* would always be a PyObject and not really the same test.
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
/**
* Standard tests for the updateBy table operation. Calculates a tick-based exponential moving maximum for specified
* columns and places the result into a new column for each row.
* <p/>
* <p>
* Note: When there are no Group Keys, EmMaxTick has a much faster rate than EmMinTick. This is likely because of branch
* prediction on x86 systems. This disparity does not happen on Mac M1.
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
/**
* Standard tests for the updateBy table operation. Calculates a time-based exponential moving maximum for specified
* columns and places the result into a new column for each row.
* <p/>
* <p>
* Note: When there are no Group Keys, EmMaxTime has a much faster rate than EmMinTime. This is likely because of branch
* prediction on x86 systems. This disparity does not happen on Mac M1.
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
/**
* Standard tests for the updateBy table operation. Calculates a tick-based exponential moving minimum for specified
* columns and places the result into a new column for each row.
* <p/>
* <p>
* Note: When there are no Group Keys, EmMinTick has a much slower rate than EmMaxTick. This is likely because of branch
* prediction on x86 systems. This disparity does not happen on Mac M1.
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
/**
* Standard tests for the updateBy table operation. Calculates a time-based exponential moving minimum for specified
* columns and places the result into a new column for each row.
* <p/>
* <p>
* Note: When there are no Group Keys, EmMinTime has a much slower rate than EmMaxTime. This is likely because of branch
* prediction on x86 systems. This disparity does not happen on Mac M1.
*/
Expand Down
28 changes: 26 additions & 2 deletions src/main/java/io/deephaven/benchmark/api/Bench.java
Original file line number Diff line number Diff line change
Expand Up @@ -17,19 +17,43 @@

/**
* The root accessor class for the API. Use <code>Bench.create(this)</code> in a typical JUnit test to start things off
* <p/>
* <p>
* Bench API methods are not thread-safe, nor are they intended to be. It makes no sense to run benchmark tests in
* parallel. If parallel tests are desired to shorten overall test time, use the standalone uber-jar and select separate
* sets of test packages to run on different systems simultaneously.
*/
final public class Bench {
/**
* The root benchmark result directory
*/
static final public Path rootOutputDir = Paths.get("results");
/**
* The name of the benchmark results csv file
*/
static final public String resultFileName = "benchmark-results.csv";
/**
* The name of the benchmark metrics csv file
*/
static final public String metricsFileName = "benchmark-metrics.csv";
/**
* The name of the benchmark platform csv file
*/
static final public String platformFileName = "benchmark-platform.csv";
/**
* The profile that includes the properties defined in the properties file, system and startup
*/
static final Profile profile = new Profile();
/**
* The benchmark output directory for the current run
*/
static final public Path outputDir = initializeOutputDirectory();

/**
* The starting point for the Bench API
*
* @param testInst the test case instance in operation
* @return the root of the Bench API for the given test
*/
static public Bench create(Object testInst) {
Bench v = new Bench(testInst.getClass());
v.setName(testInst.getClass().getSimpleName());
Expand Down Expand Up @@ -105,7 +129,7 @@ public boolean propertyAsBoolean(String name, String defaultValue) {
/**
* Get an integral property from the profile, System, Environment or return a default value. Values are specified to
* match the following regular expression:
* <p/>
* <p>
* <code>[0-9]+ ( nanos | nano | millis | milli | seconds | second | minutes | minute )</code>
*
* @param name the property name
Expand Down
4 changes: 4 additions & 0 deletions src/main/java/io/deephaven/benchmark/api/BenchLog.java
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ public void close() {
/**
* Add the log info that was collected during the test run
*
* @param origin where the log message came from
* @param info the log info (i.e. the docker log)
*/
public void add(String origin, String info) {
Expand All @@ -53,6 +54,9 @@ public void add(String origin, String info) {

/**
* Set the name of the current test. This will be used at the beginning and end of the test's log info.
* <p>
* Note: The symbol "#" is used is some components like <code>QueryLog</code> to treat with special behavior. In
* this log it is removed and treated like any other test heading.
*
* @param name the name of the current log section (i.e. test name)
*/
Expand Down
8 changes: 8 additions & 0 deletions src/main/java/io/deephaven/benchmark/api/BenchPlatform.java
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,14 @@ public class BenchPlatform {
this.platformFile = parent.resolve(platformFileName);
}

/**
* Add a platform detail
*
* @param origin where the platform detail came from
* @param name name of the platform detail
* @param value value of the platform detail
* @return this instance
*/
public BenchPlatform add(String origin, String name, Object value) {
benchApiAddProperty(properties, origin, name, value);
return this;
Expand Down
4 changes: 2 additions & 2 deletions src/main/java/io/deephaven/benchmark/api/BenchResult.java
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ final public class BenchResult {
/**
* Record a test rate for this result instance
*
* @param the place where the measurement was collected
* @param origin the place where the measurement was collected
* @param timer a started timer measuring the test
* @param count the processed item count (e.g. rowCount)
* @return this result instance
Expand All @@ -50,7 +50,7 @@ public BenchResult test(String origin, Timer timer, long count) {
* Record a test rate for this result instance
*
* @param origin the place where the measurement was collected
* @param timer a started timer measuring the test
* @param duration the time it took the test to run
* @param count the processed item count (e.g. rowCount)
* @return this result instance
*/
Expand Down
2 changes: 1 addition & 1 deletion src/main/java/io/deephaven/benchmark/api/BenchTable.java
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ public BenchTable withCompression(String codec) {
* Direct the table generator to produce column values according to an incremental distribution and rows up the
* maximum defined by all column ranges. For example, if col1 has range [1-10] and col2 has range [1-20] the total
* number of rows generated will be 20, unless {@code withRowCount()} is used to override it.
* <p/>
* <p>
* Calling this method will override the default of fixed = false and distribution = random.
*
* @return this instance
Expand Down
2 changes: 1 addition & 1 deletion src/main/java/io/deephaven/benchmark/api/Profile.java
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ boolean propertyAsBoolean(String name, String defaultValue) {
/**
* Get the value of the given property name as a Duration. Supported unit types are; nano, milli, second, minute.
* Fractional amounts are not supported.
* <p/>
* <p>
* ex. 100000 nanos, 10 millis, 1 second, 5 minutes
*
* @param name a property name
Expand Down
8 changes: 4 additions & 4 deletions src/main/java/io/deephaven/benchmark/api/Snippets.java
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
class Snippets {
/**
* Provides a consumer to a kafka topic according to the APIs properties (e.g. kafka.consumer.addr)
* <p/>
* <p>
* ex. mytable = bench_api_kafka_consume('mytopic', 'append')
*
* @param topic a kafka topic name
Expand Down Expand Up @@ -35,7 +35,7 @@ def bench_api_kafka_consume(topic: str, table_type: str):
/**
* Captures table size every Deephaven ticking interval and does not allow advancement in the current query logic
* until the given table size is reached
* <p/>
* <p>
* ex. bench_api_await_table_size(table, 1000000)
*
* @param table the table to monitor
Expand All @@ -55,7 +55,7 @@ with exclusive_lock(table):
/**
* Captures the value of the first column in a table every Deephaven ticking interval and does not allow advancement
* in the current query logic until that value is reached
* <p/>
* <p>
* ex. bench_api_await_column_value_limit(table, 'count', 1000000)
*
* @param table the table to monitor
Expand Down Expand Up @@ -105,7 +105,7 @@ def add(bean, getters: [], notes = ''):

/**
* Collect any snapshots of metrics and turn them into a Deephaven table that can be fetched from the bench api.
* <p/>
* <p>
* ex. bench_api_metrics_table = bench_api_collect()
*/
static String bench_api_metrics_collect = """
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
/**
* Client that communicates with the Deephaven Server, allows queries to be executed, and results to be retrieved. At
* present, this connector only supports python queries.
* <p/>
* <p>
* The typical workflow will be initialize connection, execute query, fetch results, close. Note: This class is meant to
* be used through the Bench api rather than directly.
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
* Create an in-memory table from either CSV or a Deephaven table. Provides some basic accessors for getting column
* values. No data typing is done on import of the data. Use typed methods like {@code getNumber()} to convert from
* whatever row value came from the import.
* <p/>
* <p>
* Note: This class is not a general purpose class for reading CSV or Deephaven Table data. It fits specific cases used
* by the Benchmark framework.
*/
Expand Down
4 changes: 2 additions & 2 deletions src/main/java/io/deephaven/benchmark/connect/ResultTable.java
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
/**
* Table used to fetch during or after executing a query through a connector (e.g. <code>BarrageConnector</code>) or
* through the Bench API
* <p/>
* <p>
* ex. api.query(query).fetchAfter("myTableName", table -> { // do something }).execute();
*/
public interface ResultTable {
Expand Down Expand Up @@ -81,7 +81,7 @@ public interface ResultTable {
/**
* Return a string containing the Csv representation of this table. Allow setting column justification to right or
* left depending on the 'R' or 'L' specified in the alignment description.
* <p/>
* <p>
* ex. "|", "LRRRRL" where there are 6 columns in the data
*
* @param delim the delimiter to use between column
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
/**
* Represents a mechanism that can manage the external service (e.g. Deephaven Engine) the benchmarks are running
* against. This includes; start, stop, logging, etc.
* <p/>
* <p>
* Note: For now, this is not part of the Bench API but is used by runners that wrap the Bench API to provide normalized
* behavior or generated data reuse.
*/
Expand Down
Loading

0 comments on commit 1d70bc6

Please sign in to comment.