Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

C ABI target #4793

Merged
merged 48 commits into from
Apr 12, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
7e6deb2
Add C ABI target with support for iOS and Android
ospfranco Mar 15, 2024
3b0948d
Start undoing merge problems
ospfranco Mar 15, 2024
ece7da3
Undo merge frocks
ospfranco Mar 15, 2024
ba23979
Undo merge frocks
ospfranco Mar 15, 2024
dc73adb
Undo merge frocks
ospfranco Mar 15, 2024
a49b96a
Various fixes
ospfranco Mar 15, 2024
1456a36
Update query-engine/query-engine-c-abi/README.md
ospfranco Mar 15, 2024
c09dbb9
Update README
ospfranco Mar 15, 2024
0edea72
Lint
ospfranco Mar 19, 2024
77652cd
Safety comments
ospfranco Mar 19, 2024
22c5ed7
Shellcheck
ospfranco Mar 19, 2024
2d2b088
PR changes
ospfranco Mar 20, 2024
b666f9c
Update query-engine/query-engine-c-abi/README.md
ospfranco Mar 20, 2024
4055d1d
PR comments
ospfranco Mar 20, 2024
f61fd34
Get rid of functions file
ospfranco Mar 20, 2024
7cc1932
Remove comments
ospfranco Mar 20, 2024
88ec9b1
Replace all custom errors with the common ApiError
ospfranco Mar 21, 2024
b69f07c
Remove hanging TODO
ospfranco Mar 21, 2024
3a7a7d6
Remove unnecessary from serde error
ospfranco Mar 21, 2024
1f02632
Add failed at field to migrations table and throw on failed migration
ospfranco Mar 21, 2024
43cd408
Update query-engine/query-engine-c-abi/src/engine.rs
ospfranco Mar 22, 2024
d607828
Update react native tests to take real emulator url
ospfranco Mar 22, 2024
3e5f92e
Return null adapter on RN case and handle null case
ospfranco Mar 22, 2024
f283add
Fix tests
Mar 22, 2024
1090749
Fix mssql & cockroach
Mar 22, 2024
5e3ec08
Pass raw error pointer to C context and resign de allocation to calli…
ospfranco Mar 22, 2024
23fe69d
Serialize API errors on start, commit and rollback transaction
ospfranco Mar 22, 2024
15d11aa
Re introduce dispatcher and spans
ospfranco Mar 22, 2024
8b88997
Add metrics feature to query-engine-common
ospfranco Mar 26, 2024
8ebb16e
Merge branch 'main' into c-abi
ospfranco Mar 26, 2024
f4a032e
change base path missing message to tracing
ospfranco Mar 26, 2024
e18bab5
Fix copy xcframework script
ospfranco Mar 26, 2024
40fb1d6
Store URL in the query engine instance
ospfranco Mar 26, 2024
0e97e0e
Fix non cloned url error and linting
ospfranco Mar 26, 2024
47890da
Add ReactNative preview feature
Apr 3, 2024
8a94ab8
Modify make scripts and add e2e workflow for RN
ospfranco Apr 5, 2024
e4e2f8c
Tweak CI
ospfranco Apr 5, 2024
ef223fc
Tweak CI
ospfranco Apr 5, 2024
c584262
Tweak CI
ospfranco Apr 5, 2024
27629a7
Tweak CI
ospfranco Apr 5, 2024
b5bbcd0
Tweak CI
ospfranco Apr 5, 2024
008591d
Tweak CI
ospfranco Apr 5, 2024
facdb56
Comment e2e test for RN
ospfranco Apr 5, 2024
9e61f64
Update query-engine/query-engine-c-abi/src/engine.rs
SevInf Apr 12, 2024
32e69ad
Merge branch 'main' into c-abi
Apr 12, 2024
a33143a
Adjust to main
Apr 12, 2024
4a05e40
Merge remote-tracking branch 'origin/main' into c-abi
Apr 12, 2024
639970d
Remove non-working workflow
Apr 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/codspeed.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,7 @@ jobs:
run: cargo codspeed build -p schema --features all_connectors

- name: "Build the benchmark targets: request-handlers"
run: cargo codspeed build -p request-handlers --features native
run: cargo codspeed build -p request-handlers --features native,all

- name: Run the benchmarks
uses: CodSpeedHQ/action@v2
Expand Down
70 changes: 68 additions & 2 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@ members = [
"query-engine/query-engine",
"query-engine/query-engine-node-api",
"query-engine/query-engine-wasm",
"query-engine/query-engine-c-abi",
"query-engine/request-handlers",
"query-engine/schema",
"libs/*",
Expand Down Expand Up @@ -86,6 +87,9 @@ strip = "symbols"
[profile.release.package.query-engine]
strip = "symbols"

[profile.release.package.query-engine-c-abi]
strip = "symbols"

[profile.release]
lto = "fat"
codegen-units = 1
Expand Down
3 changes: 3 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,9 @@ start-sqlite:
dev-sqlite:
cp $(CONFIG_PATH)/sqlite $(CONFIG_FILE)

dev-react-native:
cp $(CONFIG_PATH)/react-native $(CONFIG_FILE)

dev-libsql-js: build-qe-napi build-driver-adapters-kit
cp $(CONFIG_PATH)/libsql-js $(CONFIG_FILE)

Expand Down
9 changes: 5 additions & 4 deletions libs/query-engine-common/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@ name = "query-engine-common"
version = "0.1.0"
edition = "2021"

# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
[features]
metrics = []

[dependencies]
thiserror = "1"
Expand All @@ -20,11 +21,11 @@ tracing.workspace = true
tracing-subscriber = { version = "0.3" }
tracing-futures = "0.2"
tracing-opentelemetry = "0.17.3"
opentelemetry = { version = "0.17"}
opentelemetry = { version = "0.17" }

[target.'cfg(not(target_arch = "wasm32"))'.dependencies]
napi.workspace = true
[target.'cfg(all(not(target_arch = "wasm32")))'.dependencies]
query-engine-metrics = { path = "../../query-engine/metrics" }
napi.workspace = true

[target.'cfg(target_arch = "wasm32")'.dependencies]
wasm-bindgen.workspace = true
Expand Down
1 change: 1 addition & 0 deletions libs/query-engine-common/src/engine.rs
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,7 @@ pub struct EngineBuilder {
pub struct ConnectedEngineNative {
pub config_dir: PathBuf,
pub env: HashMap<String, String>,
#[cfg(feature = "metrics")]
pub metrics: Option<query_engine_metrics::MetricRegistry>,
}

Expand Down
1 change: 1 addition & 0 deletions libs/query-engine-common/src/error.rs
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ impl From<serde_json::Error> for ApiError {
}

#[cfg(not(target_arch = "wasm32"))]
#[cfg(not(any(target_os = "android", target_os = "ios")))]
impl From<ApiError> for napi::Error {
fn from(e: ApiError) -> Self {
let user_facing = user_facing_errors::Error::from(e);
Expand Down
5 changes: 3 additions & 2 deletions psl/psl-core/src/common/preview_features.rs
Original file line number Diff line number Diff line change
Expand Up @@ -78,8 +78,9 @@ features!(
UncheckedScalarInputs,
Views,
RelationJoins,
ReactNative,
PrismaSchemaFolder,
OmitApi,
OmitApi
);

/// Generator preview features (alphabetically sorted)
Expand Down Expand Up @@ -131,7 +132,7 @@ pub const ALL_PREVIEW_FEATURES: FeatureMap = FeatureMap {
| TransactionApi
| UncheckedScalarInputs
}),
hidden: enumflags2::make_bitflags!(PreviewFeature::{PrismaSchemaFolder}),
hidden: enumflags2::make_bitflags!(PreviewFeature::{PrismaSchemaFolder | ReactNative}),
};

#[derive(Debug)]
Expand Down
2 changes: 1 addition & 1 deletion query-engine/connector-test-kit-rs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ To run tests through a driver adapters, you should also configure the following

* `DRIVER_ADAPTER`: tells the test executor to use a particular driver adapter. Set to `neon`, `planetscale` or any other supported adapter.
* `DRIVER_ADAPTER_CONFIG`: a json string with the configuration for the driver adapter. This is adapter specific. See the [github workflow for driver adapter tests](.github/workflows/query-engine-driver-adapters.yml) for examples on how to configure the driver adapters.
* `ENGINE`: can be used to run either `wasm` or `napi` version of the engine.
* `ENGINE`: can be used to run either `wasm` or `napi` or `c-abi` version of the engine.

Example:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ use query_engine_tests::test_suite;
exclude(
Vitess("planetscale.js", "planetscale.js.wasm"),
Postgres("neon.js", "pg.js", "neon.js.wasm", "pg.js.wasm"),
Sqlite("libsql.js", "libsql.js.wasm", "cfd1")
Sqlite("libsql.js", "libsql.js.wasm", "cfd1", "react-native")
)
)]
mod metrics {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,18 +8,20 @@ use std::{convert::TryFrom, env, fmt::Display, fs::File, io::Read, path::PathBuf

static TEST_CONFIG_FILE_NAME: &str = ".test_config";

#[derive(Debug, Deserialize, Default, Clone)]
#[derive(Debug, Deserialize, Default, Clone, Copy, PartialEq)]
pub enum TestExecutor {
#[default]
Napi,
Wasm,
Mobile,
}

impl Display for TestExecutor {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
TestExecutor::Napi => f.write_str("Napi"),
TestExecutor::Wasm => f.write_str("Wasm"),
TestExecutor::Mobile => f.write_str("Mobile"),
}
}
}
Expand Down Expand Up @@ -62,6 +64,11 @@ pub struct TestConfigFromSerde {
/// test executor by setting the `DRIVER_ADAPTER_CONFIG` env var when spawning the executor.
/// Correctness: if set, [`TestConfigFromSerde::driver_adapter`] must be set as well.
pub(crate) driver_adapter_config: Option<DriverAdapterConfig>,

/// For mobile tests a running device with a valid http server is required.
/// This is the URL to the mobile emulator which will execute the queries against
/// the instances of the engine running on the device.
pub(crate) mobile_emulator_url: Option<String>,
}

impl TestConfigFromSerde {
Expand Down Expand Up @@ -105,10 +112,18 @@ impl TestConfigFromSerde {
Err(err) => exit_with_message(&err.to_string()),
}

if self.external_test_executor.is_some() && self.driver_adapter.is_none() {
exit_with_message(
"When using an external test executor, the driver adapter (DRIVER_ADAPTER env var) must be set.",
);
if self.external_test_executor.is_some() {
if self.external_test_executor.unwrap() == TestExecutor::Mobile && self.mobile_emulator_url.is_none() {
exit_with_message(
"When using the mobile external test executor, the mobile emulator URL (MOBILE_EMULATOR_URL env var) must be set.",
);
}

if self.external_test_executor.unwrap() != TestExecutor::Mobile && self.driver_adapter.is_none() {
exit_with_message(
"When using an external test executor, the driver adapter (DRIVER_ADAPTER env var) must be set.",
);
}
}

if self.driver_adapter.is_some() && self.external_test_executor.is_none() {
Expand Down Expand Up @@ -154,6 +169,7 @@ pub struct TestConfig {
pub(crate) connector_version: Option<String>,
pub(crate) with_driver_adapter: Option<WithDriverAdapter>,
pub(crate) is_ci: bool,
pub(crate) mobile_emulator_url: Option<String>,
}

impl From<TestConfigFromSerde> for TestConfig {
Expand All @@ -174,6 +190,7 @@ impl From<TestConfigFromSerde> for TestConfig {
connector_version: config.connector_version,
is_ci: config.is_ci,
with_driver_adapter,
mobile_emulator_url: config.mobile_emulator_url,
}
}
}
Expand Down Expand Up @@ -213,6 +230,7 @@ And optionally, to test driver adapters
- EXTERNAL_TEST_EXECUTOR
- DRIVER_ADAPTER
- DRIVER_ADAPTER_CONFIG (optional, not required by all driver adapters)
- MOBILE_EMULATOR_URL (optional, only required by mobile external test executor)

📁 Config file

Expand Down Expand Up @@ -278,6 +296,8 @@ impl TestConfig {
.map(|config| serde_json::from_str::<DriverAdapterConfig>(config.as_str()).ok())
.unwrap_or_default();

let mobile_emulator_url = std::env::var("MOBILE_EMULATOR_URL").ok();

// Just care for a set value for now.
let is_ci = std::env::var("BUILDKITE").is_ok();

Expand All @@ -289,13 +309,13 @@ impl TestConfig {
external_test_executor,
driver_adapter,
driver_adapter_config,
mobile_emulator_url,
})
.map(Self::from)
}

fn from_file() -> Option<Self> {
let current_dir = env::current_dir().ok();

current_dir
.and_then(|path| Self::try_path(config_path(path)))
.or_else(|| Self::workspace_root().and_then(|path| Self::try_path(config_path(path))))
Expand Down Expand Up @@ -402,6 +422,10 @@ impl TestConfig {
"PRISMA_DISABLE_QUAINT_EXECUTORS".to_string(),
"1".to_string(),
),
(
"MOBILE_EMULATOR_URL".to_string(),
self.mobile_emulator_url.clone().unwrap_or_default()
),
)
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -196,8 +196,9 @@ fn start_rpc_thread(mut receiver: mpsc::Receiver<ReqImpl>) -> Result<()> {
.build()
.unwrap()
.block_on(async move {
let environment = CONFIG.for_external_executor();
let process = match Command::new(&path)
.envs(CONFIG.for_external_executor())
.envs(environment)
.stdin(Stdio::piped())
.stdout(Stdio::piped())
.stderr(Stdio::inherit())
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ impl ConnectorTagInterface for SqliteConnectorTag {
#[derive(Clone, Debug, PartialEq, Eq)]
pub enum SqliteVersion {
V3,
ReactNative,
LibsqlJsNapi,
LibsqlJsWasm,
CloudflareD1,
Expand All @@ -37,6 +38,7 @@ pub enum SqliteVersion {
impl ToString for SqliteVersion {
fn to_string(&self) -> String {
match self {
SqliteVersion::ReactNative => "react-native".to_string(),
SqliteVersion::V3 => "3".to_string(),
SqliteVersion::LibsqlJsNapi => "libsql.js".to_string(),
SqliteVersion::LibsqlJsWasm => "libsql.js.wasm".to_string(),
Expand All @@ -53,6 +55,7 @@ impl TryFrom<&str> for SqliteVersion {
"3" => Self::V3,
"libsql.js" => Self::LibsqlJsNapi,
"libsql.js.wasm" => Self::LibsqlJsWasm,
"react-native" => Self::ReactNative,
"cfd1" => Self::CloudflareD1,
_ => return Err(TestError::parse_error(format!("Unknown SQLite version `{s}`"))),
};
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
{
"connector": "sqlite",
"version": "react-native",
"external_test_executor": "Mobile",
"mobile_emulator_url": "http://localhost:3000"
}
Loading
Loading