-
Notifications
You must be signed in to change notification settings - Fork 5.4k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[native] Add Prestissimo documentation page to Developer Guide
Adding a new documentation page for Prestissimo to the Developer Guide, and adding some of the configuration parameters for remote function execution.
- Loading branch information
Showing
1 changed file
with
95 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,95 @@ | ||
=========================== | ||
Presto Native - Prestissimo | ||
=========================== | ||
|
||
Prestissimo is a C++ drop-in replacement for Presto workers based on the Velox | ||
library. It implements the same RESTful endpoints as Java workers using the | ||
Proxygen C++ HTTP framework. Since communication with the Java coordinator and | ||
across workers is only done using the REST endpoints, Prestissimo does not use | ||
JNI and it is commonly deployed in such a way as to avoid having a JVM process | ||
on worker nodes. | ||
|
||
Prestissimo's codebase is located at `presto-native-execution | ||
<https://github.com/prestodb/presto/tree/master/presto-native-execution>`_. | ||
|
||
|
||
Endpoints | ||
--------- | ||
|
||
HTTP endpoints related to tasks are registered to Proxygen in | ||
`TaskResource.cpp`. Important endpoints implemented include: | ||
|
||
* POST: v1/task: This processes a `TaskUpdateRequest` | ||
* GET: v1/task: This returns a serialized `TaskInfo` (used for comprehensive | ||
metrics, may be reported less frequently) | ||
* GET: v1/task/status: This returns | ||
a serialized `TaskStatus` (used for query progress tracking, must be reported | ||
frequently) | ||
|
||
Other HTTP endpoints include: | ||
|
||
* POST: v1/memory | ||
* Reports memory, but no assignments are adjusted unlike in Java workers. | ||
* GET: v1/info | ||
* GET: v1/status | ||
|
||
The request/response flow of Prestissimo is identical to Java workers. The | ||
tasks or new splits are registered via `TaskUpdateRequest`. Resource | ||
utilization and query progress are sent to the coordinator via task endpoints. | ||
|
||
|
||
Remote Function Execution | ||
------------------------- | ||
|
||
Prestissimo supports remote execution of scalar functions. This feature is | ||
useful for cases when the function code is not written in C++, or if for | ||
security or flexibility reasons the function code cannot be linked to the same | ||
executable as the main engine. | ||
|
||
Remote function signatures need to be provided using a JSON file, following | ||
the format implemented by `JsonFileBasedFunctionNamespaceManager | ||
<https://github.com/prestodb/presto/blob/master/presto-function-namespace-managers/src/main/java/com/facebook/presto/functionNamespace/json/JsonFileBasedFunctionNamespaceManager.java>`_. | ||
The following properties allow the configuration of remote function execution: | ||
|
||
``remote-function-server.signature.files.directory.path`` | ||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
||
* **Type:** ``string`` | ||
* **Default value:** ``""`` | ||
|
||
The local filesystem path where JSON files containing remote function | ||
signatures are located. If not empty, the Presto native worker will | ||
recursively search, open, parse, and register function definitions from | ||
these JSON files. | ||
|
||
|
||
``remote-function-server.thrift.address`` | ||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
||
* **Type:** ``string`` | ||
* **Default value:** ``""`` | ||
|
||
The location (ip address or hostname) that hosts the remote function | ||
server, if any remote functions were registered using | ||
``remote-function-server.signature.files.directory.path``. | ||
If not specified, falls back to the loopback interface (``::1``) | ||
|
||
``remote-function-server.thrift.port`` | ||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
||
* **Type:** ``integer`` | ||
* **Default value:** ``0`` | ||
|
||
The port that hosts the remote function server. If not specified and remote | ||
functions are trying to be registered, an exception is thrown. | ||
|
||
``remote-function-server.thrift.uds-path`` | ||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | ||
|
||
* **Type:** ``string`` | ||
* **Default value:** ``""`` | ||
|
||
The UDS (unix domain socket) path to communicate with a local remote | ||
function server. If specified, takes precedence over | ||
``remote-function-server.thrift.address`` and | ||
``remote-function-server.thrift.port``. |