npm -i @evercoder/sharedb
ShareDB is a realtime database backend based on Operational Transformation (OT) of JSON documents. It is the realtime backend for the DerbyJS web application framework.
For questions, discussion and announcements, join the ShareJS mailing list or check the FAQ.
Please report any bugs you find to the issue tracker.
- Realtime synchronization of any JSON document
- Concurrent multi-user collaboration
- Synchronous editing API with asynchronous eventual consistency
- Realtime query subscriptions
- Simple integration with any database - MongoDB, PostgresQL (experimental)
- Horizontally scalable with pub/sub integration
- Projections to select desired fields from documents and operations
- Middleware for implementing access control and custom extensions
- Ideal for use in browsers or on the server
- Offline change syncing upon reconnection
- In-memory implementations of database and pub/sub for unit testing
TLDR
const WebSocket = require('reconnecting-websocket');
var socket = new WebSocket('ws://' + window.location.host);
var connection = new sharedb.Connection(socket);
The native Websocket object that you feed to ShareDB's Connection
constructor does not handle reconnections.
The easiest way is to give it a WebSocket object that does reconnect. There are plenty of example on the web. The most important thing is that the custom reconnecting websocket, must have the same API as the native rfc6455 version.
In the "textarea" example we show this off using a Reconnecting Websocket implementation from reconnecting-websocket.
Simple app demonstrating realtime sync
Leaderboard app demonstrating live queries
In ShareDB's view of the world, every document has 3 properties:
- version - An incrementing number starting at 0
- type - An OT type. OT types are defined in
share/ottypes. Documents
which don't exist implicitly have a type of
null
. - data - The actual data that the document contains. This must be pure acyclic JSON. Its also type-specific. (JSON type uses raw JSON, text documents use a string, etc).
ShareDB implicitly has a record for every document you can access. New documents have version 0, a null type and no data. To use a document, you must first submit a create operation, which will set the document's type and give it initial data. Then you can submit editing operations on the document (using OT). Finally you can delete the document with a delete operation. By default, ShareDB stores all operations forever - nothing is truly deleted.
First, create a ShareDB server instance:
var ShareDB = require('sharedb');
var share = new ShareDB(options);
Options
options.db
(instance ofShareDB.DB
) Store documents and ops with this database adapter. Defaults toShareDB.MemoryDB()
.options.pubsub
(instance ofShareDB.PubSub
) Notify other ShareDB processes when data changes through this pub/sub adapter. Defaults toShareDB.MemoryPubSub()
.options.milestoneDb
(instance of ShareDB.MilestoneDB`) Store snapshots of documents at a specified interval of versions
ShareDB.MemoryDB
, backed by a non-persistent database with no queriesShareDBMongo
, backed by a real Mongo database and full query supportShareDBMingoMemory
, backed by a non-persistent database supporting most Mongo queries. Useful for faster testing of a Mongo-based app.ShareDBPostgres
, backed by PostgresQL. No query support.
ShareDB.MemoryPubSub
can be used with a single processShareDBRedisPubSub
can be used with multiple processes using Redis' pub/sub mechanism
Community Provided Pub/Sub Adapters
sharedb-milestone-mongo
, backed by Mongo
var WebSocketJSONStream = require('@teamwork/websocket-json-stream');
// 'ws' is a websocket server connection, as passed into
// new (require('ws').Server).on('connection', ...)
var stream = new WebSocketJSONStream(ws);
share.listen(stream);
For transports other than WebSockets, expose a duplex
stream that writes and reads JavaScript objects. Then
pass that stream directly into share.listen
.
Middlewares let you hook into the ShareDB server pipeline. In middleware code you can read and also modify objects as they flow through ShareDB. For example, sharedb-access uses middlewares to implement access control.
share.use(action, fn)
Register a new middleware.
action
(String) One of:'connect'
: A new client connected to the server.'op'
: An operation was loaded from the database.'readSnapshots'
: Snapshot(s) were loaded from the database for a fetch or subscribe of a query or document'query'
: A query is about to be sent to the database'submit'
: An operation is about to be submitted to the database'apply'
: An operation is about to be applied to a snapshot before being committed to the database'commit'
: An operation was applied to a snapshot; The operation and new snapshot are about to be written to the database.'afterWrite'
: An operation was successfully written to the database.'receive'
: Received a message from a client'reply'
: About to send a non-error reply to a client message
fn
(Function(context, callback)) Call this function at the time specified byaction
.context
will always have the following properties:action
: The action this middleware is hanldingagent
: A reference to the server agent handling this clientbackend
: A reference to this ShareDB backend instance
context
can also have additional properties, as relevant for the action:collection
: The collection name being handledid
: The document id being handledop
: The op being handledreq
: HTTP request being handled, if provided toshare.listen
(for 'connect')stream
: The duplex Stream provided toshare.listen
(for 'connect')query
: The query object being handled (for 'query')snapshots
: Array of retrieved snapshots (for 'readSnapshots')data
: Received client message (for 'receive')request
: Client message being replied to (for 'reply')reply
: Reply to be sent to the client (for 'reply')
ShareDB supports exposing a projection of a real collection, with a specified (limited) set of allowed fields. Once configured, the projected collection looks just like a real collection - except documents only have the fields you've requested. Operations (gets, queries, sets, etc) on the fake collection work, but you only see a small portion of the data.
addProjection(name, collection, fields)
Configure a projection.
name
The name of the projected collection.collection
The name of the existing collection.fields
A map (object) of the allowed fields in documents.- Keys are field names.
- Values should be
true
.
For example, you could make a users_limited
projection which lets users view each other's names and profile pictures, but not password hashes. You would configure this by calling:
share.addProjection('users_limited', 'users', { name:true, profileUrl:true });
Note that only the JSON0 OT type is supported for projections.
By default, ShareDB logs to console
. This can be overridden if you wish to silence logs, or to log to your own logging driver or alert service.
Methods can be overridden by passing a console
-like object to logger.setMethods
:
var ShareDB = require('sharedb');
ShareDB.logger.setMethods({
info: () => {}, // Silence info
warn: () => alerts.warn(arguments), // Forward warnings to alerting service
error: () => alerts.critical(arguments) // Remap errors to critical alerts
});
ShareDB only supports the following logger methods:
info
warn
error
share.close(callback)
Closes connections to the database and pub/sub adapters.
The client API can be used from either Node or a browser. First, get a ShareDB.Connection
object by connecting to the ShareDB server instance:
From Node:
// `share` should be a ShareDB server instance
var connection = share.connect();
To use ShareDB from a browser, use a client bundler like Browserify or Webpack. The following code connects to the ShareDB server instance over WebSockets:
var ShareDB = require('sharedb/lib/client');
var socket = new WebSocket('ws://localhost:8080');
var connection = new ShareDB.Connection(socket);
For transports other than WebSockets, create an object implementing
the WebSocket specification and pass it into the ShareDB.Connection
constructor.
connection.get(collectionName, documentId)
Get a ShareDB.Doc
instance on a given collection and document ID.
connection.createFetchQuery(collectionName, query, options, callback)
connection.createSubscribeQuery(collectionName, query, options, callback)
Get query results from the server. createSubscribeQuery
also subscribes to
changes. Returns a ShareDB.Query
instance.
query
(Object) A descriptor of a database query with structure defined by the database adapter.callback
(Function) Called with(err, results)
when server responds, or on error.options.results
(Array) Prior query results if available, such as from server rendering.options.*
All other options are passed through to the database adapter.
connection.fetchSnapshot(collection, id, version, callback): void;
Get a read-only snapshot of a document at the requested version.
-
collection
(String) Collection name of the snapshot -
id
(String) ID of the snapshot -
version
(number) [optional] The version number of the desired snapshot. Ifnull
, the latest version is fetched. -
callback
(Function) Called with(error, snapshot)
, wheresnapshot
takes the following form:{ id: string; // ID of the snapshot v: number; // version number of the snapshot type: string; // the OT type of the snapshot, or null if it doesn't exist or is deleted data: any; // the snapshot }
connection.fetchSnapshotByTimestamp(collection, id, timestamp, callback): void;
Get a read-only snapshot of a document at the requested version.
-
collection
(String) Collection name of the snapshot -
id
(String) ID of the snapshot -
timestamp
(number) [optional] The timestamp of the desired snapshot. The returned snapshot will be the latest snapshot before the provided timestamp. Ifnull
, the latest version is fetched. -
callback
(Function) Called with(error, snapshot)
, wheresnapshot
takes the following form:{ id: string; // ID of the snapshot v: number; // version number of the snapshot type: string; // the OT type of the snapshot, or null if it doesn't exist or is deleted data: any; // the snapshot }
doc.type
(String)
The OT type of this document
doc.id
(String)
Unique document ID
doc.data
(Object)
Document contents. Available after document is fetched or subscribed to.
doc.fetch(function(err) {...})
Populate the fields on doc
with a snapshot of the document from the server.
doc.subscribe(function(err) {...})
Populate the fields on doc
with a snapshot of the document from the server, and
fire events on subsequent changes.
doc.unsubscribe(function (err) {...})
Stop listening for document updates. The document data at the time of unsubscribing remains in memory, but no longer stays up-to-date. Resubscribe with doc.subscribe
.
doc.ingestSnapshot(snapshot, callback)
Ingest snapshot data. The snapshot
param must include the fields v
(doc version), data
, and type
(OT type). This method is generally called interally as a result of fetch or subscribe and not directly from user code. However, it may still be called directly from user code to pass data that was transferred to the client external to the client's ShareDB connection, such as snapshot data sent along with server rendering of a webpage.
doc.destroy()
Unsubscribe and stop firing events.
doc.on('load', function() {...})
The initial snapshot of the document was loaded from the server. Fires at the
same time as callbacks to fetch
and subscribe
.
doc.on('create', function(source) {...})
The document was created. Technically, this means it has a type. source
will be false
for ops received from the server and defaults to true
for ops generated locally.
doc.on('before op batch'), function() {...})
An operation batch is about to be applied to the data. For each partial operation a pair of before op
and op
events will be emitted after this event.
doc.on('before op'), function(op, source) {...})
An operation is about to be applied to the data. source
will be false
for ops received from the server and defaults to true
for ops generated locally.
doc.on('op', function(op, source) {...})
An operation was applied to the data. source
will be false
for ops received from the server and defaults to true
for ops generated locally.
doc.on('after op batch'), function() {...})
An operation batch was applied to the data.
doc.on('del', function(data, source) {...})
The document was deleted. Document contents before deletion are passed in as an argument. source
will be false
for ops received from the server and defaults to true
for ops generated locally.
doc.on('error', function(err) {...})
There was an error fetching the document or applying an operation.
doc.removeListener(eventName, listener)
Removes any listener you added with doc.on
. eventName
should be one of 'load'
, 'create'
, 'before op'
, 'op'
, 'del'
, or 'error'
. listener
should be the function you passed in as the second argument to on
. Note that both on
and removeListener
are inherited from EventEmitter.
doc.create(data[, type][, options][, function(err) {...}])
Create the document locally and send create operation to the server.
data
Initial document contentstype
(OT type) Defaults to'ot-json0'
, for whichdata
is an Objectoptions.source
Argument passed to the'create'
event locally. This is not sent to the server or other clients. Defaults totrue
.
doc.submitOp(op, [, options][, function(err) {...}])
Apply operation to document and send it to the server.
op
structure depends on the document type. See the
operations for the default 'ot-json0'
type.
Call this after you've either fetched or subscribed to the document.
options.source
Argument passed to the'op'
event locally. This is not sent to the server or other clients. Defaults totrue
.
doc.del([options][, function(err) {...}])
Delete the document locally and send delete operation to the server.
Call this after you've either fetched or subscribed to the document.
options.source
Argument passed to the'del'
event locally. This is not sent to the server or other clients. Defaults totrue
.
doc.whenNothingPending(function(err) {...})
Invokes the given callback function after
- all ops submitted via
doc.submitOp
have been sent to the server, and - all pending fetch, subscribe, and unsubscribe requests have been resolved.
Note that whenNothingPending
does NOT wait for pending model.query()
calls.
query.ready
(Boolean)
True if query results are ready and available on query.results
query.results
(Array)
Query results, as an array of ShareDB.Doc
instances.
query.extra
(Type depends on database adapter and query)
Extra query results that aren't an array of documents. Available for certain database adapters and queries.
query.on('ready', function() {...}))
The initial query results were loaded from the server. Fires at the same time as
the callbacks to createFetchQuery
and createSubscribeQuery
.
query.on('error', function(err) {...}))
There was an error receiving updates to a subscription.
query.destroy()
Unsubscribe and stop firing events.
query.on('changed', function(results) {...}))
(Only fires on subscription queries) The query results changed. Fires only once
after a sequence of diffs are handled.
query.on('insert', function(docs, atIndex) {...}))
(Only fires on subscription queries) A contiguous sequence of documents were added to the query result array.
query.on('move', function(docs, from, to) {...}))
(Only fires on subscription queries) A contiguous sequence of documents moved position in the query result array.
query.on('remove', function(docs, atIndex) {...}))
(Only fires on subscription queries) A contiguous sequence of documents were removed from the query result array.
query.on('extra', function() {...}))
(Only fires on subscription queries) query.extra
changed.
Backend
represents the server-side instance of ShareDB. It is primarily responsible for connecting to clients, and sending requests to the database adapters. It is also responsible for some configuration, such as setting up middleware and projections.
var Backend = require('sharedb');
var backend = new Backend(options);
Constructs a new Backend
instance, with the provided options:
db
DB (optional): an instance of a ShareDB database adapter that provides the data store for ShareDB. If omitted, a new, non-persistent, in-memory adapter will be created, which should not be used in production, but may be useful for testingpubsub
PubSub (optional): an instance of a ShareDB Pub/Sub adapter that provides a channel for notifying other ShareDB instances of changes to data. If omitted, a new, in-memory adapter will be created. Unlike the database adapter, the in-memory instance may be used in a production environment where pub/sub state need only persist across a single, stand-alone servermilestoneDb
MilestoneDB (optional): an instance of a ShareDB milestone adapter that provides the data store for milestone snapshots, which are historical snapshots of documents stored at a specified version interval. If omitted, this functionality will not be enabledextraDbs
Object (optional): an object whose values are extraDB
instances which can be queried. The keys are the names that can be passed into the query optionsdb
fieldsuppressPublish
boolean (optional): if set totrue
, any changes committed will not be published onpubsub
maxSubmitRetries
number (optional): the number of times to allow a submit to be retried. If omitted, the request will retry an unlimited number of times
var connection = backend.connect();
Connects to ShareDB and returns an instance of a Connection
. This is the server-side equivalent of new ShareDBClient.Connection(socket)
in the browser.
This method also supports infrequently used optional arguments:
var connection = backend.connect(connection, req);
connection
Connection (optional): aConnection
instance to bind to theBackend
req
Object (optional): a connection context object that can contain information such as cookies or session data that will be available in the middleware
Returns a Connection
.
var agent = backend.listen(stream, req);
Registers a Stream
with the backend. This should be called when the server receives a new connection from a client.
stream
Stream: aStream
(orStream
-like object) that will be used to communicate between the newAgent
and theBackend
req
Object (optional): a connection context object that can contain information such as cookies or session data that will be available in the middleware
Returns an Agent
, which is also available in the middleware.
backend.close(callback);
Disconnects ShareDB and all of its underlying services (database, pubsub, etc.).
callback
Function: a callback with the signaturefunction (error: Error): void
that will be called once the services have stopped, or with anerror
if at least one of them could not be stopped
backend.use(action, middleware);
Adds middleware to the Backend
.
action
string | string[]: an action, or array of action names defining when to apply the middlewaremiddleware
Function: a middleware function with the signaturefunction (context: Object, callback: Function): void;
. See middleware for more details
Returns the Backend
instance, which allows for multiple chained calls.
backend.addProjection(name, collection, fields);
Adds a projection.
name
string: the name of the projectioncollection
string: the name of the collection on which to apply the projectionfields
Object: a declaration of which fields to include in the projection, such as{ field1: true }
. Defining sub-field projections is not supported.
backend.submit(agent, index, id, op, options, callback);
Submits an operation to the Backend
.
agent
Agent
: connection agent to pass to the middlewareindex
string: the name of the target collection or projectionid
string: the document IDop
Object: the operation to submitoptions
Object: these options are passed through to the database adapter'scommit
method, so any options that are valid there can be used herecallback
Function: a callback with the signaturefunction (error: Error, ops: Object[]): void;
, whereops
are the ops committed by other clients between the submittedop
being submitted and committed
backend.getOps(agent, index, id, from, to, options, callback);
Fetches the ops for a document between the requested version numbers, where the from
value is inclusive, but the to
value is non-inclusive.
agent
Agent
: connection agent to pass to the middlewareindex
string: the name of the target collection or projectionid
string: the document IDfrom
number: the first op version to fetch. If set tonull
, then ops will be fetched from the earliest versionto
number: The last op version. This version will not be fetched (ieto
is non-inclusive). If set tonull
, then ops will be fetched up to the latest versionoptions
: Object (optional): options can be passed directly to the database driver'sgetOps
inside theopsOptions
property:{opsOptions: {metadata: true}}
callback
: Function: a callback with the signaturefunction (error: Error, ops: Object[]): void;
, whereops
is an array of the requested ops
backend.getOpsBulk(agent, index, fromMap, toMap, options, callback);
Fetches the ops for multiple documents in a collection between the requested version numbers, where the from
value is inclusive, but the to
value is non-inclusive.
agent
Agent
: connection agent to pass to the middlewareindex
string: the name of the target collection or projectionid
string: the document IDfromMap
Object: an object whose keys are the IDs of the target documents. The values are the first versions requested of each document. For example,{abc: 3}
will fetch ops for document with IDabc
from version3
(inclusive)toMap
Object: an object whose keys are the IDs of the target documents. The values are the last versions requested of each document (non-inclusive). For example,{abc: 3}
will fetch ops for document with IDabc
up to version3
(not inclusive)options
: Object (optional): options can be passed directly to the database driver'sgetOpsBulk
inside theopsOptions
property:{opsOptions: {metadata: true}}
callback
: Function: a callback with the signaturefunction (error: Error, opsMap: Object): void;
, whereopsMap
is an object whose keys are the IDs of the requested documents, and their values are the arrays of requested ops, eg{abc: []}
backend.fetch(agent, index, id, options, callback);
Fetch the current snapshot of a document.
agent
Agent
: connection agent to pass to the middlewareindex
string: the name of the target collection or projectionid
string: the document IDoptions
: Object (optional): options can be passed directly to the database driver'sfetch
inside thesnapshotOptions
property:{snapshotOptions: {metadata: true}}
callback
: Function: a callback with the signaturefunction (error: Error, snapshot: Snapshot): void;
, wheresnapshot
is the requested snapshot
backend.fetchBulk(agent, index, ids, options, callback);
Fetch multiple document snapshots from a collection.
agent
Agent
: connection agent to pass to the middlewareindex
string: the name of the target collection or projectionids
string[]: array of document IDsoptions
: Object (optional): options can be passed directly to the database driver'sfetchBulk
inside thesnapshotOptions
property:{snapshotOptions: {metadata: true}}
callback
: Function: a callback with the signaturefunction (error: Error, snapshotMap: Object): void;
, wheresnapshotMap
is an object whose keys are the requested IDs, and the values are the requestedSnapshot
s
backend.queryFetch(agent, index, query, options, callback);
Fetch snapshots that match the provided query. In most cases, querying the backing database directly should be preferred, but queryFetch
can be used in order to apply middleware, whilst avoiding the overheads associated with using a Doc
instance.
agent
Agent
: connection agent to pass to the middlewareindex
string: the name of the target collection or projectionquery
Object: a query object, whose format will depend on the database adapter being usedoptions
Object: an object that may contain adb
property, which specifies which database to run the query against. These extra databases can be attached via theextraDbs
option in theBackend
constructorcallback
Function: a callback with the signaturefunction (error: Error, snapshots: Snapshot[], extra: Object): void;
, wheresnapshots
is an array of the snapshots matching the query, andextra
is an (optional) object that the database adapter might return with more information about the results (such as counts)
An Agent
is the representation of a client's Connection
state on the server. If the Connection
was created through backend.connect
(ie the client is running on the server), then the Agent
associated with a Connection
can be accessed through a direct reference: connection.agent
.
The Agent
will be made available in all middleware requests. The agent.custom
field is an object that can be used for storing arbitrary information for use in middleware. For example:
backend.useMiddleware('connect', function (request, callback) {
// Best practice to clone to prevent mutating the object after connection.
// You may also want to consider a deep clone, depending on the shape of request.req.
Object.assign(request.agent.custom, request.req);
callback();
});
backend.useMiddleware('readSnapshots', function (request, callback) {
var connectionInfo = request.agent.custom;
var snapshots = request.snapshots;
// Use the information provided at connection to determine if a user can access snapshots.
// This should also be checked when fetching and submitting ops.
if (!userCanAccessSnapshots(connectionInfo, snapshots)) {
return callback(new Error('Authentication error'));
}
callback();
});
// Here you should determine what permissions a user has, probably by reading a cookie and
// potentially making some database request to check which documents they can access, or which
// roles they have, etc. If doing this asynchronously, make sure you call backend.connect
// after the permissions have been fetched.
var connectionInfo = getUserPermissions();
// Pass info in as the second argument. This will be made available as request.req in the
// 'connection' middleware.
var connection = backend.connect(null, connectionInfo);
By default, ShareDB logs to console
. This can be overridden if you wish to silence logs, or to log to your own logging driver or alert service.
Methods can be overridden by passing a console
-like object to logger.setMethods
var ShareDB = require('sharedb/lib/client');
ShareDB.logger.setMethods({
info: () => {}, // Silence info
warn: () => alerts.warn(arguments), // Forward warnings to alerting service
error: () => alerts.critical(arguments) // Remap errors to critical alerts
});
ShareDB only supports the following logger methods:
info
warn
error
ShareDB returns errors as plain JavaScript objects with the format:
{
code: 5000,
message: 'ShareDB internal error'
}
Additional fields may be added to the error object for debugging context depending on the error. Common additional fields include collection
, id
, and op
.
- 4001 - Unknown error type
- 4002 - Database adapter does not support subscribe
- 4003 - Database adapter not found
- 4004 - Missing op
- 4005 - Op must be an array
- 4006 - Create data in op must be an object
- 4007 - Create op missing type
- 4008 - Unknown type
- 4009 - del value must be true
- 4010 - Missing op, create or del
- 4011 - Invalid src
- 4012 - Invalid seq
- 4013 - Found seq but not src
- 4014 - op.m invalid
- 4015 - Document does not exist
- 4016 - Document already exists
- 4017 - Document was deleted
- 4018 - Document was created remotely
- 4019 - Invalid protocol version
- 4020 - Invalid default type
- 4021 - Invalid client id
- 4022 - Database adapter does not support queries
- 4023 - Cannot project snapshots of this type
- 4024 - Invalid version
- 4025 - Passing options to subscribe has not been implemented
The 41xx
and 51xx
codes are reserved for use by ShareDB DB adapters, and the 42xx
and 52xx
codes are reserved for use by ShareDB PubSub adapters.
- 5001 - No new ops returned when retrying unsuccessful submit
- 5002 - Missing snapshot
- 5003 - Snapshot and op version don't match
- 5004 - Missing op
- 5005 - Missing document
- 5006 - Version mismatch
- 5007 - Invalid state transition
- 5008 - Missing version in snapshot
- 5009 - Cannot ingest snapshot with null version
- 5010 - No op to send
- 5011 - Commit DB method unimplemented
- 5012 - getSnapshot DB method unimplemented
- 5013 - getOps DB method unimplemented
- 5014 - queryPollDoc DB method unimplemented
- 5015 - _subscribe PubSub method unimplemented
- 5016 - _unsubscribe PubSub method unimplemented
- 5017 - _publish PubSub method unimplemented
- 5018 - Required QueryEmitter listener not assigned
- 5019 - getMilestoneSnapshot MilestoneDB method unimplemented
- 5020 - saveMilestoneSnapshot MilestoneDB method unimplemented
- 5021 - getMilestoneSnapshotAtOrBeforeTime MilestoneDB method unimplemented
- 5022 - getMilestoneSnapshotAtOrAfterTime MilestoneDB method unimplemented