🚧 This documentation is for the developer preview of m-ld.

m-ld Javascript clone engine

The Javascript engine can be used in a modern browser or a server engine like Node.js.

The Javascript clone engine conforms to the m-ld specification. Its support for transaction pattern complexity is detailed below. Its concurrency model is based on immutable states.

Getting Started

npm install @m-ld/m-ld

There are two starter projects available:

• The Node.js project uses Node processes to initialise two clones, and an MQTT broker for messaging.
• The Web project shows one way to build a multi-collaborator forms application for browsers, using Socket.io for messaging.

Data Persistence

m-ld uses levelup to interface with a LevelDB-compatible storage backend.

• For the fastest responses use memdown (memory storage).
• In the browser, use level-js (browser-local storage).
• In a service or native application, use leveldown (file system storage).

Connecting to Other Clones

A m-ld clone uses a 'remotes' object to communicate with other clones.

• With an MQTT broker, use MqttRemotes.
• For a scalable global managed service, use AblyRemotes.
• If you have a live web server (not just CDN or serverless), you can use IoRemotes (🚀new!).

🚧 If your architecture includes some other publish/subscribe service like AMQP, or you would like to use a fully peer-to-peer protocol, please contact us to discuss your use-case.

Initialisation

The clone function initialises the m-ld engine with a leveldb back-end and the clone configuration.

import MemDown from 'memdown';
import { clone, uuid } from '@m-ld/m-ld';
import { MqttRemotes, MeldMqttConfig } from '@m-ld/m-ld/dist/mqtt';

const config: MeldMqttConfig = {
  '@id': uuid(),
  '@domain': 'test.example.org',
  genesis: true,
  mqtt: { hostname: 'mqtt.example.org' }
};

const meld = await clone(new MemDown, MqttRemotes, config);

The clone function returns control as soon as it is safe to start making data transactions against the domain. If this clone has has been re-started from persisted state, it may still be receiving updates from the domain. This can cause a UI to start showing these updates. If instead, you want to wait until the clone has the most recent data, you can add:

await meld.status.becomes({ online: true, outdated: false });

Remotes

MQTT Remotes

MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol. It is convenient to use it for local development or if the deployment environment has an MQTT broker available. See below for specific broker requirements.

The MqttRemotes class and its companion configuration class MeldMqttConfig can be imported or required from '@m-ld/m-ld/dist/mqtt'. You must also install the async-mqtt package as a peer of @m-ld/m-ld.

The configuration interface adds an mqtt key to the base MeldConfig. The content of this key is a client options object for MQTT.js. It must not include the will and clientId options, as these are set internally. It must include a hostname or a host and port.

MQTT Broker Requirements

MqttRemotes requires broker support for:

1. MQTT 3.1
2. QoS 0 and 1
3. Retained messages
4. Last Will and Testament (LWT) messages

A good choice for local development is Aedes.

Ably Remotes

Ably provides infrastructure and APIs to power realtime experiences at scale. It is a managed service, and includes pay-as-you-go developer pricing. It is also convenient to use for global deployments without the need to self-manage a broker.

The AblyRemotes class and its companion configuration class MeldAblyConfig can be imported or required from '@m-ld/m-ld/dist/ably'. You must also install the ably package as a peer of @m-ld/m-ld.

The configuration interface adds an ably key to the base MeldConfig. The content of this key is an Ably client options object. It must not include the echoMessages and clientId options, as these are set internally.

If using token authentication, ensure that the clientId the token is generated for corresponds to the @id given in the MeldConfig.

Socket.io Remotes

Socket.IO enables real-time, bidirectional and event-based communication. It works on every platform, browser or device, focusing equally on reliability and speed. It is convenient to use when the app architecture has a live web server or app server, using HTTP.

The IoRemotes class and its companion configuration class MeldIoConfig can be imported or required from '@m-ld/m-ld/dist/socket.io'. You must also install the socket.io-client package as a peer of @m-ld/m-ld.

The configuration interface adds an io key to the base MeldConfig. The value is an optional object having:

• uri: The server URL (defaults to the browser URL with no path)
• opts: A Socket.io factory options object, which can be used to customise the server connection

Socket.io Server

When using Socket.io, the server must correctly route m-ld protocol operations to their intended recipients. The Javascript engine package bundles a class for Node.js servers, IoRemotesService, which can be imported from '@m-ld/m-ld/dist/socket.io/server'.

To use, initialise the Socket.io server as normal, and then construct an IoRemotesService, passing the namespace you want to make available to m-ld. To use the global namespace, pass the sockets member of the Server class. For example:

const socket = require('socket.io');
const httpServer = require('http').createServer();
// Start the Socket.io server, and attach the m-ld message-passing service
const io = new socket.Server(httpServer);
new IoRemotesService(io.sockets);

For a complete example, see the web starter project .

For other server types, contact us.

Transactions

A m-ld transaction is a json-rql pattern, which represents a data read or a data write. See the m-ld specification for a walk-through of the syntax.

Supported pattern types for this engine are (follow the links for examples):

• Describe
• Select
• Construct (🚀new!)
• Group or Subject (the shorthand way to insert data)
• Update (the longhand way to insert or delete data)

🚧 If you have a requirement for an unsupported pattern, please contact us to discuss your use-case. You can browse the full json-rql syntax at json-rql.org.

Concurrency

A m-ld clone contains realtime domain data in principle. This means that any clone operation may be occurring concurrently with operations on other clones, and these operations combine to realise the final convergent state of every clone.

The Javascript clone engine API supports bounded procedures on immutable state, for situations where a query or other data operation may want to operate on a state that is unaffected by concurrent operations. In general this means that in order to guarantee data consistency, it is not necessary for the app to use the clone's local clock ticks (which nevertheless appear in places in the API for consistency with other engines).

An immutable state can be obtained using the read and write methods. The state is passed to a procedure which operates on it. In both cases, the state remains immutable until the procedure's returned Promise resolves or rejects.

In the case of write, the state can be transitioned to a new state from within the procedure using its own write method, which returns a new immutable state.

In the case of read, changes to the state following the procedure can be listened to using the second parameter, a handler for new states. As well as each update in turn, this handler also receives an immutable state following the given update.

Example: Initialising an App

await clone.read(async (state: MeldReadState) => {
  // State in a procedure is locked until sync complete or returned promise resolves
  let currentData = await state.read(someQuery);
  populateTheUi(currentData);
}, async (update: MeldUpdate, state: MeldReadState) => {
  // The handler happens for every update after the proc
  // State in a handler is locked until sync complete or returned promise resolves
  updateTheUi(update); // See §Handling Updates, below
});

Example: UI Action Handler

ui.on('action', async () => {
  clone.write(async (state: MeldState) => {
    let currentData = await state.read(something);
    let externalStuff = await doExternals(currentData);
    let todo = decideWhatToDo(externalStuff);
    // Writing to the current state creates a new live state
    state = await state.write(todo);
    await checkStuff(state);
  });
});

Example: UI Show Handler

ui.on('show', async () => {
  clone.read((state: MeldReadState) => {
    let currentData = await state.read(something);
    showTheNewUi(currentData);
  });
});

Handling Updates

Clone updates obtained from a read handler specify the exact Subject property values that have been deleted or inserted during the update. Utilities are provided to help update app views of data based on updates notified via the read method:

• updateSubject
• SubjectUpdater
• asSubjectUpdates
• includesValue

{
  "@select": "?name",
  "@where": { "employeeNo": 7, "name": "?name" }
}