Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MAM Refactor #3599

Merged
merged 6 commits into from
Mar 3, 2025
+1,698 −1,182
Merged

MAM Refactor #3599

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
7ef1b91
Bugfix. Bind to right 'this'
jcbrand Mar 2, 2025
82a6f3b
Handle null being passed to parseErrorStanza
jcbrand Mar 2, 2025
b080fe2
Refactor MAM to fetch a limited number of newest messages first. Fixe…
jcbrand Feb 22, 2025
3db1db9
Avoid gaps in history when new message is received before MAM query i…
jcbrand Mar 3, 2025
3daa1ac
MUC bugfix: Don't attempt to rejoin too eagerly
jcbrand Mar 3, 2025
4eb6588
Don't treat messages with a `body` as affiliation change messages
jcbrand Mar 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -68,3 +68,6 @@ node_modules
.sv?
/vendor/
.aider*
.prompts/

Session.vim
2 changes: 2 additions & 0 deletions CHANGES.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,12 +15,14 @@
- #1303: Display non-contacts who sent us a message somehow in fullscreen
- #1349: XEP-0392 Consistent Color Generation
- #1700: Deleted pending contacts reappear after page reload
- #1810: Create clickable link to load older MAM messages if there is no scrollbars.
- #2118: Show reflected message in MUC
- #2383: Add modal to start chats with JIDs not in the roster
- #2586: Add support for XEP-0402 Bookmarks
- #2623: Merge MUC join and bookmark, leave and unset autojoin
- #2716: Fix issue with chat display when opening via URL
- #2844: Contact stays in "pending contacts"
- #2940: Avoid gaps in history when new message is received before MAM query is made
- #2980: Allow setting an avatar for MUCs
- #3033: Add the `muc_grouped_by_domain` option to display MUCs on the same domain in collapsible groups
- #3038: Message to self from other client is ignored
Expand Down
27 changes: 13 additions & 14 deletions docs/source/configuration.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -328,6 +328,14 @@ autocomplete_add_contact

Determines whether search suggestions are shown in the "Add Contact" modal.

auto_fill_history_gaps
----------------------

* Default: ``true``

Determins whether Converse automatically fills gaps in the chat history.
If set to false, a placeholder appears which can be clicked to fetch the
missing messages.

auto_focus
----------
Expand Down Expand Up @@ -1113,25 +1121,16 @@ If no nickame value is found, then an error will be raised.
mam_request_all_pages
---------------------

* Default: ``true``
* Default: ``false``

When requesting messages from the archive, Converse will ask only for messages
When requesting messages from the archive, Converse will query for messages
newer than the most recent cached message.

When there are many archived messages since that one, the returned results will
When there are many archived messages that matches the query, the returned results will
be broken up in to pages, set by `archived_messages_page_size`_.

By default Converse will request all the pages until all messages have been
fetched, however for large archives this can slow things down dramatically.

This setting turns the paging off, and Converse will only fetch the latest
page.

.. note::

If paging is turned off, there will appear gaps in the message history.
Converse currently doesn't yet have a way to inform the user of these gaps or
to let them be filled.
Set this option to ``true`` to request all pages of archived messages, but be
aware that this can have performance implications.


muc_hats
Expand Down
2 changes: 2 additions & 0 deletions local.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@
});

converse.initialize({
auto_fill_history_gaps: false,
archived_messages_page_size: 2,
muc_subscribe_to_rai: true,
theme: 'dracula',
auto_away: 300,
Expand Down
6 changes: 3 additions & 3 deletions package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion src/headless/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@
"pluggable.js": "3.0.1",
"sizzle": "^2.3.5",
"sprintf-js": "^1.1.2",
"strophe.js": "strophe/strophejs#f54d83199ec62272ee8b4987b2e63f188e1b2e29",
"strophe.js": "strophe/strophejs#fb70dcb4e202f632bc9932915b4522f70ad4d47c",
"urijs": "^1.19.10"
},
"devDependencies": {}
Expand Down
4 changes: 2 additions & 2 deletions src/headless/plugins/bookmarks/collection.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -136,9 +136,9 @@ class Bookmarks extends Collection {
${model.get('password') ? stx`<password>${model.get('password')}</password>` : ''}
${
extensions.length
? stx`<extensions>${extensions.map((e) => Stanza.unsafeXML(e))}</extensions>`
? stx`<extensions>${extensions.map((e) => Stanza.fromString(e))}</extensions>`
: ''
};
}
</conference>
</item>`;
}
Expand Down
6 changes: 3 additions & 3 deletions src/headless/plugins/disco/entity.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,8 +53,8 @@ class DiscoEntity extends Model {
* Returns a Promise which resolves with a map indicating
* whether a given identity is provided by this entity.
* @method _converse.DiscoEntity#getIdentity
* @param { String } category - The identity category
* @param { String } type - The identity type
* @param {String} category - The identity category
* @param {String} type - The identity type
*/
async getIdentity (category, type) {
await this.waitUntilFeaturesDiscovered;
Expand All @@ -68,7 +68,7 @@ class DiscoEntity extends Model {
* Returns a Promise which resolves with a map indicating
* whether a given feature is supported.
* @method _converse.DiscoEntity#getFeature
* @param { String } feature - The feature that might be supported.
* @param {String} feature - The feature that might be supported.
*/
async getFeature (feature) {
await this.waitUntilFeaturesDiscovered;
Expand Down
134 changes: 80 additions & 54 deletions src/headless/plugins/mam/api.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,11 +5,11 @@ import dayjs from 'dayjs';
import log from '../../log.js';
import sizzle from "sizzle";
import { RSM } from '../../shared/rsm';
import { Strophe, $iq } from 'strophe.js';
import { Strophe, Stanza } from 'strophe.js';
import { TimeoutError } from '../../shared/errors.js';

const { NS } = Strophe;
const u = converse.env.utils;
const { stx, u } = converse.env;


export default {
Expand All @@ -33,7 +33,7 @@ export default {
* RSM to enable easy querying between results pages.
*
* @method _converse.api.archive.query
* @param {import('./types').ArchiveQueryOptions} options - An object containing query parameters
* @param {import('./types').ArchiveQueryOptions} [options={}] - Optional query parameters
* @throws {Error} An error is thrown if the XMPP server responds with an error.
* @returns {Promise<import('./types').MAMQueryResult>}
*
Expand Down Expand Up @@ -66,15 +66,15 @@ export default {
* // For a particular user
* let result;
* try {
* result = await api.archive.query({'with': '[email protected]'});
* result = await api.archive.query({ mam: { with: '[email protected]' }});
* } catch (e) {
* // The query was not successful
* }
*
* // For a particular room
* let result;
* try {
* result = await api.archive.query({'with': '[email protected]', 'groupchat': true});
* result = await api.archive.query({ mam: { with: '[email protected]' }}, is_groupchat: true });
* } catch (e) {
* // The query was not successful
* }
Expand All @@ -83,14 +83,16 @@ export default {
* // Requesting all archived messages before or after a certain date
* // ===============================================================
* //
* // The `start` and `end` parameters are used to query for messages
* // The MAM `start` and `end` parameters are used to query for messages
* // within a certain timeframe. The passed in date values may either be ISO8601
* // formatted date strings, or JavaScript Date objects.
*
* const options = {
* 'with': '[email protected]',
* 'start': '2010-06-07T00:00:00Z',
* 'end': '2010-07-07T13:23:54Z'
* mam: {
* 'with': '[email protected]',
* 'start': '2010-06-07T00:00:00Z',
* 'end': '2010-07-07T13:23:54Z'
* },
* };
* let result;
* try {
Expand All @@ -109,7 +111,7 @@ export default {
* // Return maximum 10 archived messages
* let result;
* try {
* result = await api.archive.query({'with': '[email protected]', 'max':10});
* result = await api.archive.query({ mam: { with: '[email protected]', max:10 }});
* } catch (e) {
* // The query was not successful
* }
Expand All @@ -131,7 +133,7 @@ export default {
* // archived messages. Please note, when calling these methods, pass in an integer
* // to limit your results.
*
* const options = {'with': '[email protected]', 'max':10};
* const options = { mam: { with: '[email protected]' }, rsm: { max:10 }};
* let result;
* try {
* result = await api.archive.query(options);
Expand All @@ -143,7 +145,13 @@ export default {
*
* while (!result.complete) {
* try {
* result = await api.archive.query(Object.assign(options, rsm.next(10).query));
* result = await api.archive.query({
* mam: { ...options.mam },
* rsm: {
* ...options.rsm,
* ...rsm.next(10).query
* }
* });
* } catch (e) {
* // The query was not successful
* }
Expand All @@ -161,7 +169,7 @@ export default {
* // message, pass in the `before` parameter with an empty string value `''`.
*
* let result;
* const options = {'before': '', 'max':5};
* const options = { rsm: { before: '', max:5 }};
* try {
* result = await api.archive.query(options);
* } catch (e) {
Expand All @@ -172,65 +180,83 @@ export default {
*
* // Now we query again, to get the previous batch.
* try {
* result = await api.archive.query(Object.assign(options, rsm.previous(5).query));
* try {
* result = await api.archive.query({
* mam: { ...options.mam },
* rsm: {
* ...options.rsm,
* ...rsm.previous(5).query
* }
* });
* } catch (e) {
* // The query was not successful
* }
* // Do something with the messages, like showing them in your webpage.
* result.messages.forEach(m => this.showMessage(m));
*
*/
async query (options) {
async query (options={}) {
if (!api.connection.connected()) {
throw new Error('Can\'t call `api.archive.query` before having established an XMPP session');
}
const attrs = {'type':'set'};
if (options && options.groupchat) {
if (!options['with']) {

let toJID;
if (options && options.is_groupchat) {
if (!options.mam?.with) {
throw new Error(
'You need to specify a "with" value containing '+
'the chat room JID, when querying groupchat messages.');
'the groupchat JID, when querying groupchat messages.');
}
attrs.to = options['with'];
toJID = options.mam.with;
}

const withJID = !options.is_groupchat && options.mam?.with || null;

const bare_jid = _converse.session.get('bare_jid');
const jid = attrs.to || bare_jid;
const jid = toJID || bare_jid;
const supported = await api.disco.supports(NS.MAM, jid);
if (!supported) {
log.warn(`Did not fetch MAM archive for ${jid} because it doesn't support ${NS.MAM}`);
return {'messages': []};
return { messages: [] };
}

const queryid = u.getUniqueId();
const stanza = $iq(attrs).c('query', {'xmlns':NS.MAM, 'queryid':queryid});
if (options) {
stanza.c('x', {'xmlns':NS.XFORM, 'type': 'submit'})
.c('field', {'var':'FORM_TYPE', 'type': 'hidden'})
.c('value').t(NS.MAM).up().up();

if (options['with'] && !options.groupchat) {
stanza.c('field', {'var':'with'}).c('value')
.t(options['with']).up().up();
}
['start', 'end'].forEach(t => {
if (options[t]) {
const date = dayjs(options[t]);
if (date.isValid()) {
stanza.c('field', {'var':t}).c('value').t(date.toISOString()).up().up();
} else {
throw new TypeError(`archive.query: invalid date provided for: ${t}`);
}
// Validate start and end dates and add them to attrs (in the right format)
const { start: startDate, end: endDate } = ['start', 'end'].reduce((acc, t) => {
if (options.mam?.[t]) {
const date = dayjs(options.mam[t]);
if (date.isValid()) {
acc[t] = date.toISOString();
} else {
throw new TypeError(`archive.query: invalid date provided for: ${t}`);
}
});
stanza.up();
const rsm = new RSM(options);
if (Object.keys(rsm.query).length) {
stanza.cnode(rsm.toXML());
}
}
return acc;
}, { start: null, end: null });

const connection = api.connection.get();
const rsm = options.rsm ? new RSM(options.rsm) : {};
const queryid = u.getUniqueId();

const stanza = stx`
<iq id="${u.getUniqueId()}"
${toJID ? Stanza.unsafeXML(`to="${Strophe.xmlescape(toJID)}"`) : ""}
type="set"
xmlns="jabber:client">
<query queryid="${queryid}" xmlns="${NS.MAM}">
${
withJID || startDate || endDate
? stx`
<x type="submit" xmlns="${NS.XFORM}">
<field type="hidden" var="FORM_TYPE"><value>${NS.MAM}</value></field>
${withJID ? stx`<field var="with"><value>${withJID}</value></field>` : ""}
${startDate ? stx`<field var="start"><value>${startDate}</value></field>` : ""}
${endDate ? stx`<field var="end"><value>${endDate}</value></field>` : ""}
</x>`
: ""
}
${Object.keys(rsm.query ?? {}).length ? stx`${Stanza.unsafeXML(rsm.toXML().outerHTML)}` : ""}
</query>
</iq>`;

const messages = [];
const message_handler = connection.addHandler(/** @param {Element} stanza */(stanza) => {
Expand All @@ -239,8 +265,8 @@ export default {
return true;
}
const from = stanza.getAttribute('from') || bare_jid;
if (options.groupchat) {
if (from !== options['with']) {
if (options.is_groupchat) {
if (from !== options.mam?.with) {
log.warn(`Ignoring alleged groupchat MAM message from ${stanza.getAttribute('from')}`);
return true;
}
Expand All @@ -254,7 +280,7 @@ export default {

let error;
const timeout = api.settings.get('message_archiving_timeout');
const iq_result = await api.sendIQ(stanza, timeout, false)
const iq_result = await api.sendIQ(stanza, timeout, false);
if (iq_result === null) {
const { __ } = _converse;
const err_msg = __("Timeout while trying to fetch archived messages.");
Expand All @@ -272,14 +298,14 @@ export default {
}
connection.deleteHandler(message_handler);

let rsm;
let rsm_result;
const fin = iq_result && sizzle(`fin[xmlns="${NS.MAM}"]`, iq_result).pop();
const complete = fin?.getAttribute('complete') === 'true'
const complete = fin?.getAttribute('complete') === 'true';
const set = sizzle(`set[xmlns="${NS.RSM}"]`, fin).pop();
if (set) {
rsm = new RSM({...options, 'xml': set});
rsm_result = new RSM({...options.rsm, xml: set});
}
return { messages, rsm, complete };
return { messages, rsm: rsm_result, complete };
}
}
}
Loading
Loading