Update Google Calendar integration #856

Niloth-p · 2025-02-20T23:43:38Z

Fixes: #847
Fixes: #850 (to auto-close absorbed PRs)
Fixes: #851 (to auto-close absorbed PRs)

Please don't get intimidated by the number of commits, it's an attempt to make things more readable and easy to review, not less so.

Updates to the integration - summarized

Major changes in the behavior of the integration include:

Bugs

Replace deprecated oauth2client library, which prevented the script from being used.
Update outdated send_message parameters that would generate errors when the script sent Zulip messages.
Previously, the integration script would only run up to 1 hour before erroring out, as the auth token will always expire in 1 hour, and the user needs to run the get-google-credentials script to re-generate the auth token.
Previously, the zuliprc of the user was needed, and hence the user could only run this integration locally and just for themselves. Now, a generic bot's zuliprc is sufficient. The integration can be used to send reminders from shared/public calendars in public channels.
Can now be run from a server. Previously, the OAuth process would require interaction with the browser for authorization.

Integration Features

Can now send reminders to channels (previously, was restricted to DMs).
Added more details to the reminders being sent. Previously, only "title starts at x" was sent. Now, messages include links to the calendar, google meet links to their call, the event description, location, etc.

Security Improvements

user's zuliprc -> bot's zuliprc.
Narrow the permission scope to calendar events.

Usability Features

Vastly improved the usage help.
- No longer require passing in the user's email to the command directly.
Previously required running 3 different commands to run the integration. Now, all in one script.
Added error handling for unaddressed cases.
Vastly improved customization of the integration. Options can be passed in either through command line flags or via settings in the .zuliprc file.
- Pass in custom paths to the two auth-related files.
- Choose the channel and topic name.
- Use your own formatting for the message.
Previously, all reminder messages were being printed to the terminal. Now, key processes of the integration are logged when the --verbose flag is set.

Developer Features - Vastly improved the code readability. All decisions are explained either in the comments or the commit messages (which add only 1 feature at a time, to clarify the intention, and create verifiable history).

Failing CI is unrelated

The failing CI seems to be unrelated to the changes in this PR.

Error:

ERROR:  schema "pgroonga" does not exist

Error running a subcommand of /__w/python-zulip-api/python-zulip-api/server/tools/lib/provision_inner.py: tools/setup/postgresql-init-dev-db

Which has nothing to do with this PR.

User Experience - Feedback welcome!

Includes the message template, and the default file location

Do we want to change the default location (and filename) of the client secret file that the user needs to download, (rename) and place before running the integration? It is currently set to ~/client_secret.json.
- Google API docs sometimes mention it as credentials.json. But, we went against naming it that way to avoid confusing with other credentials.
- It could also help to not pollute the ~ directory with a vague file name like client_secret.json.
- I've left the "Improve CLIENT_SECRET_FILE occurrences" commit a bit bare, with some minor and miscellaneous changes, so that we can add to it, if we want to change the default file path or file name.
Usage help:
Formatting of the message being sent:

{title}, at {location}, is scheduled from {start} to {end}.
> {description}
Join call.

The title is linked to open the event in the calendar.
If there's a location included - append ", at {location}".
If the event is an all-day event - use "is today" instead of "is scheduled from {start} to {end}."
The description is placed inside a quote.
"Join call." is linked to the Google Meet call link, if any.
The reason I went with "is scheduled from" is because I've been considering adding support for the status of the event, which could be tentative. But that has been dropped for the moment.

Manual and automated testing

Events API Link for reviewers
Will update the integration doc a bit later, for reference for manual testing.
- I'm considering posting just the main instructions here, because updating that PR fully may take a while longer.
It would have definitely been awesome to have automated tests, for these many commits, but kept putting it off since I wasn't sure they were a priority for the moment. Did not expect the PR to keep growing to 27 commits. Though I considered it, I decided not to wait until the tests are added, to push this branch.

Self-review checklist

Self-reviewed the changes for clarity and maintainability
(variable names, code reuse, readability, etc.).

Communicate decisions, questions, and potential concerns.

Explains differences from previous plans (e.g., issue description).
Highlights technical choices and bugs encountered.
Calls out remaining decisions and concerns.
Automated tests verify logic where appropriate.

Individual commits are ready for review (see commit discipline).

Each commit is a coherent idea.
Commit message(s) explain reasoning and motivation for changes.

Completed manual review and testing of the following:

Visual appearance of the changes.
Responsiveness and internationalization.
Strings and tooltips.
End-to-end functionality of buttons, interactions and flows.
Corner cases, error conditions, and easily imagined bugs.

Fixes: zulip#847 Co-authored-by: Vedant Joshi <[email protected]> Co-authored-by: Pransh Gupta <[email protected]>

Remove "sender" parameter. Rename the "type" parameter value from "private" to "direct". Pass in a list to the "to" parameter. Switch to using the {} syntax for the dict. Co-authored-by: Vedant Joshi <[email protected]>

Previously used scope: `calendar.readonly` New scope: `calendar.events.readonly` Other than events, a calendar contains `settings`, `addons` ,`app`, `calendarlist`, `calendars`, `acls` (permissions), `freebusy` (availability), and more. Since this integration only sends reminders, we need access only to the events. More narrow scopes like `calendar.events.owned.readonly` and `calendar.events.public.readonly` are available, but we want to be able to support shared calendars as well, so we're not using them. Also removed a comment regarding SCOPES that has now become redundant.

Instead of using the hardcoded file name value everywhere directly. This enables us to edit the file name with a single change in the following commit.

Replaces some occurrences of the term "credentials" with "tokens" for clarity, to conform with the terms used in Google API documentation. Renamed: - "google-credentials.json" to "google-tokens.json" - `credential_path` to `tokens_path` Google documentation refers to the client secret file sometimes as "credentials.json", and the tokens file as "tokens.json". We have been using "client-secret.json" and "google-credentials.json" respectively, which can be confusing. So, this commit switches to using the term "tokens" instead of "credentials" wherever appropriate, to avoid confusion. The term "credentials" is still used to refer to the Credentials object returned after authorization.

- Added a link to the integration doc. - Removed the initial space. - Fixed the command usage for the calendar option. - Added the integration-provided options to the first line. - Mentioned downloading the client secret file in the instructions. - Made other minor edits to the writing.

- Removed add_argument() parameters set to default values. - Renamed `dest` of calendarID to calendar_id, in order to make flag names uniform, in preparation for adding support for loading them from a config file. - Cleared up spacing, and made minor clarifications to `help`.

Leveraged the provisioning support provided by `init_from_options`. Enabled the `--provision` argument by setting `allow_provisiong` to True in `add_default_arguments`. Re-ordered the script such that the imports are executed after `init_from_options`. Updated the import error message to recommend using the command with the --provision argument to install the dependencies.

Currently, the integration prints all reminders to the terminal. Set logging level to info when --verbose is set, and switched `print` to `logging.info`, to restrict that output to only when the --verbose option is set.

Log the error response without halting the script.

Removed it from google-calendar script, as it is only used for the first run authorization by get-google-credentials. Edited a comment clarifying its purpose in get-google-credentials.

The auth tokens expire every hour. So, the google-calendar script would stop functioning one hour after start, if it only loads from the token file. The get-google-credential script needs to be called directly from the google-calendar script, without the user needing to run that as a separate command. The get-google-credentials script is not a module. And it uses hyphens in its name, hence it cannot be directly imported into google-calendar. To avoid renaming files, and smoothly pass in arguments, runpy is used. Though google-calendar script is currently the only google service integration that uses get-google-credentials, the script is not merged with the google-calendar script, to keep the logic separate, and easy to expand. The get-google-credentials script can no longer be run directly, as the get_credentials() arguments do not have any default values, with the constants deleted, to avoid redundancy with the google-calendar script. Updated the function docstrings, usage help and error messages.

The script currently mandatorily requires the --user argument to be passed, this commit removes the need for the --user flag.

The reminders were previously being sent to the DMs of the current user, requiring the zuliprc of the user. But, we can create a generic bot for the Google Calendar integration, and use its zuliprc to send direct messages to its owner. This tightens security. The support for sending DMs to the same user is retained, for cases where a bot is not used.

Previously, until last commit, the zuliprc of the user was being used directly. Now that we're using the zuliprc of a bot, it is safe for multiple users with a shared calendar to have a bot send the reminders to a channel. Added channel and topic arguments, with the default behavior set to sending a direct message.

This allows the integration to be run from non-interactive environments or on devices without browsers, like remote servers. Co-authored-by: Vedant Joshi <[email protected]>

For passing in custom file paths.

By loading from the new "google-calendar" section of the zuliprc. Moved the default values out of the argparse arguments, and into a dataclass, to implement the below hierarchy of loading options: - initialize options to default values - overwrite with the options present in the config file - overwrite with the options passed-in via the command line

Currently, all the reminders are collected together before sending a message, and are formatted as bullet points in the same message. This commit uses a message per event, in preparation of adding more details to each reminder message.

The event tuple is switched to a TypedDict. Moved the code logic to construct the message string from the event into its own function. This is in preparation for adding support for more event fields.

Moved the parsing logic into `get_start_or_end` to enable re-using the function when we add support for the `end` field of event.

Added the following Event fields to the message template: - event end time - description - link to event in Google Calendar - location of event, if any - link to Google Meet, if any Co-authored-by: Vedant Joshi <[email protected]>

By adding a --format-message command line option, and a format_message config file option.

alya · 2025-02-21T19:32:16Z

@laurynmm Are you up for reviewing this one? I haven't tried to wrap my head around it. Not specifically a release goal.

zulip/integrations/google/google-calendar

timabbott · 2025-04-07T19:08:17Z

zulip/integrations/google/google-calendar

@@ -166,7 +163,7 @@ def send_reminders() -> Optional[None]:
        message = "Reminder:\n\n" + "\n".join("* " + m for m in messages)

    result = zulip_client.send_message(
-        {"type": "direct", "to": [options.zulip_email], "content": message}
+        {"type": "direct", "to": [zulip_client.get_profile()["email"]], "content": message}


The zulip_client.get_profile()["email"]] call should probably be at the beginning, not done every time one sends -- at least if we use a user ID rather than email. (I've not checked if later commits do that already).

This moves around, but I think we might want to add a tweak to do ["id"] instead -- and also probably a to parameter to just set a user ID who it should DM, like you have for the channel/topic parameters.

(Maybe there's some precedent in zulip/examples for how to name it)

timabbott · 2025-04-07T19:11:06Z

zulip/integrations/google/google-calendar

+        # Boolean arguments (store-true) have a default value of False when not passed in.
+        # So, we ignore them, to prevent overwriting the config file option that is set.
+        if key in valid_keys and value is not None and value is not False:
+            setattr(calendar_options, key, value)


Probably these two functions will want to eventually move to common code that other integrations can use too. But this is fine for this PR.

timabbott · 2025-04-07T19:14:08Z

zulip/integrations/google/google-calendar

@@ -251,6 +258,19 @@ def populate_events() -> Optional[None]:


 def construct_message_from_event(event: Event) -> str:
+    if calendar_options.format_message:


This feature might be overkill. I guess I'll merge it.

timabbott · 2025-04-07T19:14:56Z

Looks great, merged, thanks @Niloth-p!

google-calendar: Update requirements.txt dependencies.

352a475

zulipbot added size: XL bug good first issue labels Feb 20, 2025

Niloth-p force-pushed the gcal branch from ad41f5e to 0181d2c Compare February 20, 2025 23:49

Niloth-p and others added 25 commits February 21, 2025 05:20

google-calendar: Replace deprecated oauth2client library.

81387d1

Fixes: zulip#847 Co-authored-by: Vedant Joshi <[email protected]> Co-authored-by: Pransh Gupta <[email protected]>

google-calendar: Update outdated send_message parameters.

e722e90

Remove "sender" parameter. Rename the "type" parameter value from "private" to "direct". Pass in a list to the "to" parameter. Switch to using the {} syntax for the dict. Co-authored-by: Vedant Joshi <[email protected]>

google-calendar: Use a constant for the tokens filename.

4b9dbb3

Instead of using the hardcoded file name value everywhere directly. This enables us to edit the file name with a single change in the following commit.

google-calendar: Add error handling for missing client secret file.

07810c5

google-calendar: Add error handling for send_message.

8b2b9dd

Log the error response without halting the script.

google-calendar: Improve CLIENT_SECRET_FILE occurrences.

06db61b

Removed it from google-calendar script, as it is only used for the first run authorization by get-google-credentials. Edited a comment clarifying its purpose in get-google-credentials.

google-calendar: Use current user's email id for send_message.

8b7c536

The script currently mandatorily requires the --user argument to be passed, this commit removes the need for the --user flag.

google-calendar: Support manual authorization using auth code.

4c56d9b

This allows the integration to be run from non-interactive environments or on devices without browsers, like remote servers. Co-authored-by: Vedant Joshi <[email protected]>

google-calendar: Log writing to the tokens file.

441dce9

google-calendar: Add options --client-secret-file and --tokens-file.

c004199

For passing in custom file paths.

google-calendar: Fix type of event id, switch from int to str.

e38a8e6

google-calendar: Add TypedDict and string conversion function for event.

8794800

The event tuple is switched to a TypedDict. Moved the code logic to construct the message string from the event into its own function. This is in preparation for adding support for more event fields.

google-calendar: Generalize the datetime parsing for event fields.

b468931

Moved the parsing logic into `get_start_or_end` to enable re-using the function when we add support for the `end` field of event.

google-calendar: Support user customization of the message template.

974863e

By adding a --format-message command line option, and a format_message config file option.

Niloth-p force-pushed the gcal branch from 0181d2c to 974863e Compare February 20, 2025 23:50

Niloth-p marked this pull request as ready for review February 21, 2025 00:53

alya added the maintainer review PR is ready for review by Zulip maintainers. label Feb 21, 2025

alya assigned laurynmm Feb 21, 2025

Niloth-p mentioned this pull request Feb 22, 2025

zulip-integrations: Update google oauth and reminders for google calendar integration. #850

Closed

12 tasks

This was referenced Apr 7, 2025

Replace oauth2client with google-auth and google-auth-oauthlib in google-calendar integration #860

Closed

api: Exclude None values from request payload in Endpoint class. #851

Closed

timabbott reviewed Apr 7, 2025

View reviewed changes

zulip/integrations/google/google-calendar Outdated Show resolved Hide resolved

timabbott reviewed Apr 7, 2025

View reviewed changes

timabbott merged commit 929e71e into zulip:main Apr 7, 2025
13 of 14 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update Google Calendar integration #856

Update Google Calendar integration #856

Niloth-p commented Feb 20, 2025 •

edited

Loading

alya commented Feb 21, 2025 •

edited

Loading

timabbott Apr 7, 2025

timabbott Apr 7, 2025 •

edited

Loading

timabbott Apr 7, 2025

timabbott Apr 7, 2025

timabbott commented Apr 7, 2025

		@@ -251,6 +258,19 @@ def populate_events() -> Optional[None]:


		def construct_message_from_event(event: Event) -> str:
		if calendar_options.format_message:

Update Google Calendar integration #856

Update Google Calendar integration #856

Conversation

Niloth-p commented Feb 20, 2025 • edited Loading

Updates to the integration - summarized

Failing CI is unrelated

User Experience - Feedback welcome!

Manual and automated testing

alya commented Feb 21, 2025 • edited Loading

timabbott Apr 7, 2025

Choose a reason for hiding this comment

timabbott Apr 7, 2025 • edited Loading

Choose a reason for hiding this comment

timabbott Apr 7, 2025

Choose a reason for hiding this comment

timabbott Apr 7, 2025

Choose a reason for hiding this comment

timabbott commented Apr 7, 2025

Niloth-p commented Feb 20, 2025 •

edited

Loading

alya commented Feb 21, 2025 •

edited

Loading

timabbott Apr 7, 2025 •

edited

Loading