Debian package for 3.3.0

[to-go]

Hi @aureq !
Is it possible to build a Debian package with the latest version 3.3.0 and never the less with the new name Matomo?

Torsten

[aureq]

@to-go
Hi Torsten,

Yes, I'm well aware of this as you can imagine. I just spoke at length with @mattab and we have a plan that I need to action. We're targeting around mid-feb (18/02) if everything goes according to plan.
To share more with you, I lack of personal free time and we also want to make sure the work we put in the package will be useful in the long term.

I will use this issue to track my commit so you should be aware of the progresses made.

Thanks

ref:

• https://wiki.debian.org/RenamingPackages
• https://wiki.debian.org/PackageTransition

[aureq]

Ok, for the people who are following this thread, I'll be brain-dumping as I progress through.

There's a lot of left-over from piwik and matomo is not everywhere yet. To name a few:

• https://builds.matomo.org/ contains only archives named piwik-*
• piwik-3.3.0.(tar.gz|zip) has an initial folder named piwik

Here's an incomplete list of things to do :

• Merge origin/matomo into master
• Update scripts/build-package.sh to support Matomo and Piwik at the same time. Then @mattab to republish version 3.3.0 and the following beta version.
• Create a transition package for Debian and Ubuntu without breaking everything
• Make the transition package coexist alongside with Matomo in this repository
• Remove as many traces of piwik from the Matomo Debian package (where sensible to do so)
• Move /etc/piwik to /etc/matomo due to the symlink named config/. I think the Piwik transitional package should own /etc/piwik and Matomo package should own /etc/matomo. But both pointing at the same location.
• Similar to /etc/piwik, the web server alias /piwik should belong to Piwik package and /matomo should belong to Matomo package
• Update content for https://debian.matomo.org/ with up to date instructions or alternatively have a proper installation documentation on https://matomo.org

[aureq]

@mattab I noticed that in scripts/build-package.sh#L439 the script recommends sending an email to Microsoft App Store, but the filename in the message body is piwik-X.X.X-WAG.zip.

Should the script use instead matomo-X.X.X-WAG.zip instead?

[aureq]

@mattab the latest build script has a few features. It publishes and archive for piwik and has piwik/ as its main folder, and for the matomo archive the main folder is matomo/. This should give users who download these archives to transition smoothly.

On your side, if you continue to execute the script as usual, you will publish both matomo and piwik archives on the main website. However, only matomo is published for the Web App folder. I noticed that https://webgallery.microsoft.com/apps/Piwik is quite outdated and I don't know if this is intentional.

Last, if you could review my code changes, that would be great.

Cheers

[mattab]

Hi @aureq

Thanks for the update! Feedback:

• Looks like Microsoft is not updating the app anymore, unfortunately. It's not so important for now so we can still leave the Webgallery package generation. It's OK to have only the Matomo package for the webgallery 👍
• Would you mind doing future changes to the script in a pull request? this way I can review the changes
• fyi: I will review the changes next time I deploy a beta, next week

[aureq]

@mattab Have you had a chance to test the latest script version? Would it be possible to publish the matomo package alongside with the piwik package please?

[aureq]

In order to create a transitional package from piwik to matomo, we have 2 options

1 - update debian/control and add a second Package: section to reference Matomo.

2 - duplicate the current repository back into piwik-package (or create a new one) and:
2a - update the new piwik-package to include only transitional information (symlinks, configuration migration and so on)
2b - update matomo-package so it contains the Matomo application, docs and configuration files

If we do 1 each time we generate a Matomo release, a Piwik release will be generated at the same time and both with share the same version number. This is kind of annoying as it will be confusing to many users, but also because if the transitional package has to be updated, then Matomo will receive a new version at the same time (and vice versa).

If we do 2, it create a bit more work and duplicates some work, however we have control over each package independently which is more desirable.

Last question is to understand whether an entirely new/forked repository is more appropriate (better visibility) rather than a branch in this repository instead (less management overhead).

[aureq]

I spoke with @mattab and we both agreed that 2b is the best solution of all. We should be rather close to have a transition package and having Matomo 3.3.0 released at an internal test version. If all goes well then I'll publish version 3.4.0 soon after.

[aureq]

@mattab

1. I can't publish the package because one of my command is missing (reprepro) which hasn't been reinstalled with the server upgrade.
2. I finally have something that seems to be working fine, but I haven't tested it through a proper Debian repository.

[mackuba]

Hi!

Sorry for asking that kind of question, but do you have any approximate timeline for when 3.3/3.4 Debian packages might be available on http://debian.matomo.org? Is it like probably this/next week if things go well, or rather further out?

[aureq]

Hi @mackuba,

Don't be sorry, I'm equally impatient as you are. I'm hoping to have version 3.3.0 out in the next 10 days or so. Maybe less but I can't guarantee due to work commitments (I'm presenting at the AWS Sydney Summit). But definitely keep an eye on this issue as I'll update it as soon as it's available.

[aureq]

@mattab If it could be possible to have inoticoming installed on the server that would be great.

[mattab]

inoticoming is installed @aureq 👍

[aureq]

Status update for those following this issue.

• my Vmware refuses to provide me with network connectivity which makes my work useless right now (VM boots but no network)
• I need to create an EC2 instance and have my lab in it rather than in Vmware
• find a way to extra any work that hasn't been committed into GIT and move it to my lab EC2 instance
• do some basic checks between Debian 8 (Vmware) and Debian 9 (EC2)

An important aspect to this is securely transferring my GPG signing keys to my lab EC2 instance.

[mattab]

Hi @aureq - would it be possible for you to run all the code from the SSH account provided for debian.matomo.org ? This way it's fully backed up, etc. maybe it could make things simpler?

[aureq]

Hi @mattab and all

TL;DR - Remove the human component and automate everything.

Long version:

@mattab technically yes, but probably not a good idea. It's a shared box, ideally the environment should be tightly controlled bu us only due to sensitive material like GPG signing keys. I also remember it's tricky to get all the packages required installed in a timely manner (I don't have any list either yet).

While I'm super time poor at the moment, that Vmware event shows that I need to take the bullet, step up and make the process of creating a .deb fully automated (or as much as it is possible to).
I need to do the following:

• Manually generate an EC2 instance from a vanilla AMI
• Record the list of installed packages and specific commands
• Record the deployment process for Matomo's package
• Secure and automate the GPG keys handling, SSH private key and remote server public keys (using S3 and KMS)

Once the steps above are done:

• Create a bootstrap script to include all the above
• Create a CloudFormation template to deploy an EC2 instance(*) and inject the bootstrap script
• Push the resulting .deb into debian.matomo.org for publication

From a more strategic and long term perspective...

I would say this is probably the best initial steps we should be taking. As additional improvements we should consider the following:

• Use AWS CodeBuild with a buildspec.yml instead of EC2 (Better/longer AWS free-tier coverage)
• Automatically trigger a build when a new release is published
• Move the Debian/Ubuntu repository outside of our dedicated hosting
• Automate the Debian/Ubuntu repository and publish it to CloudFront and S3(*).

Benefits would be:

• Near real-time package publication for Debian/Ubuntu
• Ability to publish beta and rc packages (great for early adopters and testers)
• No web hosting infrastructure due to S3 and CloudFront for a possibly small fee
• A very secure environment to generate, publish and host all packages due to architecture and AWS services like KMS and Server Side Encryption (SSE)
• Portability of such environment into any AWS account (Infrastructure as code)
• No/Minimal human intervention required
• Happy community users

(*) Cost should be considered on the following items.

[aureq]

People interested in this issue should follow #64 as well.

[mattab]

@aureq Sounds all good if you can maintain the EC2 setup so it keeps running for the years ahead....! 👍

I've just release the 3.5.0-b3 using the updated script so you now have both matomo and piwik packages available.

All the best for this project and your upcoming event... 🥇

[mackuba]

Hi!

So... are we any closer to a 3.3+ Matomo Debian package than we were in April?... 🙂 (sorry)

[aureq]

@mackuba I hope so. Aiming for the end of this weekend.

[aureq]

Status report:

• Installed piwik transitional package successfully
• Installed matomo 3.3.0 successfully, including migrating config.ini.php (file copy)

I had to:

• restart apache2 service bit it should only be a warning, or a reload
• remove the link in /etc/apache2/conf.d/piwik.conf (but unsure it belongs to our package)

[aureq]

Status report:
I was hoping to keep the transitional package and the main package together inside the same branch, but it doesn't appear to be possible. I'm going to create a separate branch piwik-transition that will only contain the dummy/transitional package while master will hold the real package.

Doing this allows me to:

• bump piwik version to 9.99.9 to signal death of this package, while pulling matomo as a required dependency.
• not have to update the piwik package as frequently as matomo
• keep code separated and easier to maintain over time

[aureq]

Status report:

• master contains matomo package's main branch.
• piwik-package contains only the transitional package.
• no new package has been pushed yet to the Debian/Ubuntu repository yet

Both branches have been pushed to Github.

The next step is to test the upgrade process as much as possible.

[aureq]

Status report:

• published piwik (9.9.1-1)
• published matomo (3.3.0-1)

I'm going to publish the other missing versions very soon.

[aureq]

Status report:

• published matomo (3.4.0-1)
• published matomo (3.5.0-1)
• published matomo (3.5.1-1)

I used the official package repository to upgrade my own Matomo installation. The only hurdle is switching to newer webserver configuration but everything else should be mostly smooth.

[jawrat]

thank you!!! works as advertised. couple of steps necessary, but nothing too difficult.

$ sudo a2disconf piwik
removing dangling symlink /etc/apache2/conf-enabled/piwik.conf

vi /etc/apache2/sites-enabled/000-default.conf and replace most instances of piwik with matomo

then service apache2 reload

and bob's your uncle! again, great job aureq, and thank you VERY much! :)

EDIT: there were a couple of minor issues I discovered after deploying this. firstly, make sure you edit the cron job to point to the new files (/usr/share/matomo in place of /usr/share/piwik)...secondly, I had a weird issue with a custom plugin that I needed to cp -Rp over from the piwik dir to the new matomo dir. once I did those things, it appears that all is well.

[ptr1120]

thank you, working!
The directories in apaches matamo.conf point to "/usr/share/piwik". Is that intended?

[aureq]

@ptr1120 Not sure what you are referring to. Could you please clarify ?

If you are talking about /etc/apache2/conf-available/matomo.conf, then it should point to /etc/matomo/apache.conf. This link is created during the installation process (see)

[aureq]

@jawrat thanks for reporting all the details:

1. a2disconf: I'll look into that.
2. /etc/apache2/sites-enabled/000-default.conf: I don't think this is related to the Debian package.
3. Plugins: I'll look what could be done regarding Matomo plugins (Had a similar problem)
4. Cron: there should be a new cron file named /etc/cron.d/matomo-archive. But I guess the old cron should be removed for sure.

[ptr1120]

@aureq: sorry, I talk about the directory directives inside /etc/apache2/conf-available/matomo.conf that point to /usr/share/piwik(see). Shouldn't they point to matomo folder?

[aureq]

@ptr1120 You're absolutely right. Let me fix that quickly.

[aureq]

@ptr1120 I pushed the updated package. Let me know if this is all good for you. Cheers

[ptr1120]

Thanks, installed 3.5.1-2, looks perfect.

[rlafuente]

Thank you so much for this development! When updating from the existing package, I just had to do @jawrat's indication to sudo a2disconf piwik, edit the virtualhost aliases in conf-enabled/matomo.conf and everything's going smooth.

[nomandera]

Some feedback from someone who has just completed their first migration from Piwik to Matamo.

I too had the dangling symlink /etc/apache2/conf-available/piwik.conf which for such a small issue results in a hard fail of Apache startup.

I also seen the issue with addons. All the native addons migrated successfully but my extra addon installed from the store (free) QueuedTracking failed to transfer. Matomo expected this addon to be there based on the stored config and complained loudly about it. I can only assume it also broke the install but after consulting this thread I moved directly to the cp -Rp /usr/share/piwik/plugins/QueuedTracking/ /usr/share/matomo/plugins/QueuedTracking/ fix which work immediately as as expected.

On an upgrade ideally both the Piwik and Matomo Aliases in /etc/apache2/conf-available/matomo.conf should be uncommented as I dont think it is normal behavior for apt update to turn daemons off. It also helps clear up potential confusion and loss of tracking after the apt upgrade is complete but before the users have reworked their sites to use the new Matomo tracking.

On that note I expected the new tracking code to be fully rebranded and was surprised to see a mentions of Piwik

<!-- Matomo -->
...
    var u="https://example.com/matomo/";
    _paq.push(['setTrackerUrl', u+'piwik.php']);
...
    g.type='text/javascript'; g.async=true; g.defer=true; g.src=u+'piwik.js'; s.parentNode.insertBefore(g,s);
...
<!-- End Matomo Code -->

Perhaps this is just waiting for a fix upstream but since it is likely to require users to do a second rework of their sites tracking code for optimal deployment it is worth special mention

It has been suggested that /etc/cron.d/piwik-archive should be deleted but this file can often user crafted configuration. Better would be to rename the file to new /etc/cron.d/matomo-archive and add the new sample cron entry for reference. After all the currently upgrade leaves the old one running anyway.

Lastly I am still a bit vague about owners and permissions. This is an area Matomo does quite badly at itself with vague advice and quite a bit of trial and error debates in their forums. I would say that it feels odd to have web assets owned by root:root by default after this upgrade. I think this is something given we have a relatively fixed environment we can do better with.

Let this not take away from the excellent work done here. I cannot begin to imagine how much time you have saved me. Kudos

Update: I have completed 5 more updates with no new issues.

[mattab]

Perhaps this is just waiting for a fix upstream but since it is likely to require users to do a second rework of their sites tracking code for optimal deployment it is worth special mention

fyi indeed we haven't yet fixed it into core: matomo-org/matomo#12785
once it will be matomo.js and matomo.php we will still keep backward compatibility with piwik.js|php so you won't have to change your tracking codes.

[jawrat]

question: am I missing something or have packages for the latest builds not been published to the repo? I was going to update to 3.6 but it doesn't appear that the repo has been updated to that....
@aureq @mattab ?