Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] Funnels Plugin #23078

Open
4 tasks done
wiseguy1394 opened this issue Feb 26, 2025 · 14 comments
Open
4 tasks done

[Bug] Funnels Plugin #23078

wiseguy1394 opened this issue Feb 26, 2025 · 14 comments
Labels
Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. To Triage An issue awaiting triage by a Matomo core team member

Comments

@wiseguy1394
Copy link

What happened?

We have bought and activated the funnels plugin. On last friday we started to use this plugin. Now all our stats are deprecated, because the archive cron is hanging with following logs:

INFO      [2025-02-21 15:01:35] 142  Finished archiving for site 1, 232 API requests, Time elapsed: 3802.717s [1 / 12 done]
INFO      [2025-02-21 15:01:35] 142  Start processing archives for site 2.
INFO      [2025-02-21 15:01:35] 142  Finished archiving for site 2, 0 API requests, Time elapsed: 0.029s [2 / 12 done]
INFO      [2025-02-21 15:01:35] 142  Start processing archives for site 3.
INFO      [2025-02-21 15:01:35] 142    Will invalidate archived reports for today in site ID = 3's timezone (2025-02-21 00:00:00).
INFO      [2025-02-21 15:01:37] 142    Will invalidate archived reports for yesterday in site ID = 3's timezone (2025-02-20 00:00:00).
INFO      [2025-02-22 00:57:39] 142  Archived website id 3, period = day, date = 2025-02-21, segment = '', 16498 visits found. Time elapsed: 27.612s
INFO      [2025-02-22 00:57:40] 142  Archived website id 3, period = day, date = 2025-02-20, segment = '', plugin = Funnels, report = Funnels_funnel_nb_conversions_5, 17834 visits found. Time elapsed: 16.536s
INFO      [2025-02-22 00:57:40] 142  Archived website id 3, period = day, date = 2025-02-19, segment = '', plugin = Funnels, report = Funnels_funnel_nb_conversions_5, 18480 visits found. Time elapsed: 17.066s
INFO      [2025-02-22 00:57:40] 142  Archived website id 3, period = day, date = 2025-02-18, segment = '', plugin = Funnels, report = Funnels_funnel_nb_conversions_5, 17677 visits found. Time elapsed: 14.077s
INFO      [2025-02-22 00:57:41] 142  Archived website id 3, period = day, date = 2025-02-17, segment = '', plugin = Funnels, report = Funnels_funnel_nb_conversions_5, 17118 visits found. Time elapsed: 14.919s
INFO      [2025-02-22 16:49:13] 142  Archived website id 3, period = day, date = 2025-02-20, segment = '', plugin = Funnels, report = Funnels_funnel_sum_exits_5, 17834 visits found. Time elapsed: 0.804s
INFO      [2025-02-22 16:49:13] 142  Archived website id 3, period = day, date = 2025-02-19, segment = '', plugin = Funnels, report = Funnels_funnel_sum_exits_5, 18480 visits found. Time elapsed: 0.804s
INFO      [2025-02-22 16:49:14] 142  Archived website id 3, period = day, date = 2025-02-18, segment = '', plugin = Funnels, report = Funnels_funnel_sum_exits_5, 17677 visits found. Time elapsed: 0.804s
INFO      [2025-02-22 16:49:14] 142  Archived website id 3, period = day, date = 2025-02-17, segment = '', plugin = Funnels, report = Funnels_funnel_sum_exits_5, 17118 visits found. Time elapsed: 0.804s
INFO      [2025-02-22 16:49:14] 142  Archived website id 3, period = day, date = 2025-02-16, segment = '', plugin = Funnels, report = Funnels_funnel_nb_conversions_5, 15891 visits found. Time elapsed: 9.528s
INFO      [2025-02-23 08:23:22] 142  Archived website id 3, period = day, date = 2025-02-20, segment = '', plugin = Funnels, report = Funnels_funnel_sum_entries_5, 17834 visits found. Time elapsed: 0.792s
INFO      [2025-02-23 08:23:22] 142  Archived website id 3, period = day, date = 2025-02-19, segment = '', plugin = Funnels, report = Funnels_funnel_sum_entries_5, 18480 visits found. Time elapsed: 0.792s
INFO      [2025-02-23 08:23:22] 142  Archived website id 3, period = day, date = 2025-02-18, segment = '', plugin = Funnels, report = Funnels_funnel_sum_entries_5, 17677 visits found. Time elapsed: 0.792s
INFO      [2025-02-23 08:23:23] 142  Archived website id 3, period = day, date = 2025-02-17, segment = '', plugin = Funnels, report = Funnels_funnel_sum_entries_5, 17118 visits found. Time elapsed: 0.792s
INFO      [2025-02-23 08:23:23] 142  Archived website id 3, period = day, date = 2025-02-16, segment = '', plugin = Funnels, report = Funnels_funnel_sum_exits_5, 15891 visits found. Time elapsed: 0.792s
INFO      [2025-02-24 00:33:16] 142  Archived website id 3, period = day, date = 2025-02-20, segment = '', plugin = Funnels, report = Funnels_flow_5, 17834 visits found. Time elapsed: 1.050s
INFO      [2025-02-24 00:33:17] 142  Archived website id 3, period = day, date = 2025-02-19, segment = '', plugin = Funnels, report = Funnels_flow_5, 18480 visits found. Time elapsed: 1.050s
INFO      [2025-02-24 00:33:17] 142  Archived website id 3, period = day, date = 2025-02-18, segment = '', plugin = Funnels, report = Funnels_flow_5, 17677 visits found. Time elapsed: 1.050s
INFO      [2025-02-24 00:33:18] 142  Archived website id 3, period = day, date = 2025-02-17, segment = '', plugin = Funnels, report = Funnels_flow_5, 17118 visits found. Time elapsed: 1.050s
INFO      [2025-02-24 00:33:18] 142  Archived website id 3, period = day, date = 2025-02-16, segment = '', plugin = Funnels, report = Funnels_funnel_sum_entries_5, 15891 visits found. Time elapsed: 1.050s
INFO      [2025-02-24 23:13:30] 142  Archived website id 3, period = day, date = 2025-02-20, segment = '', plugin = Funnels, report = Funnels_exits_5, 17834 visits found. Time elapsed: 0.795s
INFO      [2025-02-24 23:13:30] 142  Archived website id 3, period = day, date = 2025-02-19, segment = '', plugin = Funnels, report = Funnels_exits_5, 18480 visits found. Time elapsed: 1.253s
INFO      [2025-02-24 23:13:31] 142  Archived website id 3, period = day, date = 2025-02-18, segment = '', plugin = Funnels, report = Funnels_exits_5, 17677 visits found. Time elapsed: 1.253s
INFO      [2025-02-24 23:13:32] 142  Archived website id 3, period = day, date = 2025-02-17, segment = '', plugin = Funnels, report = Funnels_exits_5, 17118 visits found. Time elapsed: 1.253s
INFO      [2025-02-24 23:13:33] 142  Archived website id 3, period = day, date = 2025-02-16, segment = '', plugin = Funnels, report = Funnels_flow_5, 15891 visits found. Time elapsed: 1.003s

There are no errors logged but it is hanging.

What should happen?

Archive cron should run as always.

How can this be reproduced?

We have activated the plugin and created some funnels and now the archive cron hangs.

Matomo version

5.2.2

PHP version

8.2.27

Server operating system

No response

What browsers are you seeing the problem on?

No response

Computer operating system

No response

Relevant log output

INFO      [2025-02-21 15:01:35] 142  Finished archiving for site 1, 232 API requests, Time elapsed: 3802.717s [1 / 12 done]
INFO      [2025-02-21 15:01:35] 142  Start processing archives for site 2.
INFO      [2025-02-21 15:01:35] 142  Finished archiving for site 2, 0 API requests, Time elapsed: 0.029s [2 / 12 done]
INFO      [2025-02-21 15:01:35] 142  Start processing archives for site 3.
INFO      [2025-02-21 15:01:35] 142    Will invalidate archived reports for today in site ID = 3's timezone (2025-02-21 00:00:00).
INFO      [2025-02-21 15:01:37] 142    Will invalidate archived reports for yesterday in site ID = 3's timezone (2025-02-20 00:00:00).
INFO      [2025-02-22 00:57:39] 142  Archived website id 3, period = day, date = 2025-02-21, segment = '', 16498 visits found. Time elapsed: 27.612s
INFO      [2025-02-22 00:57:40] 142  Archived website id 3, period = day, date = 2025-02-20, segment = '', plugin = Funnels, report = Funnels_funnel_nb_conversions_5, 17834 visits found. Time elapsed: 16.536s
INFO      [2025-02-22 00:57:40] 142  Archived website id 3, period = day, date = 2025-02-19, segment = '', plugin = Funnels, report = Funnels_funnel_nb_conversions_5, 18480 visits found. Time elapsed: 17.066s
INFO      [2025-02-22 00:57:40] 142  Archived website id 3, period = day, date = 2025-02-18, segment = '', plugin = Funnels, report = Funnels_funnel_nb_conversions_5, 17677 visits found. Time elapsed: 14.077s
INFO      [2025-02-22 00:57:41] 142  Archived website id 3, period = day, date = 2025-02-17, segment = '', plugin = Funnels, report = Funnels_funnel_nb_conversions_5, 17118 visits found. Time elapsed: 14.919s
INFO      [2025-02-22 16:49:13] 142  Archived website id 3, period = day, date = 2025-02-20, segment = '', plugin = Funnels, report = Funnels_funnel_sum_exits_5, 17834 visits found. Time elapsed: 0.804s
INFO      [2025-02-22 16:49:13] 142  Archived website id 3, period = day, date = 2025-02-19, segment = '', plugin = Funnels, report = Funnels_funnel_sum_exits_5, 18480 visits found. Time elapsed: 0.804s
INFO      [2025-02-22 16:49:14] 142  Archived website id 3, period = day, date = 2025-02-18, segment = '', plugin = Funnels, report = Funnels_funnel_sum_exits_5, 17677 visits found. Time elapsed: 0.804s
INFO      [2025-02-22 16:49:14] 142  Archived website id 3, period = day, date = 2025-02-17, segment = '', plugin = Funnels, report = Funnels_funnel_sum_exits_5, 17118 visits found. Time elapsed: 0.804s
INFO      [2025-02-22 16:49:14] 142  Archived website id 3, period = day, date = 2025-02-16, segment = '', plugin = Funnels, report = Funnels_funnel_nb_conversions_5, 15891 visits found. Time elapsed: 9.528s
INFO      [2025-02-23 08:23:22] 142  Archived website id 3, period = day, date = 2025-02-20, segment = '', plugin = Funnels, report = Funnels_funnel_sum_entries_5, 17834 visits found. Time elapsed: 0.792s
INFO      [2025-02-23 08:23:22] 142  Archived website id 3, period = day, date = 2025-02-19, segment = '', plugin = Funnels, report = Funnels_funnel_sum_entries_5, 18480 visits found. Time elapsed: 0.792s
INFO      [2025-02-23 08:23:22] 142  Archived website id 3, period = day, date = 2025-02-18, segment = '', plugin = Funnels, report = Funnels_funnel_sum_entries_5, 17677 visits found. Time elapsed: 0.792s
INFO      [2025-02-23 08:23:23] 142  Archived website id 3, period = day, date = 2025-02-17, segment = '', plugin = Funnels, report = Funnels_funnel_sum_entries_5, 17118 visits found. Time elapsed: 0.792s
INFO      [2025-02-23 08:23:23] 142  Archived website id 3, period = day, date = 2025-02-16, segment = '', plugin = Funnels, report = Funnels_funnel_sum_exits_5, 15891 visits found. Time elapsed: 0.792s
INFO      [2025-02-24 00:33:16] 142  Archived website id 3, period = day, date = 2025-02-20, segment = '', plugin = Funnels, report = Funnels_flow_5, 17834 visits found. Time elapsed: 1.050s
INFO      [2025-02-24 00:33:17] 142  Archived website id 3, period = day, date = 2025-02-19, segment = '', plugin = Funnels, report = Funnels_flow_5, 18480 visits found. Time elapsed: 1.050s
INFO      [2025-02-24 00:33:17] 142  Archived website id 3, period = day, date = 2025-02-18, segment = '', plugin = Funnels, report = Funnels_flow_5, 17677 visits found. Time elapsed: 1.050s
INFO      [2025-02-24 00:33:18] 142  Archived website id 3, period = day, date = 2025-02-17, segment = '', plugin = Funnels, report = Funnels_flow_5, 17118 visits found. Time elapsed: 1.050s
INFO      [2025-02-24 00:33:18] 142  Archived website id 3, period = day, date = 2025-02-16, segment = '', plugin = Funnels, report = Funnels_funnel_sum_entries_5, 15891 visits found. Time elapsed: 1.050s
INFO      [2025-02-24 23:13:30] 142  Archived website id 3, period = day, date = 2025-02-20, segment = '', plugin = Funnels, report = Funnels_exits_5, 17834 visits found. Time elapsed: 0.795s
INFO      [2025-02-24 23:13:30] 142  Archived website id 3, period = day, date = 2025-02-19, segment = '', plugin = Funnels, report = Funnels_exits_5, 18480 visits found. Time elapsed: 1.253s
INFO      [2025-02-24 23:13:31] 142  Archived website id 3, period = day, date = 2025-02-18, segment = '', plugin = Funnels, report = Funnels_exits_5, 17677 visits found. Time elapsed: 1.253s
INFO      [2025-02-24 23:13:32] 142  Archived website id 3, period = day, date = 2025-02-17, segment = '', plugin = Funnels, report = Funnels_exits_5, 17118 visits found. Time elapsed: 1.253s
INFO      [2025-02-24 23:13:33] 142  Archived website id 3, period = day, date = 2025-02-16, segment = '', plugin = Funnels, report = Funnels_flow_5, 15891 visits found. Time elapsed: 1.003s

Validations

@wiseguy1394 wiseguy1394 added Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. To Triage An issue awaiting triage by a Matomo core team member labels Feb 26, 2025
@sgiehl
Copy link
Member

sgiehl commented Feb 26, 2025

Hey @wiseguy1394,
Sorry to hear you are having problems with archiving. Setting up Funnels might cause old data to be rearchived for the new Funnels. Depending on the amount of data that might take some time.
If you run archiving in debug mode -vvv it should print what it is currently doing. That might help to understand where it is hanging though.
Did you already try aborting the archiving command and restarting it?

@wiseguy1394
Copy link
Author

@sgiehl I checked the logs again and now it has some new entries:

INFO      [2025-02-26 10:42:06] 142  Archived website id 3, period = day, date = 2025-02-20, segment = '', plugin = Funnels, report = Funnels_entries_5, 17834 visits found. Time elapsed: 1.077s
INFO      [2025-02-26 10:42:07] 142  Archived website id 3, period = day, date = 2025-02-19, segment = '', plugin = Funnels, report = Funnels_entries_5, 18480 visits found. Time elapsed: 1.077s
INFO      [2025-02-26 10:42:08] 142  Archived website id 3, period = day, date = 2025-02-18, segment = '', plugin = Funnels, report = Funnels_entries_5, 17677 visits found. Time elapsed: 1.078s
INFO      [2025-02-26 10:42:08] 142  Archived website id 3, period = day, date = 2025-02-17, segment = '', plugin = Funnels, report = Funnels_entries_5, 17118 visits found. Time elapsed: 1.078s
INFO      [2025-02-26 10:42:09] 142  Archived website id 3, period = day, date = 2025-02-16, segment = '', plugin = Funnels, report = Funnels_exits_5, 15891 visits found. Time elapsed: 1.078s

Is it normal that it takes over 24h to calculate these reports for one day?
Should I restart regardless of the progress?

@wiseguy1394
Copy link
Author

If I run the archive command manually it is not hanging anymore...

@wiseguy1394
Copy link
Author

@sgiehl We have restarted our cron yesterday with -vvv option applied. This is the log output since restart:

INFO      [2025-02-27 14:50:08] 146  ---------------------------
INFO      [2025-02-27 14:50:08] 146  INIT
INFO      [2025-02-27 14:50:08] 146  Running Matomo 5.2.2 as Super User
INFO      [2025-02-27 14:50:08] 146  ---------------------------
INFO      [2025-02-27 14:50:08] 146  NOTES
INFO      [2025-02-27 14:50:08] 146  - Async process archiving supported, using CliMulti.
INFO      [2025-02-27 14:50:08] 146  - Reports for today will be processed at most every 21600 seconds. You can change this value in Matomo UI > Settings > General Settings.
INFO      [2025-02-27 14:50:08] 146  - Maximum 1000 archives will be processed (soft limit).
INFO      [2025-02-27 14:50:08] 146  - Archiving was last executed without error 4 hours 35 min ago.
INFO      [2025-02-27 14:50:08] 146  - Continuing ongoing archiving run by pulling from shared idSite queue.
INFO      [2025-02-27 14:50:08] 146  Will skip segments archiving for today unless they were created recently
INFO      [2025-02-27 14:50:08] 146  ---------------------------
INFO      [2025-02-27 14:50:08] 146  START
INFO      [2025-02-27 14:50:08] 146  Starting Matomo reports archiving...
INFO      [2025-02-27 14:50:08] 146  2 out of 3 archivers running currently
DEBUG     [2025-02-27 14:50:08] 146  Applying queued rearchiving...

We have no clue whats the problem.
Our cluster has no load:

Image

Our database has no load either:

Image

So with these stats we would say that the cron is doing nothing, it's not going anywhere.

@sgiehl
Copy link
Member

sgiehl commented Mar 3, 2025

Hm... Did you already create a couple of Funnels? Might be possible that the cron takes a while to create proper invalidations for those Funnels then. This might happen after the Applying queued rearchiving... message is printed.
If that's the case, this might be self-healing after a while. If all archiving jobs are remaining hanging after that message we might need to check in detail what's happening.

@dieisraels
Copy link

Hi @sgiehl,

Here is some more information about our problems:

The cron job has been running in debug mode for over 2 days now (see below for log output). There are thousands of DEBUG log entries before the next INFO entry is made. The last INFO entry was made 53 hours ago:

INFO [2025-03-03 00:56:42] 145 Will invalidate archived reports for today in site ID = 3's timezone (2025-03-03 00:00:00).
INFO [2025-03-03 00:56:42] 145 Will invalidate archived reports for yesterday in site ID = 3's timezone (2025-03-02 00:00:00).

There seems to be no parallel processing and the actual preparation of the reports is extremely delayed or does not happen at all. This is also shown by the graphs on the utilization of our systems above. More resources would therefore probably not solve the problem.

These problems have only occurred since the “Funnels” plugin was installed and activated. We therefore assume a direct connection.

What do we need to do to ensure that our reports are displayed correctly again?

Cron log output:

matomo-archive-output-2025-03.log

@sgiehl
Copy link
Member

sgiehl commented Mar 5, 2025

Thanks for the additional information. I'm actually not familiar enough with the Funnels plugins in order to say if that one could cause such problems. I would assume it might have triggered rearchiving of a lot of Funnels reports possibly.

There is a "known" issue that slows down archiving if there are too many invalidations queued.

Could you check your archive_invalidations table. How many records does it have? Are there many of them where name like "%Funnels%"?

@dieisraels
Copy link

Oh, yes. There are a lot of records: The table has 3,801,869 records. 3,800,610 of these records have “Funnels” in their name.

@sgiehl
Copy link
Member

sgiehl commented Mar 5, 2025

@AltamashShaikh are there any known issues with Funnels plugin creating so many invalidations?

@AltamashShaikh
Copy link
Contributor

@sgiehl When a new Funnel is created, it will try to archive for older dates based on rearchive_reports_in_past_last_n_months Config setting.

@dieisraels What is the value of rearchive_reports_in_past_last_n_months in your config/config.ini.php file ?

@dieisraels
Copy link

The option rearchive_reports_in_past_last_n_months is not included in the config/config.ini.php file. I therefore assume that the Matomo standard is used.

@AltamashShaikh
Copy link
Contributor

In that case it will fallback to default which is 6 months.

@sgiehl Do you think deleting all the archive_invalidations and setting rearchive_reports_in_past_last_n_months=1 will solve the issue ?

@dieisraels
Copy link

We also use a lot of segments. Could it also help to activate the following option?
rearchive_reports_in_past_exclude_segments = 1

Just to be sure: These options do not affect the creation of reports that have been explicitly invalidated?

@sgiehl
Copy link
Member

sgiehl commented Mar 10, 2025

@dieisraels As far as I know that option should only have an affect on reports that are automatically invalidated by the system. You should then still be able to manually invalidate such reports using console command.

You could try removing the Funnel invalidations that are for dates/segments you may not need.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Potential Bug Something that might be a bug, but needs validation and confirmation it can be reproduced. To Triage An issue awaiting triage by a Matomo core team member
Projects
None yet
Development

No branches or pull requests

4 participants