Workqueue: kmmcd mmc_rescan crash with Raspbian kernel 4.4.38-v7

[fkbreitl]

My Raspberry Pi 3 runs unstable and crashes every few days. Before the crash /var/log/messages shows the following error repeatedly with a separation of a few minutes:

Jan 25 18:36:22 raspberrypi kernel: [1103404.218839] kworker/u8:2 D 805b770c 0 20119 2 0x00000000
Jan 25 18:36:22 raspberrypi kernel: [1103404.218886] Workqueue: kmmcd mmc_rescan
Jan 25 18:36:22 raspberrypi kernel: [1103404.218923] [<805b770c>] (__schedule) from [<805b7c84>] (schedule+0x50/0xa8)
Jan 25 18:36:22 raspberrypi kernel: [1103404.218945] [<805b7c84>] (schedule) from [<8046d568>] (__mmc_claim_host+0xb8/0x1cc)
Jan 25 18:36:22 raspberrypi kernel: [1103404.218974] [<8046d568>] (__mmc_claim_host) from [<8046d6ac>] (mmc_get_card+0x30/0x34)
Jan 25 18:36:22 raspberrypi kernel: [1103404.219006] [<8046d6ac>] (mmc_get_card) from [<80475780>] (mmc_sd_detect+0x2c/0x80)
Jan 25 18:36:22 raspberrypi kernel: [1103404.219030] [<80475780>] (mmc_sd_detect) from [<8046fcbc>] (mmc_rescan+0xc8/0x324)
Jan 25 18:36:22 raspberrypi kernel: [1103404.219057] [<8046fcbc>] (mmc_rescan) from [<8003c82c>] (process_one_work+0x154/0x458)
Jan 25 18:36:22 raspberrypi kernel: [1103404.219080] [<8003c82c>] (process_one_work) from [<8003cb84>] (worker_thread+0x54/0x500)
Jan 25 18:36:22 raspberrypi kernel: [1103404.219100] [<8003cb84>] (worker_thread) from [<80042860>] (kthread+0xec/0x104)
Jan 25 18:36:22 raspberrypi kernel: [1103404.219122] [<80042860>] (kthread) from [<8000fbc8>] (ret_from_fork+0x14/0x2c)
Jan 25 18:36:23 raspberrypi rsyslogd-2007: action 'action 17' suspended, next retry is Wed Jan 25 18:37:53 2017 [try http://www.rsyslog.com/e/2007 ]

I am running Raspbian 8.0 (Jessie) with kernel

Linux raspberrypi 4.4.38-v7+ #938 SMP Thu Dec 15 15:22:21 GMT 2016 armv7l GNU/Linux

I am also discussing this at

http://raspberrypi.stackexchange.com/q/61050/58211.

I have also reported this at raspberrypi/firmware#731 but I think it fits better here. Unfortunately I cannot move the issue, so its kind of a duplicate, now. Maybe it should be deleted at firmware.

[Electron752]

1. Are you over-clocking the RPI?
2. Have you tried a different SD card? I know these cheap cards can wear out or break after being used for awhile. And they are relatively cheap so it might be worth trying a replacement.

I doubt this is a firmware issue. It could actually be a combo of a bad card plus a bug in the driver. It might well be that the driver thinks you've removed the card even though you haven't. And since pulling out the SD card while the RPI is running is a no-no, I could see why that might not have been tested well.

I know some effort is being made to replace this driver with an upstream version. Also 4.9 is now considered the stable version of the kernel on the RPI.

[fkbreitl]

1. No, I didn't change the clock speed.
2. Its is a high end 40 EUR Samsung MicroSDXC 128GB EVO Plus UHS-I Grade 1 Class 10 which two 8 GB partitions for /root and /home.

I have completely write tested the card with badblocks -w with the 5 different patterns before usage and found not a single bad block. A recent badblocks read test didn't show any errors either. I assume the card is perfect.

I just ordered another Pi 3 and a SanDisk Ultra 64GB SDXC (80 MB/Sek) Class 10 to rule out hardware issues.

[pelwell]

You missed of the header of that kernel, but a stall at that point looks like a symptom that something else hasn't released the MMC device, possibly because something failed during a previous operation. The sdhost driver will log error messages, so in the absence of those I'm leaning towards the ARM<->VPU mailbox stalling while trying to configure the clock for the SD interface.

Errors like this are often the sign of an inadequate power supply. What does "vcgencmd get_throttled" report on your system after some typical activity?

Also 4.9 is now considered the stable version of the kernel on the RPI.

No - it's still the next version until we make a Raspbian release using it.

[Electron752]

That is puzzling then...

Personally, I would have tried a very, very cheap card first to rule out hardware issue. Like the kind you get for $5 in the US. I've actually collected a whole bunch of those over the years.

What I meant by cheap cards, is the all these SD cards are cheap. I don't care how much you pay for them, they are essentially designed for the camera/camcorder/phone market so they don't typically get the abuse of something that's may well be on all the time.

[Electron752]

@pelwell I actually see mailbox stalls from time to time and I'm not 100% sure it's the power supply since I can sometimes see it even if the RPI is connected to a bench variable power supply that can do 5 amps.

I've found that especially in 64 bit, the RPI is unstable at 1.2GHz. Setting the max at 1.0GHz seems to help. But I also haven't ruled out a firmware issue.

[fkbreitl]

@pelwell What kernel header to you need? I am using the default Raspbian 8 Jessie kernel
Linux raspberrypi 4.4.34-v7+ #930 SMP Wed Nov 23 15:20:41 GMT 2016 armv7l GNU/Linux.

My power supply seems stable and sufficient as concluded from throttled=0x20000.
I obtained this value during a stress test that raised my CPU temperature to 80°C.

By the way if you google for kmmcd mmc_rescan you will find other people reporting this issue, too.

[fkbreitl]

Meanwhile I noticed that this problem is triggered by the chromium-browser (Version 51.0.2704.91 Built on Ubuntu 14.04, running on Raspbian 8.0) when several sites are open.

[JamesH65]

Please update to the latest kernel which may help, 4.9 is now the release version. Please report back the results.

[fkbreitl]

Unfortunately I still see the same problem after the upgrade.
Meanwhile I am using:
Linux raspberrypi 4.9.24-v7+ #993 SMP Wed Apr 26 18:01:23 BST 2017 armv7l GNU/Linux
and Chromium Version 56.0.2924.84 Built on Ubuntu 14.04, running on Raspbian 8.0

[dreamlayers]

I never saw this before, and now saw it a bunch of times running 4.9.80-v7+ during a very fast wget download. I guess this is what made the Raspberry Pi 2 B temporarily unresponsive, but it didn't crash and the download didn't fail.

[39933.969207] INFO: task kworker/2:2:5185 blocked for more than 120 seconds.
[39933.969232]       Tainted: G         C      4.9.80-v7+ #1098
[39933.969236] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[39933.969247] kworker/2:2     D    0  5185      2 0x00000000
[39933.969295] Workqueue: events_freezable mmc_rescan
[39933.969347] [<8072252c>] (__schedule) from [<80722a54>] (schedule+0x50/0xa8)
[39933.969363] [<80722a54>] (schedule) from [<805c30d8>] (__mmc_claim_host+0xb8/0x1cc)
[39933.969377] [<805c30d8>] (__mmc_claim_host) from [<805c321c>] (mmc_get_card+0x30/0x34)
[39933.969394] [<805c321c>] (mmc_get_card) from [<805cbc70>] (mmc_sd_detect+0x2c/0x80)
[39933.969410] [<805cbc70>] (mmc_sd_detect) from [<805c5ee0>] (mmc_rescan+0x1c8/0x38c)
[39933.969429] [<805c5ee0>] (mmc_rescan) from [<801367f8>] (process_one_work+0x158/0x454)
[39933.969450] [<801367f8>] (process_one_work) from [<80136b58>] (worker_thread+0x64/0x5b8)
[39933.969464] [<80136b58>] (worker_thread) from [<8013ca74>] (kthread+0x10c/0x124)
[39933.969483] [<8013ca74>] (kthread) from [<80108154>] (ret_from_fork+0x14/0x20)

[JamesH65]

Does this still happen on the latest 4.14 kernel?

This issue will be closed within 30 days unless further interactions are posted. If you wish this issue to remain open, please add a comment. A closed issue may be reopened if requested.