get_dirsize() is causing file upload time outs and dashboards slowdowns

[brianlayman]

In our configuration, we have blogs with large numbers of uploaded files. We've many authors for some blogs and a number of years worth of their uploads. One of our blogs has 170K uploaded files.

We'd removed the quota stats section from the dashboard when we found it was causing the dashboard to take a minute or more to load on some blogs. Eventually, uploads started behaving the same way. Taking incredibly long for the smallest files and timing out for anything of moderate size.

I tracked it down to get_dirsize()

The tiny file I was uploading was taking 36.65015 seconds from beginning to end of wp_handle_upload. I made one change and it took the time down to 0.00601 seconds.

I changed get_dirsize() to:
function get_dirsize($dir) {

return 1;

}

Admittedly we are an outside case with crazy numbers of files, but this has to be affecting others to lesser degrees too. Constantly scaling the directory structures calling filesize() for every file, recursing into each sub-directory, is just not scalable.

[donncha]

For cases like this it probably makes sense if the quota system and display of sizes were optional.

[donncha]

In [2025] I added simple caching of the dirsize. It's invalidated when a file is uploaded or deleted but should still help with loading the dashboard.

Will still add an option to disable quota checking.

[donncha]

In [2026] I added a new option to disable the space checks. Closing ticket.

[jamescollins]

Replying to donncha:

In [2025] I added simple caching of the dirsize. It's invalidated when a file is uploaded or deleted but should still help with loading the dashboard.

Donncha, it's not a big deal but wouldn't you be better off using transients for caching the blog size? It would save you from having to manually check if the cache should have expired.

set_transient('dirsize_cache', $dirsize, 3600);

instead of

update_option()

and

get_transient('dirsize_cache')

instead of

get_option()

[donncha]

Good idea! Changed to transients in [2027]