Using wp-content/uploads to store dynamically created js and CSS

[Nicola Peluchetti]

Hi,

our plugin at the moment generates CSS and Javascript files in PHP. This
means that we append a link /script to the website and then a PHP script
generates the relevant CSS/js.

We need to do this with CSS because we generate it from LESS files and give
the user the ability to modify variables, we do this with Javascript
because we use requirejs and we wanted to avoid using global variables in
javascript, so we create modules dinamically and build things on the fly.

We had a fallback system for storing CSS ( APC/Filesystem/DB ) while js was
just read.

In any case this has proven to be too slow and unreliable, so we'd like to
go back serving static css and js files and we thought we can use
wp-content/upload to store compilade .js and .css files.

I've read this article
http://ottopress.com/2011/tutorial-using-the-wp_filesystem/ where Otto says
to inline, but if i inline my js / css is not cached by the browser
right?And we are talking about 500kb so it's not trivial.

What do you think of the approach and have you got anything better to
suggest?


*Nicola Peluchetti - Senior PHP Developer @ Timely*
Twitter: @nik_peluchetti <https://twitter.com/#!/nik_peluchetti>
Facebook: nicola.peluchetti <https://www.facebook.com/nicola.peluchetti>
Stackoverflow: Stackoverflow<http://stackoverflow.com/users/397861/nicola-peluchetti>
Mobile: [phone]

[Otto]

On Wed, Jan 8, 2014 at 2:39 PM, Nicola Peluchetti

Post by Nicola Peluchetti
I've read this article
http://ottopress.com/2011/tutorial-using-the-wp_filesystem/ where Otto says
to inline, but if i inline my js / css is not cached by the browser
right?And we are talking about 500kb so it's not trivial.

First, are you seriously dynamically generating over 500kb of CSS/JS
files? Or are you really only generating some of the pieces of those
files while the rest of them is relatively static? What is the
"static" to "dynamic" ratio?

I ask because it's perfectly possible to make the static parts static,
and the dynamic parts inline. JS variables, inline CSS, etc. Just
because you have it all clumped together currently doesn't mean that
that is the only possible way.

Secondly, if you do generate a ton of content for whatever reasons,
you should not store any CSS or JS content in the /uploads directory.
Instead, make your own directory under /wp-content and store the files
there instead. The WP_Filesystem object has a function called
wp_content_dir() that will return the content directory on the
"remote" filesystem. Using this and the mkdir() function in that same
object, you can create your own directory to store the files in, write
them, and then use those URLs instead.

Storing content that will be included into the page in the uploads
folder is generally unsafe, due to some configurations of shared
hosting. It's relatively safe for inline media images, video, etc),
but it is not safe for CSS or JS content.

-Otto

[Nicola Peluchetti]

Yes all the CSS for the plugin ( including bootstrap3 ) is generated
dynamically and it's quite big. Of course it's not generated on every
request, it's stored and regenerated only if needed. Js is just built from
pieces at the moment.

We've had a folder under wp-content for a long time where we stored themes
but that has always been a pain, one time out of three there was no write
access. In fact we are deprecating that in 2.0.

I thought about wp-content/uploads because i guess that, if there is just
one folder which is writable, it must be that.

Why do you say it's unsafe?Is there some possibility that malitious users
could exploit that?I mean is it a safety risk?Because obviously i don't
care if anyone can access my CSS and JS code.


*Nicola Peluchetti - Senior PHP Developer @ Timely*
Twitter: @nik_peluchetti <https://twitter.com/#!/nik_peluchetti>
Facebook: nicola.peluchetti <https://www.facebook.com/nicola.peluchetti>
Stackoverflow: Stackoverflow<http://stackoverflow.com/users/397861/nicola-peluchetti>
Mobile: [phone]

Post by Otto
On Wed, Jan 8, 2014 at 2:39 PM, Nicola Peluchetti

Post by Nicola Peluchetti
I've read this article
http://ottopress.com/2011/tutorial-using-the-wp_filesystem/ where Otto

says

Post by Nicola Peluchetti
to inline, but if i inline my js / css is not cached by the browser
right?And we are talking about 500kb so it's not trivial.

First, are you seriously dynamically generating over 500kb of CSS/JS
files? Or are you really only generating some of the pieces of those
files while the rest of them is relatively static? What is the
"static" to "dynamic" ratio?
I ask because it's perfectly possible to make the static parts static,
and the dynamic parts inline. JS variables, inline CSS, etc. Just
because you have it all clumped together currently doesn't mean that
that is the only possible way.
Secondly, if you do generate a ton of content for whatever reasons,
you should not store any CSS or JS content in the /uploads directory.
Instead, make your own directory under /wp-content and store the files
there instead. The WP_Filesystem object has a function called
wp_content_dir() that will return the content directory on the
"remote" filesystem. Using this and the mkdir() function in that same
object, you can create your own directory to store the files in, write
them, and then use those URLs instead.
Storing content that will be included into the page in the uploads
folder is generally unsafe, due to some configurations of shared
hosting. It's relatively safe for inline media images, video, etc),
but it is not safe for CSS or JS content.
-Otto
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Otto]

On Wed, Jan 8, 2014 at 4:02 PM, Nicola Peluchetti

Post by Nicola Peluchetti
We've had a folder under wp-content for a long time where we stored themes
but that has always been a pain, one time out of three there was no write
access. In fact we are deprecating that in 2.0.

This is what the WP_Filesystem is for. If you need to write a file,
and you have no direct access, then WP_Filesystem will get credentials
from the user and you can use those to get write access through the
WP_Filesystem object.

Post by Nicola Peluchetti
Why do you say it's unsafe?Is there some possibility that malitious users
could exploit that?I mean is it a safety risk?Because obviously i don't
care if anyone can access my CSS and JS code.

On some setups, with some configurations, yes, it can be a safety
risk. If I'm on the same server as you (shared hosting), then I could,
in theory, overwrite your files in the uploads folder with contents of
my choosing. Depending on the server configuration, of course.

For images, this doesn't matter so much. For CSS or JS files, I can
write code there which will exploit your site when your site includes
it on the page.

-Otto

[Brian Fegter]

Nicola,

Instead of generating an actual static file, why not use a rewrite? You
won't have to deal with the file system at all and any quirks that come
with your setup.

Heres a gist with the setup I use for this type of thing. This allows you
to use object caching and your CDN sees this as a static file as well.
https://gist.github.com/inspectorfegter/8325711

Let me know if you have any question on this approach. Just another way to
skin a cat.

Thanks!
Brian





On Wed, Jan 8, 2014 at 4:02 PM, Nicola Peluchetti <

Post by Nicola Peluchetti
Yes all the CSS for the plugin ( including bootstrap3 ) is generated
dynamically and it's quite big. Of course it's not generated on every
request, it's stored and regenerated only if needed. Js is just built from
pieces at the moment.
We've had a folder under wp-content for a long time where we stored themes
but that has always been a pain, one time out of three there was no write
access. In fact we are deprecating that in 2.0.
I thought about wp-content/uploads because i guess that, if there is just
one folder which is writable, it must be that.
Why do you say it's unsafe?Is there some possibility that malitious users
could exploit that?I mean is it a safety risk?Because obviously i don't
care if anyone can access my CSS and JS code.
Facebook: nicola.peluchetti <https://www.facebook.com/nicola.peluchetti>
Stackoverflow: Stackoverflow<
http://stackoverflow.com/users/397861/nicola-peluchetti>
Mobile: [phone]

Post by Otto
On Wed, Jan 8, 2014 at 2:39 PM, Nicola Peluchetti

Post by Nicola Peluchetti
I've read this article
http://ottopress.com/2011/tutorial-using-the-wp_filesystem/ where Otto

says

Post by Nicola Peluchetti
to inline, but if i inline my js / css is not cached by the browser
right?And we are talking about 500kb so it's not trivial.

First, are you seriously dynamically generating over 500kb of CSS/JS
files? Or are you really only generating some of the pieces of those
files while the rest of them is relatively static? What is the
"static" to "dynamic" ratio?
I ask because it's perfectly possible to make the static parts static,
and the dynamic parts inline. JS variables, inline CSS, etc. Just
because you have it all clumped together currently doesn't mean that
that is the only possible way.
Secondly, if you do generate a ton of content for whatever reasons,
you should not store any CSS or JS content in the /uploads directory.
Instead, make your own directory under /wp-content and store the files
there instead. The WP_Filesystem object has a function called
wp_content_dir() that will return the content directory on the
"remote" filesystem. Using this and the mkdir() function in that same
object, you can create your own directory to store the files in, write
them, and then use those URLs instead.
Storing content that will be included into the page in the uploads
folder is generally unsafe, due to some configurations of shared
hosting. It's relatively safe for inline media images, video, etc),
but it is not safe for CSS or JS content.
-Otto
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Justas Butkus]

Hello Brian.

If I see it right, and please correct me if I miss something, your
proposed solution gives best performance, when site uses a CDN, or at
least has a caching reverse HTTP proxy in front of WordPress.

In general case - sites don't have anything like this. And then every
new user (some of whom might have troubles with cache, but let's say
that's just some 15% increase in traffic) requires your site to build
that file. If you have a pre-generated JavaScript, served as an actual
static file, the WordPress isn't even touched, but with your approach
every request by each new user causes new generation, which might impose
significant increase in resources utilization.

Maybe you have some considerations how to counter-act that?
I mean except recommending each and every user to have a
CDN/caching-proxy, which would be good, just as good as asking everyone
to use PHP 5.5, which was discussed here few weeks ago. :-)
--
Regards,
Justas B.

Post by Brian Fegter
Nicola,
Instead of generating an actual static file, why not use a rewrite? You
won't have to deal with the file system at all and any quirks that come
with your setup.
Heres a gist with the setup I use for this type of thing. This allows you
to use object caching and your CDN sees this as a static file as well.
https://gist.github.com/inspectorfegter/8325711
Let me know if you have any question on this approach. Just another way to
skin a cat.
Thanks!
Brian

[Nikola Nikolov]

I've seen a similar solution, where the generated JS/CSS was stored in the
DB - I know it's not the best solution(since it's still going to load most
of WordPress just to grab that code from the DB), but it seems like an okay
one. Plus if you do output some caching headers(where it says "#Add caching
headers" in his code), then most of the users browsers should have
caching(I believe all modern browsers have caching enabled by default, so
you have to know what you're doing in order to disable it - for instance
developers do that when debugging stuff).

Post by Justas Butkus
Hello Brian.
If I see it right, and please correct me if I miss something, your
proposed solution gives best performance, when site uses a CDN, or at least
has a caching reverse HTTP proxy in front of WordPress.
In general case - sites don't have anything like this. And then every new
user (some of whom might have troubles with cache, but let's say that's
just some 15% increase in traffic) requires your site to build that file.
If you have a pre-generated JavaScript, served as an actual static file,
the WordPress isn't even touched, but with your approach every request by
each new user causes new generation, which might impose significant
increase in resources utilization.
Maybe you have some considerations how to counter-act that?
I mean except recommending each and every user to have a
CDN/caching-proxy, which would be good, just as good as asking everyone to
use PHP 5.5, which was discussed here few weeks ago. :-)
--
Regards,
Justas B.
Nicola,

Post by Brian Fegter
Instead of generating an actual static file, why not use a rewrite? You
won't have to deal with the file system at all and any quirks that come
with your setup.
Heres a gist with the setup I use for this type of thing. This allows you
to use object caching and your CDN sees this as a static file as well.
https://gist.github.com/inspectorfegter/8325711
Let me know if you have any question on this approach. Just another way to
skin a cat.
Thanks!
Brian

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Nicola Peluchetti]

Hi Brian,

we've gone that route before and it's a total pain. We have more than
100.000 installs of our plugin and we've had countless problems. Gzipping
was a pain too. We will provide this as a fallback but we will try to store
things in wp-content/ourfolder as Otto suggested.

Thanks to all of you for your help and suggestions.


*Nicola Peluchetti - Senior PHP Developer @ Timely*
Twitter: @nik_peluchetti <https://twitter.com/#!/nik_peluchetti>
Facebook: nicola.peluchetti <https://www.facebook.com/nicola.peluchetti>
Stackoverflow: Stackoverflow<http://stackoverflow.com/users/397861/nicola-peluchetti>
Mobile: [phone]

Post by Nikola Nikolov
I've seen a similar solution, where the generated JS/CSS was stored in the
DB - I know it's not the best solution(since it's still going to load most
of WordPress just to grab that code from the DB), but it seems like an okay
one. Plus if you do output some caching headers(where it says "#Add caching
headers" in his code), then most of the users browsers should have
caching(I believe all modern browsers have caching enabled by default, so
you have to know what you're doing in order to disable it - for instance
developers do that when debugging stuff).

Post by Justas Butkus
Hello Brian.
If I see it right, and please correct me if I miss something, your
proposed solution gives best performance, when site uses a CDN, or at

least

Post by Justas Butkus
has a caching reverse HTTP proxy in front of WordPress.
In general case - sites don't have anything like this. And then every new
user (some of whom might have troubles with cache, but let's say that's
just some 15% increase in traffic) requires your site to build that file.
If you have a pre-generated JavaScript, served as an actual static file,
the WordPress isn't even touched, but with your approach every request by
each new user causes new generation, which might impose significant
increase in resources utilization.
Maybe you have some considerations how to counter-act that?
I mean except recommending each and every user to have a
CDN/caching-proxy, which would be good, just as good as asking everyone

to

Post by Justas Butkus
use PHP 5.5, which was discussed here few weeks ago. :-)
--
Regards,
Justas B.
Nicola,

Post by Brian Fegter
Instead of generating an actual static file, why not use a rewrite? You
won't have to deal with the file system at all and any quirks that come
with your setup.
Heres a gist with the setup I use for this type of thing. This allows

you

Post by Justas Butkus

Post by Brian Fegter
to use object caching and your CDN sees this as a static file as well.
https://gist.github.com/inspectorfegter/8325711
Let me know if you have any question on this approach. Just another way

to

Post by Justas Butkus

Post by Brian Fegter
skin a cat.
Thanks!
Brian

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Brian Fegter]

@Justas & @Nicola - I understand your point there and I agree it's better
to hit the file system before the DB. I missed the point that this was a
distributed plugin compared to a single site..

Post by Nikola Nikolov
I've seen a similar solution, where the generated JS/CSS was stored in the
DB - I know it's not the best solution(since it's still going to load most
of WordPress just to grab that code from the DB), but it seems like an okay
one. Plus if you do output some caching headers(where it says "#Add caching
headers" in his code), then most of the users browsers should have
caching(I believe all modern browsers have caching enabled by default, so
you have to know what you're doing in order to disable it - for instance
developers do that when debugging stuff).

Post by Justas Butkus
Hello Brian.
If I see it right, and please correct me if I miss something, your
proposed solution gives best performance, when site uses a CDN, or at

least

Post by Justas Butkus
has a caching reverse HTTP proxy in front of WordPress.
In general case - sites don't have anything like this. And then every new
user (some of whom might have troubles with cache, but let's say that's
just some 15% increase in traffic) requires your site to build that file.
If you have a pre-generated JavaScript, served as an actual static file,
the WordPress isn't even touched, but with your approach every request by
each new user causes new generation, which might impose significant
increase in resources utilization.
Maybe you have some considerations how to counter-act that?
I mean except recommending each and every user to have a
CDN/caching-proxy, which would be good, just as good as asking everyone

to

Post by Justas Butkus
use PHP 5.5, which was discussed here few weeks ago. :-)
--
Regards,
Justas B.
Nicola,

Post by Brian Fegter
Instead of generating an actual static file, why not use a rewrite? You
won't have to deal with the file system at all and any quirks that come
with your setup.
Heres a gist with the setup I use for this type of thing. This allows

you

Post by Justas Butkus

Post by Brian Fegter
to use object caching and your CDN sees this as a static file as well.
https://gist.github.com/inspectorfegter/8325711
Let me know if you have any question on this approach. Just another way

to

Post by Justas Butkus

Post by Brian Fegter
skin a cat.
Thanks!
Brian

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Nicola Peluchetti]

Otto,

if i choose to ouput the CSS directly in the page without triggering
another request, what's the best action to use, wp_head?
http://codex.wordpress.org/Plugin_API/Action_Reference/wp_head


*Nicola Peluchetti - Senior PHP Developer @ Timely*
Twitter: @nik_peluchetti <https://twitter.com/#!/nik_peluchetti>
Facebook: nicola.peluchetti <https://www.facebook.com/nicola.peluchetti>
Stackoverflow: Stackoverflow
<http://stackoverflow.com/users/397861/nicola-peluchetti>
Mobile: [phone]

Post by Brian Fegter
@Justas & @Nicola - I understand your point there and I agree it's better
to hit the file system before the DB. I missed the point that this was a
distributed plugin compared to a single site..

Post by Nikola Nikolov
I've seen a similar solution, where the generated JS/CSS was stored in

the

Post by Nikola Nikolov
DB - I know it's not the best solution(since it's still going to load

most

Post by Nikola Nikolov
of WordPress just to grab that code from the DB), but it seems like an

okay

Post by Nikola Nikolov
one. Plus if you do output some caching headers(where it says "#Add

caching

Post by Nikola Nikolov
headers" in his code), then most of the users browsers should have
caching(I believe all modern browsers have caching enabled by default, so
you have to know what you're doing in order to disable it - for instance
developers do that when debugging stuff).

Post by Justas Butkus
Hello Brian.
If I see it right, and please correct me if I miss something, your
proposed solution gives best performance, when site uses a CDN, or at

least

Post by Justas Butkus
has a caching reverse HTTP proxy in front of WordPress.
In general case - sites don't have anything like this. And then every

new

Post by Nikola Nikolov

Post by Justas Butkus
user (some of whom might have troubles with cache, but let's say that's
just some 15% increase in traffic) requires your site to build that

file.

Post by Nikola Nikolov

Post by Justas Butkus
If you have a pre-generated JavaScript, served as an actual static

file,

Post by Nikola Nikolov

Post by Justas Butkus
the WordPress isn't even touched, but with your approach every request

by

Post by Nikola Nikolov

Post by Justas Butkus
each new user causes new generation, which might impose significant
increase in resources utilization.
Maybe you have some considerations how to counter-act that?
I mean except recommending each and every user to have a
CDN/caching-proxy, which would be good, just as good as asking everyone

to

Post by Justas Butkus
use PHP 5.5, which was discussed here few weeks ago. :-)
--
Regards,
Justas B.
Nicola,

Post by Brian Fegter
Instead of generating an actual static file, why not use a rewrite?

You

Post by Nikola Nikolov

Post by Justas Butkus

Post by Brian Fegter
won't have to deal with the file system at all and any quirks that

come

Post by Nikola Nikolov

Post by Justas Butkus

Post by Brian Fegter
with your setup.
Heres a gist with the setup I use for this type of thing. This allows

you

Post by Justas Butkus

Post by Brian Fegter
to use object caching and your CDN sees this as a static file as well.
https://gist.github.com/inspectorfegter/8325711
Let me know if you have any question on this approach. Just another

way

Post by Nikola Nikolov
to

Post by Justas Butkus

Post by Brian Fegter
skin a cat.
Thanks!
Brian

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Otto]

On Thu, Jul 31, 2014 at 8:05 AM, Nicola Peluchetti <

Post by Nicola Peluchetti
Otto,
if i choose to ouput the CSS directly in the page without triggering
another request, what's the best action to use, wp_head?
http://codex.wordpress.org/Plugin_API/Action_Reference/wp_head

For inline CSS, I'd really recommend adding it using the
wp_add_inline_style() function.

http://codex.wordpress.org/Function_Reference/wp_add_inline_style

-Otto