Transient key best practice

[Micky Hulse]

Hello,

I'm writing a class that gets a JSON response from YouTube's API and
caches the parsed data into a WP transient.

I was tentatively planning on creating transient keys that look like:

namespace_youtube:playlist_or_user|||cache_time|||number_of_items

Where:

1. "namespace_youtube" = Make sure I don't stomp on other keys and
easier to find.

2. "playlist_or_user" = Will be either a playlist ID or a username.

3. "cache_time" = How long to cache this transient.

4. "number_of_items" = Number of videos that were cached in this transient.

5. "|||" = Just an easy delimiter for me to search on if I need to
get/parse transient keys.

Goal:

I want the ability to cache multiple different types of YouTube API
calls and have them be cached based on playlist ID or username, how
long it was it's supposed to be cached for and the number of items
cached. In other words, there might be times where I need to request
the same playlist but vary it's output by the cache time and number of
items requested from the API.

Question(s):

Based upon my goal, does the above transient key naming convention make sense?

What's common transient caching naming practice for situations like I
describe above?

For when it comes to transients, is there a way to have simpler
transient key (just simply "namespace_youtube", for example) naming
convention yet vary, and have the ability to access, the data stored
in that transient?

Please let me know if I need to clarify my questions.

Thanks!
M

[Filippo Pisano]

You can simply md5 your key and query db for the md5 string corresponding
to your API request.

Post by Micky Hulse
Hello,
I'm writing a class that gets a JSON response from YouTube's API and
caches the parsed data into a WP transient.
namespace_youtube:playlist_or_user|||cache_time|||number_of_items
1. "namespace_youtube" = Make sure I don't stomp on other keys and
easier to find.
2. "playlist_or_user" = Will be either a playlist ID or a username.
3. "cache_time" = How long to cache this transient.
4. "number_of_items" = Number of videos that were cached in this transient.
5. "|||" = Just an easy delimiter for me to search on if I need to
get/parse transient keys.
I want the ability to cache multiple different types of YouTube API
calls and have them be cached based on playlist ID or username, how
long it was it's supposed to be cached for and the number of items
cached. In other words, there might be times where I need to request
the same playlist but vary it's output by the cache time and number of
items requested from the API.
Based upon my goal, does the above transient key naming convention make sense?
What's common transient caching naming practice for situations like I
describe above?
For when it comes to transients, is there a way to have simpler
transient key (just simply "namespace_youtube", for example) naming
convention yet vary, and have the ability to access, the data stored
in that transient?
Please let me know if I need to clarify my questions.
Thanks!
M
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Micky Hulse]

Thanks for the blazing fast reply Filippo! I really appreciate it! :)

On Mon, Sep 9, 2013 at 12:42 PM, Filippo Pisano

Post by Filippo Pisano
You can simply md5 your key and query db for the md5 string corresponding
to your API request.

So, are you saying that I should do something like:

$transient_key =
wp_hash("namespace_youtube:playlist_or_user|||cache_time|||number_of_items"
, 'md5');

In that case, could I drop "namespace_youtube:" and just md5 the rest
for the key?

Sorry if I'm asking silly questions ... I just want to make sure I'm
following standard procedure here. :)

Thanks so much!

Cheers,
M

[Nikola Nikolov]

Additionally, if you want to be able to get all transients created by your
plugin(for whatever reasons you might have), you can simply prefix the
md5-ed hash with something(like "nsyt_{$md5_hash}").

If you wouldn't need to get all of your plugin's transients, you can
most-likely drop the namespace, although it can never hurt - md5() is 1)
fast, 2) the hash is always the same length. So 10 characters more or less
doesn't make a huge impact IMO.

Post by Micky Hulse
Thanks for the blazing fast reply Filippo! I really appreciate it! :)
On Mon, Sep 9, 2013 at 12:42 PM, Filippo Pisano

Post by Filippo Pisano
You can simply md5 your key and query db for the md5 string corresponding
to your API request.

$transient_key =
wp_hash("namespace_youtube:playlist_or_user|||cache_time|||number_of_items"
, 'md5');
In that case, could I drop "namespace_youtube:" and just md5 the rest
for the key?
Sorry if I'm asking silly questions ... I just want to make sure I'm
following standard procedure here. :)
Thanks so much!
Cheers,
M
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Filippo Pisano]

Post by Micky Hulse
Thanks for the blazing fast reply Filippo! I really appreciate it! :)
On Mon, Sep 9, 2013 at 12:42 PM, Filippo Pisano

Post by Filippo Pisano
You can simply md5 your key and query db for the md5 string corresponding
to your API request.

$transient_key =
wp_hash("namespace_youtube:playlist_or_user|||cache_time|||number_of_items"
, 'md5');
In that case, could I drop "namespace_youtube:" and just md5 the rest
for the key?

Sure, you have to keep in mind that the DB column options.option_name is a
64 chars varchar and so you could not use such a long string (moreover when
using transients and additional record is created using your original
transient name + '_timeout'). Thus, IMO the hash technique should suits
your needs.

Post by Micky Hulse
Sorry if I'm asking silly questions ... I just want to make sure I'm
following standard procedure here. :)

No silly question ;)

Post by Micky Hulse
Thanks so much!
Cheers,
M
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Micky Hulse]

Awesome!!! :)

Thanks so much for the details and super quick replies Filippo and
Nikola, I really appreciate it!

I now have plans to MD5 and prefix a "namespace_" to the transient key.

I'm soooo glad I asked for help! I owe you guys an Oregon micro brew. :)

Have an awesome day!

Cheers,
M

P.S., (mostly a note to my future self):

In my previous e-mail I gave this example:

$transient_key =
wp_hash("namespace_youtube:playlist_or_user|||cache_time|||number_of_items",
'md5');

Looks like that second argument doesn't work like how I thought it did ...

http://core.trac.wordpress.org/browser/tags/3.6/wp-includes/pluggable.php#L1392

Apparently it's a string that's used to salt a call to:

hash_hmac('md5', $data, $salt);

For my needs, it's looking like I should just use PHP's md5($str) function.

[Otto]

It is a very bad idea to use mutating keys for transients. You run the
risk of storing transients in ways such that you never actually try to
get them again, thus creating lots and lots of entries in the database
for those transients, for no reason.

The important part of a caching mechanism is that you read from the
cache. If your key is constantly changing in a way that makes doing
this read less likely, then your cache isn't very useful, is it?

I would absolutely not include cache time or number of items in the
key, because these can change and then you'll never look up that
cached data again, leaving it orphaned.

-Otto

Post by Micky Hulse
Hello,
I'm writing a class that gets a JSON response from YouTube's API and
caches the parsed data into a WP transient.
namespace_youtube:playlist_or_user|||cache_time|||number_of_items
1. "namespace_youtube" = Make sure I don't stomp on other keys and
easier to find.
2. "playlist_or_user" = Will be either a playlist ID or a username.
3. "cache_time" = How long to cache this transient.
4. "number_of_items" = Number of videos that were cached in this transient.
5. "|||" = Just an easy delimiter for me to search on if I need to
get/parse transient keys.
I want the ability to cache multiple different types of YouTube API
calls and have them be cached based on playlist ID or username, how
long it was it's supposed to be cached for and the number of items
cached. In other words, there might be times where I need to request
the same playlist but vary it's output by the cache time and number of
items requested from the API.
Based upon my goal, does the above transient key naming convention make sense?
What's common transient caching naming practice for situations like I
describe above?
For when it comes to transients, is there a way to have simpler
transient key (just simply "namespace_youtube", for example) naming
convention yet vary, and have the ability to access, the data stored
in that transient?
Please let me know if I need to clarify my questions.
Thanks!
M
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Nikola Nikolov]

Otto - doesn't setting transient's expiry time automatically remove it from
the database once it's time's up? I haven't looked into core precisely how
this works, but it would make sense for expired transients to get
removed(during cron or whenever).

Post by Otto
It is a very bad idea to use mutating keys for transients. You run the
risk of storing transients in ways such that you never actually try to
get them again, thus creating lots and lots of entries in the database
for those transients, for no reason.
The important part of a caching mechanism is that you read from the
cache. If your key is constantly changing in a way that makes doing
this read less likely, then your cache isn't very useful, is it?
I would absolutely not include cache time or number of items in the
key, because these can change and then you'll never look up that
cached data again, leaving it orphaned.
-Otto

Post by Micky Hulse
Hello,
I'm writing a class that gets a JSON response from YouTube's API and
caches the parsed data into a WP transient.
namespace_youtube:playlist_or_user|||cache_time|||number_of_items
1. "namespace_youtube" = Make sure I don't stomp on other keys and
easier to find.
2. "playlist_or_user" = Will be either a playlist ID or a username.
3. "cache_time" = How long to cache this transient.
4. "number_of_items" = Number of videos that were cached in this

transient.

Post by Micky Hulse
5. "|||" = Just an easy delimiter for me to search on if I need to
get/parse transient keys.
I want the ability to cache multiple different types of YouTube API
calls and have them be cached based on playlist ID or username, how
long it was it's supposed to be cached for and the number of items
cached. In other words, there might be times where I need to request
the same playlist but vary it's output by the cache time and number of
items requested from the API.
Based upon my goal, does the above transient key naming convention make

sense?

Post by Micky Hulse
What's common transient caching naming practice for situations like I
describe above?
For when it comes to transients, is there a way to have simpler
transient key (just simply "namespace_youtube", for example) naming
convention yet vary, and have the ability to access, the data stored
in that transient?
Please let me know if I need to clarify my questions.
Thanks!
M
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Nicholas Ciske]

Nope. It only gets removed the next time it's requested.

http://www.stumiller.me/does-wordpress-delete-expired-transients-from-the-database/

May be fixed in 3.7:
http://core.trac.wordpress.org/ticket/20316

There are plugins that use cron to clean up orphan records, but core does not... yet.

http://wordpress.org/plugins/artiss-transient-cleaner/

http://wordpress.org/plugins/delete-expired-transients/

_________________________
Nick Ciske
http://thoughtrefinery.com/
@nciske

Post by Nikola Nikolov
doesn't setting transient's expiry time automatically remove it from
the database once it's time's up?

[Micky Hulse]

Post by Nikola Nikolov
Otto - doesn't setting transient's expiry time automatically remove it from
the database once it's time's up? I haven't looked into core precisely how
this works, but it would make sense for expired transients to get
removed(during cron or whenever).

Great question!

Post by Nikola Nikolov
Nope. It only gets removed the next time it's requested.

Good to know. I just assumed they got swept automatically.

Also, thanks for linkages.

Post by Nikola Nikolov
http://core.trac.wordpress.org/ticket/20316
There are plugins that use cron to clean up orphan records, but core does not... yet.
http://wordpress.org/plugins/artiss-transient-cleaner/
http://wordpress.org/plugins/delete-expired-transients/

More good links and info. That's helpful. Thanks! :)

[Micky Hulse]

One last update to this thread, I noticed that my transients weren't
getting cached when using:

plugin_name + md5(values...)

When I shortened the key name, then it caching would work.

I took another look at the docs:

<http://codex.wordpress.org/Function_Reference/set_transient>

... and saw:

"Should be 45 characters or less in length."

I had overlooked that bit of info until now. Good to know! :)

Thanks again to everyone for the help!

[Micky Hulse]

One last comment for completeness's sake:

"The option_name field in the options table has a record length of 64
characters. So, taking into account the prefixes added to the
transient name, a transient name can only be a maximum of 45
characters (or 40 characters for network transients). The problem here
is that WordPress doesn’t check. It doesn’t fail either – it creates
the transients but simply truncates the name. And because each of the
transient record pairs has a different length prefix then they each
end up with a different name – this means the transient can’t be
retrieved."

Via: <http://www.artiss.co.uk/2013/05/using-transients>

[Otto]

Note that transients are not always stored in the options table, this
is just one possible place that they can be put. So that length
limitation isn't enforced in WordPress itself because other systems
might not have the same limitation.

Still, 45 characters is reasonably long for any realistic case. MD5
produces 32 characters, leaving you another 13 for a unique prefix.
Although I'd try to avoid even that long, generally speaking.

-Otto

Post by Micky Hulse
"The option_name field in the options table has a record length of 64
characters. So, taking into account the prefixes added to the
transient name, a transient name can only be a maximum of 45
characters (or 40 characters for network transients). The problem here
is that WordPress doesn’t check. It doesn’t fail either – it creates
the transients but simply truncates the name. And because each of the
transient record pairs has a different length prefix then they each
end up with a different name – this means the transient can’t be
retrieved."
Via: <http://www.artiss.co.uk/2013/05/using-transients>
_______________________________________________
wp-hackers mailing list
http://lists.automattic.com/mailman/listinfo/wp-hackers

[Micky Hulse]

Excellent advice! Thanks Otto!

As an ExpressionEngine convert, I'm loving every minute of Wordpress.
It's just so flexible and fun to work with (plus, the community of
developers is top notch). :)

Have a great day all!

Post by Otto
Note that transients are not always stored in the options table, this
is just one possible place that they can be put. So that length
limitation isn't enforced in WordPress itself because other systems
might not have the same limitation.
Still, 45 characters is reasonably long for any realistic case. MD5
produces 32 characters, leaving you another 13 for a unique prefix.
Although I'd try to avoid even that long, generally speaking.

[Micky Hulse]

Hi Otto! Thanks for the help, I really appreciate! :)

Post by Otto
It is a very bad idea to use mutating keys for transients. You run the
risk of storing transients in ways such that you never actually try to
get them again, thus creating lots and lots of entries in the database
for those transients, for no reason.
The important part of a caching mechanism is that you read from the
cache. If your key is constantly changing in a way that makes doing
this read less likely, then your cache isn't very useful, is it?

Thank you for pointing this out. I had a feeling that I might be going
a little overboard with my key names.

Post by Otto
I would absolutely not include cache time or number of items in the
key, because these can change and then you'll never look up that
cached data again, leaving it orphaned.

Sounds like good advice. I'll drop the number of items and cache time.

My thinking was that on one page, I might want to request a YouTube
playlist ID of XYZ that's 10 items total (the YT API lets you limit
the number of items it returns) and is cached for 30 minutes. On
another page I'd want to get the same playlist but with 20 items and
have that cached for 60 minutes.

I think I can live without that type of granular control though.
Thanks again for the helpful feedback, it's much appreciated! :)

Oh, I was also wondering about the automatic removal of transients
from the database ... Looks like Nikola's got that question covered.

Cheers,
M