WilWade.com A Blog of...

Random Rows from a Database Cache Table

2011 April 15

Here is a slight problem I had the other day. I had a need to retrieve a single random row from a database table that was caching some items with several conditions:

  • The rows in the database were constantly changing
  • There could be a very large number of rows (20,000)
  • There could be a small number of rows (2)
  • A count of rows in the table could become incorrect within milliseconds
  • Must be fast

I know this is likely a uncommon set of conditions. It would appear to be from the large number of posts regarding random rows from databases. Most of the solutions have items in direct contrast with some of my conditions. For example most databases have some form of this MySQL code (see a nice list from Pete Freitag)

SELECT column FROM table
ORDER BY RAND()
LIMIT 1;

However for large datasets is very slow (See the multitude of complaining about it). Thus as it violates a set of conditions it is useless.

There are many posts about using the number of rows or the a table without holes in an auto increment field, but these were also unhelpful.

A few do a double call to the database, one to get the number of rows and then another with a limit like so:

SELECT column FROM table
ORDER BY RAND()
LIMIT "[random from table count]","[number of rows]";

This would almost work until I remembered this fact about my project: After I used a row once, it would be deleted or anything older than an hour would also be deleted. All the table really is a cache where we throw already mostly random items. And depending on the number of users that means that my table row count could be changing quickly.

So the solution I came up with for a random cache table row is on the insert side of things.

Add a column to the table say randomColumn. Likely you will want to index the column for faster ordering on the other end.

Generate a random number between 1 and say 100. Increase your upper limit for greater randomness. You can also have [my]SQL generate the number for you, but likely it is faster for PHP or whatever you are coding in. FLOOR(8 * RAND()) + 1 will give you a random number in MySQL.

INSERT INTO table
(data, randomColumn) VALUES (Some data, "[Generated Random Number]"");

Then on the select side order by the randomColumn.

SELECT * FROM table
ORDER BY randomColumn
LIMIT "[number of rows]";

Now you are inserting your columns into a “random order” and the speed loss of generating a random number for each row with each column is gone! Also since the columns are being deleted after use this will always bring up a new “random” row.

Learning Git

2011 January 17

Git is a piece of version control software that is used on projects big and small. The largest project is likely the Linux Kernel. Git can be downloaded from git-scm.com.

Git however is a rather difficult system to wrap your mind around, mostly because it is so powerful. Often with great power comes great complexity, or at least more of it. (Hmm… an AK-47 is really simple, yet powerful. Perhaps that is why it is so popular and Git is not? Of course not everyone needs a version control system…) Today thanks to a post on my local PHP Users Group, I found an easy way to learn Git in more depth and a great way to teach others.

gitimmersion.com

It has a wonderful walk through of Git in 50 “labs.” Covering basic usage, to some more advanced concepts. I wish I had known about this 7 months ago when I started using Git for my own projects.

Videos in Print: A Bash Script Example

2010 May 08

One of the best things about Linux/Unix is the ability to automate through the BASH scripting. If you are not familiar with Bash then I suggest you Google something like Bash tutorials. I suggest that you skip over the technical parts, and focus on what I used Bash to accomplish.

I am currently working on typesetting our family blog so that we can have a paper version for the future, and just a way to flip through pre-commented photos. On our blog however, we have many videos. Until someone comes up with a slightly more inexpensive and usable method of paper video, it is rather difficult to print videos. So I thought that perhaps a film strip of sorts would help “replace” the video and give some transformation from film to print. However we are talking about 30+ videos. I do not have the time to go through and manually extract frames and then put them together. So I wondered if I could automate the process.

Using mplayey to output the frames to png, I was able to then automatically put them into a grid using ImageMagick. In the end it was rather easy.

Gist Script!

Note: You really should never run a shell script you download from the internet without first at least reading over it a little to make sure it is not going to wipe your drive or anything. This will not do that, but it is always good to remember.

Anyway the script is fairly simple. Most of the length is from a little bit of error checking and default setting. There is likely easier ways to do this, but I did it the way I did and you can feel free to rewrite it. Basically it is really just to commands. One is the mplayer command to extract the frames as pngs, the other is the ImageMagick command, montage, to create the strip.

To use the command you can do something as simple as feed it a video file. The default is to create a 5 by 5 grid (see the example below), but you can pass it other parameters.

For those with little experience with Bash a parameter is something that comes after the command.

command parameter1 parameter2 … parameterN

For this script the parameters are as follows:

  1. The video file name
  2. The number of frames wide for the strip to be
  3. The number of frames tall for the strip to be
  4. The width of each frame
  5. The height of each frame

(The frame aspect ratio is preserved. White is added for fill.)

Note that in order to get frames that were evenly spaced I needed to spit all the frames out to png first. There are some ways around this, but most people have more space than they know what to do with. This does mean that it is really only effective for shorter clips.

@TODO Jan 20, 2014 It is possible to do this better with ffmpeg, but I have not yet updated the script.

Here is an example. This is a video of an April fools joke on my in-laws. We “left” our daughter on their doorstep. Note that this is the best example, as it has the most visual action. The process does not work as well for audio centered videos.

Surprise

A Common Site, Using CSS3

2010 March 25

We often think about CSS3 as being in the future. This thing that we can never actually use. Well the folks over at Days of Wonder (or at least their web designers) didn’t think so. They are using the CSS3 on their page for their promotional contest for their new board game: Mystery Express. Check below the screen shots for the actual code they are using, and visit the Whodunit Contest site for yourself.

Chrome 4.1
Chrome 4.1

Firefox 3.6
Firefox 3.6

Internet Explorer 8
Internet Explorer 8

Here is the CSS for the Text (which by the way this is the largest site I have ever seen text twisted with the transform style.):

-webkit-transform: rotate(-5deg);
-moz-transform: rotate(-5deg);
-o-transform: rotate(-5deg);
transform: rotate(-5deg);

And did you notice the other bit of CSS3 on the page? There is rounding on the corners of the dark blue and at the very top. Again the code:

/* Rounded corners */
-webkit-border-top-left-radius: 5px; /* Safari 3 and up */
-webkit-border-bottom-left-radius: 5px; /* Safari 3 and up */
-moz-border-radius-topleft: 5px; /* Gecko browsers */
-moz-border-radius-bottomleft: 5px; /* Gecko browsers */
border-top-left-radius: 5px; /* Everything else - limited support at the moment */
border-bottom-left-radius: 5px; /* Everything else - limited support at the moment */

The text for Firefox looks fairly bad at this point. It has trouble with rotating text. Explorer has a different problem however, one that is not shown in the above screenshot. The text falls off the piece of “paper”! In my opinion they should have set the margins smaller for IE so that the black text would not run off into the dark brown background. Oh well. Apparently they only really tested in Safari (which I assume looks very close to Chrome) and Chrome/Chromium.

So where are you seeing CSS3 around the web?