Excel 2011 for Mac, UNIX Time, and Visual Basic for Applications

Note to people looking for a formula: Yes, the code is here (for Mac and Windows, even). I tend to go long-winded even in technical articles, but if you’re dealing with converting UNIX time to Excel time, the answers lie below. You can skip ahead or read my brilliant and entertaining *cough* analysis first.

Microsoft Excel uses a method to represent time that is both smart and frustrating. How do they manage this? They take a good engineering solution, then fiddle with it. First some background:

Long before Y2K people who knew what they were doing had already abandoned the practice of using strings of text to represent dates in a computer. Using strings like “3/10/2011” to represent a date has plenty of drawbacks, from cultural (is that March 10th or October 3rd?) to performance-related (sort 3/10/2011, 4/2/1902, 3/8/2012). Therefore long ago people who were smarter than I am came up with other ways to represent time. Happliy, time is nice and linear. All you really need is a number line. Remember them? A number line stretches from zero to infinity in both directions. To measure time all you need to do is decide on a zero point and then any point in the history of the universe can be represented by some number of time units from that instant.

My first exposure to a more rational way to measure time was in the old MacOS. I don’t remember anymore exactly when the zero point was, or even what the units chosen were. I do remember that the number gets too big for the computer to handle some time in 2014. Ancient Macs will have a problem then. I blame the Aztecs.

The UNIX boys count seconds from January 1, 1970 at 00:00. You get special Geek Cred if you went to a party to celebrate second 1234567890 of the Unix epoch. 32-bit computers that use the Unix epoch will break in 2038, when the number of seconds gets too big to fit in 32 bits. (Note also that you can only go back a finite distance before the negative number goes out of the range the processor can handle.

That’s all well and good, but I’m here to discuss Microsoft Excel today, and in particular Microsoft Excel for Mac. Excel counts in days, but allows fractional values. 12.5 represents noon twelve days after the zero point. I haven’t checked, but I think this system works for dates far, far into the future. So good on Microsoft for coming up with it. (As long as you don’t need dates before the zero time. In Excel, those are just strings again).

Of course, there are a couple of caveats. First: the historical oddity. In Excel, the day February 29, 1900 exists. Alas, there never was such a day. Microsoft included this error because they wanted to be compatible with Lotus, who simply messed up. To change it now would cause problems, because the zero point for the Microsoft time is January 1, 1900. Every date in every spreadsheet would suddenly be off by one. A thousand years from now we may still be calculating time based on the insertion of a bogus day.

Oh, except that Microsoft time doesn’t always start in 1900, and here’s where things start to get squirrely. If you’re using Excel for Mac, the default day zero is January 1, 1904, so the bogus day vanishes (no negative dates in Excel, remember?). Mac Excel dates aren’t burdened by the bogus day. Except when they are. More on that in a bit.

I descended into Excel recently to write a macro that does fancy formatting based on data I dump from a Web-based tracking tool I’m building. The dates in my data are based on the UNIX epoch, so I need to convert them. I dump the raw data into one sheet and then display it correctly converted and formatted on the main sheet that people actually look at. Here’s the code I use in a cell of the spreadsheet that needs to show a converted date:

    =DATE(1970,1,1)+import!Z3/(60*60*24)

where the unix time is in cell Z3 of the ‘import’ sheet. This divides the unix time by the number of seconds in a day, which gives me the number of days that have passed since the UNIX zero time. The formula then adds on the number of days from the spreadsheet zero to the UNIX zero time. (I could just say 86,400 instead of 60*60*24, but this way I can tell at a glance I’m dealing with days, and speed will not be an issue.) Happily, this formula will work on both Mac and Windows versions of Excel, because the DATE function will return the right value for the start of the unix epoch based on whichever version of Excel is running.

So, no problem, right?

Well… except. I also have some more fancy work to do that requires scripting. The good news: Mac Excel 2011 uses Visual Basic for Applications (VBA), which while imperfect is a zillion times better than AppleScript. So away I went, coding with a twinkle in my eye and a song in my heart. To convert dates, I naturally followed the same plan I did in the sheet’s cells: get the value for 1/1/1970, then add the unix epoch days.

And the dates came out different. Yep, when scripting, Excel always uses the Windows zero time, even when the spreadsheet in question uses the Mac zero time. Dates calculated in cells in the sheet are four years different than dates calculated using the same method in a script.

Aargh. Of course, once I realized what the problem was, it was not too difficult to work around it. I just lost some of the portability of my code, because now it has to be tweaked based on what the zero date of the spreadsheet is.

An aside here: If you’re here to resolve some date confusion in your own Mac spreadsheet, I strongly recommend you start by going to Preferences->Calculation->Workbook Options and uncheck the “use 1904 date” option. Unfortunately it will not recalculate the dates already entered in your sheets, so if that’s a problem then it’s too late for you, bucko. Read on.

Here’s some not-as-portable-as-it-could-be code. You need to choose one of the first two lines based on whether your sheet uses mac dates or windows dates:

'excelZeroDate = DateSerial(1904, 1, 1)              ' mac zero
excelZeroDate = 0      ' DateSerial(1900, 1, 1)     ' windows zero
unixZeroDate = DateDiff("d", excelZeroDate, DateSerial(1970, 1, 1))

Then if I have a date in the ‘import’ sheet to convert, I can write something like:

startDate = DateAdd("s", Worksheets("import").Cells(dateRow, dateColumn), unixZeroDate)

The nice part is that these functions handled converting seconds and days for me. Overall it’s not a bad system if you overlook the part where a single application gives two different answers to the same question.

3

New Public Key

For those who have joined my tiny push for better internet privacy, please note that during an upgrade of my key software my old keychain got knocked akimbo. Everything’s fine now, but while I was at it I created a new public key using the newer (better) encryption. Hopefully I can revoke the old key, but in the meantime please go to my key page to get the new one.

The software upgrade that caused the trouble was to get around a bug in the way GPGMail and MacGPG2 interacted. The GPGMail guys have done a great job ironing out the wrinkles I encountered (as far as I can tell so far), and they’ve built an excellent installer that makes it much easier for Mac users to get up and running protecting themselves from prying eyes. I’ll be updating (and shortening) my instruction page soon.

Trying a Different Spam Filter

Every day, literally hundreds of spam comments are sent to this blog. I have a a couple lines of defense, and generally they work pretty well. My first defense is a product called Bad Behavior, which inspects incoming messages and blocks the ones that look malicious before the WordPress code is even started up. Stopping evil at this stage can save a lot of server resources, as well as prevent this site from being hijacked by an unknown WordPress vulnerability.

Comments that get through that layer are then inspected to see if they look suspicious. Ones that the inspection service doesn’t like get thrown into a bucket behind the scenes where I can inspect them and approve innocent comments that were mistakenly flagged as spam.

I have been using Akismet for that, and in general I’ve been pleased with the results. The only downside is that now there are so many suspicious comments that I’m afraid that I’ll miss actual legit comments that were improperly flagged. Scanning through a list of hundreds of comments each day is not effective and, really, not a good use of my time. So, I began to look for alternatives.

Defensio is similar to Akismet, in that comments are shipped off to some service somewhere and then returned with a grade. The main difference is in the administration interface that I see, where Defensio sorts the rejected spam comments to allow me to more quickly spot legitimate comments that were falsely flagged as spam.

You may have noticed a surge in the amount of spam around here. This is (I hope) a learning phase for Defensio, and eventually it will stop allowing 3% of the spam comments to get through. (Akismet is still running, but mostly in a “see? I told you so” capacity right now.) I’m a little confused, because some of the comments Defensio displays are rated at 100% spamminess by Defensio’s own service.

Please bear with me through this somewhat-more-spammy-than-usual phase. I’ll be checking for spam comments regularly, and watching to see if Defensio’s performance improves. Also, this is a particularly good time to leave comments, from a training-the-filter perspective.

Advance Notice of Unplanned Outage

Just a quick note to tell you guys that I’ve run into technical difficulties renewing the muddledramblings.com domain name. LiveRack sucks. Never, ever, register a domain with LiveRack.

Actually, I’m not sure you’d be able to register with them even if you wanted to; their payment acceptance portal seems to be broken. Thus, I cannot renew this domain. There is no way to contact anyone at LiveRack. A long time ago they listed contact information but never answered queries, now they don’t even bother pretending. LiveRack sucks. Never, ever, register a domain with LiveRack.

So, I decided now would be a good time to move the domain to a new registrar. I’d tried this some time ago, without success, because LiveRack did not respond to the request. This is, as you might have surmised, because LiveRack sucks. If I were you, I’d never, ever register a domain with LiveRack.

So, with time running out, I put my nose to the grindstone and got the right codes to move the domain despite LiveRack’s unresponsiveness. But, wait! It can’t be that easy… The domain is too close to expiring to move. I have to renew, then move. But I can’t renew, because LiveRack sucks. There was a time, long ago, when LiveRack didn’t suck. Those days are long gone. You see, these days there’s really no way to describe LiveRack without using the word “sucks”. I wish I’d never registered my domain with LiveRack.

Tomorrow I’ll try LiveRack’s renewal again, in case they’ve fixed it, and I’ve appealed to Enom to intervene as well. LiveRack is a reseller for Enom; LiveRack’s only role was to process my payments. Now they can’t even do that, apparently, which leads me to the inescapable conclusion that LiveRack sucks.

So, in a couple of days, muddledramblings.com may stop working. I’m still hopeful, but I wanted to let folks know ahead of time.

Oh, and if anyone asks you about LiveRack? Well, LiveRack sucks. Under no circumstances should anyone register a domain with them. Seriously.

Well, THAT Sucked

The last few days my Web host has been having a tough time. I don’t know the exact nature of the problem and I doubt I ever will, but this site has been broken. For a while it would not load at all, and then it was in ‘read-only mode’, Which meant that it was still performing terribly and I couldn’t even put up a notice that I knew things weren’t going well but the solution was out of my hands. Not a good situation when my credibility as a programmer is an important asset.

I couldn’t even make a backup.

Things seem to be getting back to normal (though they are not there yet – the site is still quite slow). There’s even a chance that I’m running on a brand-new server that is not being shared with as many other people. Or at least a brand-new server. Unfortunately, however, while I have come to appreciate iPage the company, which was very helpful and patient getting me up and running, iPage the service has not been so great.

I have vowed that the next move I make will be to a server that I control completely, so I can choose who shares it with me. I’m looking at Co-location deals now, though I might wimp out ant take the middle road. A VPS (virtual private server) gives me all the control of having my own machine, but in fact it’s an illusion — I still share physical hardware with an unknown number of others.

1

The Drupal Attitude

I’ve been doing some geekery with Drupal lately. Drupal is a free, open-source server application that makes it easier to build really complex Web sites. It allows you to create complex data types and establish relationships and do fancy database stuff… without actually touching the database. That’s not too shabby. Drupal is rapidly becoming more popular, but there are a few things standing between Drupal and world domination. At the top of the list is the Drupal Attitude.

I will illustrate with an example. Things will get geeky for a while as I set the stage, then mellow out as I focus on the human interactions between various groups.

From a technical standpoint, Drupal’s biggest flaw is that it sucks when it comes to many-to-many relationships. Imagine I have a data type called “shirt” and another called “color”. It is very easy for me to set up “shirt” so that it can have several colors. So, when I look at a specific shirt in my database I can see that it has red and yellow in it. That’s all pretty straightforward.

The catch comes when I want a list of all shirts with yellow in them. If I had direct control over the database, many-to-many relationships like this are trivial and do not diminish the performance of the server. Drupal has no built-in way to get a list of all shirts with yellow in them.

But wait! Drupal is open source, and better yet has been built to be easy to extend by outside programers. Into this glaring hole in Drupal several folks have stepped forward with modules that solve the problem in a variety of different ways. Some of these methods are clever (one uses the indexes built by the search engine, for instance), but all have trade-offs and weaknesses.

So, you’re a Drupal developer, and you want a list of shirts with yellow in them. Which module do you use? Each module works differently, each requires some installation and fiddling to get working. Then there are the two modules by the same guy that are for similar but different purposes, yet the actual differences are not spelled out very clearly. What would help a lot would be some concrete examples of when to use which.

Now we’re getting closer to the Drupal Attitude. Remember as I rant about this that all the modules I’m evaluating are free, posted by geeks who wanted to contribute to make Drupal better. So, some slack-cutting is in order. BUT…

I had already spent more time than I had available trying to figure out which module to use, when I found a question posted by a guy asking “can I use this module for x”, where x was very similar to what I needed. “Aha!” thought I, “Now we’ll get a definitive answer!” Except that the response to the question was, “In this discussion (the article was about the differences between two modules) we want to focus on generalities, not specific applications. You should download both modules and fiddle with them for a few hours to determine which is right for you.” Or something like that. Notably absent from the answer was a pointer to where specific questions would be answered.

The guy who asked the question responded a bit harshly, pretty much saying, “Would it kill you to just answer my question? I don’t want to spend hours learning something you already know and could tell me in fifteen seconds.”

Well, this is just the sort of uppity user that the Drupal community loves to hate. Several people piled on in defense of the developer who had refused to answer the question. “He’s doing this for free, he’s helping the community, you should be grateful, blah, blah, blah.” None of them deigned to answer the original question either. There is a real, entrenched cadre in the Drupal community that says, “we learned things the hard way, and you should too.” Who needs documentation when you can read the source code?

Let’s step back for a moment and ask ourselves, “Why did the developer give this code back to the Drupal community?” The obvious answer, the one everyone talks about, is that he wants to make things easier for other Drupal users. That is a noble motivation and one I wholeheartedly support. He wants to be useful. Perhaps he just isn’t aware that a huge part of utility of software lies in the documentation. Perhaps he isn’t aware that a few choice examples of what his modules are meant to accomplish would have cost him an hour of his time and improved the acceptance of his work dramatically. He’s a coder, after all, not a marketer or a technical writer.

Even with all that, however, when someone, in the form of a question, contributes to the documentation by providing a specific example, he didn’t answer the question. No light came on that even if that was not the place for the question, then spending five minutes creating an FAQ would have helped the community far more than adding a new feature to his software. So an opportunity to spend just a few seconds and make his contribution to the community better went completely ignored. His supporters congratulated him for not capitulating to the demands of his potential users for more clarity.

Any of them could have stepped up and helped the newbie, probably in ten words or less, but none did. None of them wanted improved documentation. “We had to learn it the hard way, so you should too,” with a side order of “we make lots of money because we’ve figured all this stuff out.” Ladies and gentlemen, the Drupal Attitude.

If the guy posted his module but doesn’t seem interested in making it useful, then why did he post it? Well, he’s certainly getting lots of love from the people who figured out his work the hard way. They can all feel good about how smart they are.

And in the end, should I be thankful this guy shared his work with the rest of us? Actually, no. In my case, the presence of his modules ultimately had negative value. They cost me time, and never getting an answer about which was appropriate for my task, I went with a module developed by someone else.

So, Drupal contributors: If you don’t want to document your module, and you don’t want to answer straightforward questions from people who need to get a job done in limited time, don’t bother posting your fucking module at all. I don’t have time for endless fiddling and I sure as hell don’t have time for the Drupal Attitude.

1

Jer’s Novel Writer 1.1.10 Released!

It’s a big milestone for me, getting back on track maintaining Jer’s Novel Writer. For those of you who are not familiar with it, JersNW is a word processor with features to help my scattered thoughts fall into some sort of coherent form. It’s based on three principles:

  1. When I get an idea, I want to be able to jot it down without losing my train of thought.
  2. I don’t want to have to remember stuff, and when I need to find something I wrote before I can look it up without getting caught up in the part of the story where it happened.
  3. When I’m stuck on a line or know it could be better, or need to check a fact, or need to track continuity, I can flag it and know I’ll be able to get back to it later.

There are features to help with the mechanics of writing as well (drag-and-drop chapters, flexible database, and so forth), but mainly I just want to keep my momentum when the creative juices are flowing and know that when nitty-gritty time comes I’ll find the places that need attention. Man that’s liberating.

If you’re interested, you can learn more at Jer’s Software Hut. Enjoy!

Oh yeah, it’s Mac-only.

3

Jer’s Software Hut Returns!

Deep below the Earth’s surface, in a steaming chamber somewhere between the Hayward and San Andreas faults, there is motion in the dancing shadows cast by the guttering torchlight. Dark, twisted creatures stir. They move slowly, their once-lean haunches soft from months of torpor. They shamble from their stony sleeping-nooks into the work chamber, and one by one they take their accustomed positions on the capstans. The great wheels resist at first, then start to turn with a shriek that slowly fades to a rumble.

From the throats of the horde comes a deep ululation punctuated by coarse barks, a sound that reverberates through the chamber until it has no source, merely presence. It is enough to weaken the knees of even the stoutest hero, a sound to chill the most stalwart heart.

The creatures are singing.

Though fearsome in sound, it is joy they express; the sublime joy of one who had no purpose being useful again. Far above them, in a peaceful neighborhood San Jose, those turning capstans once more power the forges and shrieking spark-throwing wheels of Jer’s Software Hut.

On the slab in the middle of the laboratory a figure stirs; what once was dead now breathes again.

2

Things I Learned while Moving to a new Web Host

You probably can’t tell, but this site is now being served by a different host. The reasons I switched were many, but once MMHosting got hacked I decided it was time to move. Then when a particular PHP library was not on their servers (one that allows WordPress to read the date of an uploaded image), I actually did the move.

After quite a bit of looking around in which all the dang hosts started to look the same, I chose iPage. They are not quite the cheapest, but they purchase carbon offsets for wind-generated power. They also had a stronger emphasis on security.

At this writing, I still don’t know if my new host has the needed php library. I’ll be finding out when the dust settles. If not, I can install it in my site myself, I suppose, but let’s keep fingers crossed.

So, I learned a few things, and remembered a few others.

  1. Among sftp clients, RBrowser may have my favorite interface but it’s glacially slow with multiple files.
  2. Without ssh access to my site (none of the big hosting companies allow that on their cheap plans), I had to use phpMyAdmin to copy my databases over. Here’s an interesting bit of trivia: if your data has the phrase ‘drop database’ anywhere in it, phpMyAdmin will stop executing the import right there and then. This is to protect you from SQL injection attacks, where people sneak malicious data into your database that later gets executed as an instruction. ‘Drop database’ can be pretty devastating, so the software simply refuses to complete the import, even if the phrase is safe in the text of a post.

    The way phpMyAdmin is configured at my new host, however, when it stops, it doesn’t say why. It doesn’t even admit that anything went wrong, or indicate in any way that not all the data was imported. This can be inconvenient when you have a bulletin board for a product that has a drag-and-drop database feature. (Now it has a drag-and-drop database.)

  3. You can’t tell Safari not to uncompress zip archives it downloads (that I could find), but the original zip files can be found in the trash.
  4. jerssoft_phpb5 and jerssoft_phpbb5 are not the same thing, no matter how many hours you spend banging your head on them.

So iPage has been great (although their control panel is not completely Safari-friendly when it comes to processing payments). I’ve interacted with them in three different ways now — phone, chat, and a support ticket. Two of the interactions were due to the afore-mentioned payment glitch, and once for technical support trying to get my files copied and the site ready to go before I switched the domain registry. Although I didn’t get the answers I was hoping for, the tech was competent and knew what she was doing.

If I continue to be pleased with iPage, I will provide a link for those looking for a Web host. Because we all need Web hosts these days, don’t we?

Evil Flash Cookies

NOTE June 1, 2019: This is a rather old post, and most major browsers have addressed this problem directly. I ultimately solved the problem by simply not installing Flash. It’s dead tech.

For a long time now we’ve been aware of browser cookies. These are little bits of data that a Web developer can set on your computer to keep track of your visits, or which ads you’ve seen, and things like that. Cookies are regulated by your browser and you can set up rules to reduce the amount that other people learn about your habits.

Way back in the day the makers of Flash realized that it would be handy to store little bits of information on the user’s computer as well. They developed LSO’s, otherwise known as Flash Cookies, to do that. This site uses an LSO so the banner animation doesn’t run every time you change pages. (See my rant about html.)

Advertisers and less benign sites also use LSO’s, and this has people worried. There are fewer restrictions on what LSO’s can do compared to cookies, and management of these little bits of information is not done through the browser. Many people out there in the Wild Wild World of the Web have said “There’s no way to manage them! aaaah! AAAAAAH!” in mildly hysterical voices, but that is simply not true.

So, just how worried should you be? If you do nothing to manage the cookies on your browser, then you probably don’t need to get too worked up about their somewhat-more-evil cousins. You’re already telling the trackers all they want to know. The potential for outright evil is higher with Flash LSO’s, but not that much.

Still, it’s a good idea to control who leaves what on your computer, and LSO’s are a good place to start. There are two complimentary strategies – control what gets saved, and clean up after.

Control What Gets Saved

First, let’s look at how to keep most of the unwanted items from being saved in the first place. This is done by managing the settings of your Flash Player. You do this through a control panel on the Macromedia Web site. You can access this panel any time by right-clicking any Flash on your page (including the banner of this site) and choosing “Global Settings…”. This control panel is written in Flash and when you make changes it will save your settings to your computer – in an LSO file.

Let’s look at what’s already on your computer. Choose the Web Storage Settings panel:

Flash Web Storage Settings Panel

The Web Storage Settings Panel

I cleaned everything out recently, but you can see that since then I’ve been to two places that put Flash cookies on my machine. Only muddledramblings.com is actually storing anything; www.kfox.com had stored something on my machine, but it has since been cleaned up. Even after you clean up a site’s cookies, Flash will remember you were there, and if you set special rules for that site, it will remember them, too.

So at this point the easiest thing to do is probably to make Flash forget everything and clear out all the cookies stored on your machine. If you’re curious you can go down the list and see what’s there. “Delete all sites” is probably your best bet, however.

Now your Flash Player has totally forgotten where you’ve been. It’s a good time to set rules for how Flash should behave when you encounter a new site. Click the “Global Storage” tab:

Flash Global Storage Settings Panel

The Global Storage Settings Panel

Note: these settings will not affect sites that already have storage allocated.

There are two things you can do to limit who puts stuff on your computer. The first is to move the slider to 0 KB. This will force any flash animation to ask permission before storing something on your machine. If you check Never Ask Again, you have effectively turned off all Flash cookies from everywhere. That’s pretty drastic, and may break some of your favorite sites, though.

The second thing you can do is uncheck “Allow third-party blah blah blah”. That allows the Web site you’re visiting to store stuff, but no one else. For instance, let’s say that on this page I had advertising. This setting would allow only Flash from muddledramblings.com to save stuff, but an ad served from eviladvertisinggiant.com would not be allowed. Basically, only Flash that comes from the domain showing in your browser is allowed to store stuff. That way my site will still work correctly but others won’t be able to track you.

Note that in a few cases, Web sites put their own Flash stuff on different servers (there are good reasons for doing this), and this setting might break those sites. You can turn off the restriction temporarily and allow that site to run, then turn the restriction back on. There is no way that I know of to say set the “Allow third-party…” value for a particular site.

OK, now that you’re keeping most of the drek off your machine, it’s time to tackle the other prong in our battle for privacy: cleaning up unneeded LSO’s.

Periodic Cleanup

In general, benign LSO’s only need to save stuff while you’re on the site. When you go back, it’s not going to harm anything if previous data has been deleted. Any information they do store from session to session might just be snooping. For the most part, then, we can just empty out the stored data and never notice a thing.

What NOT to delete
There are two kinds of LSO’s – those set by the flash animation, and those set by the Flash Player to store information about a site. Deleting the second type can actually undermine your security if you’ve made special restrictions for specific sites using the control panel above. Also, for some few sites (pandora.com, in my case), you want Flash to remember your info between visits. As you decide on a cleanup strategy, keep that stuff in mind. There is one LSO used to store the settings you made from the control panel above, and I strongly recommend that you NOT delete it. Both the cleanup methods I mention below preserve that file by default.

Having said all that, don’t let the decisions stop you from moving forward. In the following techniques just using the defaults will work just fine for almost everyone.

BetterPrivacy
NOTE June 1, 2019: BetterPrivacy doesn’t exist anymore, because Firefox absorbed this functionality. You can learn more about Firefox cookie management here.

This is by far the best solution — If you use Firefox. Users of Firefox have access to BetterPrivacy, which provides lots of options for which LSO’s to clean up when, controlled from a pretty nice user interface. By default BetterPrivacy leaves the Flash Player preferences alone, but if you make different settings for specific Web sites, BetterPrivacy will delete those unless you tell it not to. If you do put special restrictions or grant special permissions to a site, be sure to protect the settings.sol file for that site.

If you keep your browser open pretty much all the time, you can set BetterPrivacy to clean up the LSO’s periodically.

I don’t use Firefox that much. What am I to do? A Web search will tell you there’s a Mac application called Flush, but don’t bother. As of this writing, it’s completely broken. I didn’t find a good solution for non-Firefox Mac users out there, so I made one. You don’t have to thank me, it’s what I do.

Jer’s LSO Cleanup script for Mac
First I set up a simple cron task that just deleted the folders where LSO’s live. That was too ham-fisted, however, since it also deleted beneficial LSO’s, as mentioned above. Then I wrote a little script. I used Python to write it simply because I’d never used Python before, and it seemed more appropriate than php. You are welcome to use the script as well, but there’s a little fiddling involved. Nothing major, but you’ll be using the terminal.

Right-click to download lsoclean.sh here.

OK, now for the fiddling:

  1. Download the file and put it somewhere that won’t clutter up your life. (I used /usr/local/bin/)
  2. (optional) Edit the file to choose your paranoia level and what sites you don’t want to clean up. The default leaves LSO’s from your own computer (for Flash developers) and from pandora.com
  3. Tell the OS that the file is actually a script that it can run. To do this open Terminal.app and type chmod +x /path/to/lsoclean.py. TIP: if you just type chmod +x (with a space after the “x”), then drag the file from wherever you put it into the terminal window, it will automatically fill in the path. Neat!
  4. (optional) The script can now be activated by typing the full path to the file in the terminal, but that’s not very convenient. Better to set up a way to have the thing run every so often. There are plenty of ways, like using AppleScript (ptui) or iCal (which would have been clever of me), but the simplest (if geekiest) is to set up a cron task. You can use CronniX to avoid editing the crontab file directly. Here’s what I did:
    1. Download CronniX here.
    2. Run it. It’s a little… incomplete. Start by clicking “New”
    3. Choose the “Simple” tab
    4. Check the boxes next to Month, Day of Month, Hour, and Day of Week. Leave Minute unchecked and set to 0.
    5. In the command field, put the full path to the script file. (You can copy it out of the terminal window where you dragged the file before.)
    6. Click Apply, then Save
    7. Quit CronniX

    You have now set up the task to execute the script once an hour.

  5. Test: Visit some sites that use flash, and look in ~/Library/Preferences/Macromedia/Flash Player/#SharedObjects/8JA5UY2L (the last bit is random) to see the .sol files.
  6. After an hour, go back and see that they are gone! Hooray!

CronniX

The CronniX UI when everything is set to go.

That was a quite a bit of fiddling for those not versed in the ways of cron, but now you can forget it ever happened. If you find yourself having to reenter information in a Flash-based Web site and it annoys you, add the domain for that site to the list in the script.

If anyone wants to take this little script create a reduced-fiddling version with automator or whatever, I’d love to provide that for download here.

Conclusion

You’ve just struck a blow against invasive advertisers! Hooray! Now the ads you see will be less focussed on what you are interested in. That’s OK, because it nobody’s damn business what you’re interested in. Now you can carry on with your life like none of this ever happened.

3

Appreciating Fonts

The look of this blog when viewed on a Windows machine has always subtly annoyed me. I’ve been using the default font setup for WordPress, which uses Lucida Grande first, and if that is not available it uses Verdana. Verdana to me looks, I don’t know, thin or stretched or something. Loose. Unfortunately most Windows boxes don’t come with Lucida Grande, so Verdana is what most people experience. Today I decided to do something about it.

It’s possible now to tell a broswer to load a font from the Web when displaying a particular page. I could quite easily put @font-face directives in my files, load copies of Lucida Grande onto the server, and I’d be done (except for Internet Explorer, and those people can get by with Verdana). Unfortunately, although technically pretty simple, that course of action would not be legal.

There’s a font on Windows called Lucida Sans Unicode (or something like that) which is very similar to Lucida, but is not nearly as good for italics and bold face. This will be my fall-back solution.

For a while today, however, I thought I might go look for a new font, something that caught the spirit of this blog, yet was easy to read on a screen and had a nice ink density. On top of that, it had to be free or at least reasonably priced, and it had to include good italic and bold versions, and it had to include the wacky Czech diacriticals for those few episodes where I use them, plus the full range of punctuation including a variety of dashes, copyright symbols, and stuff like that.

I came up empty. Making a good font is not at all simple, and the people who make the great ones quite understandably want to be paid for their work. If I found one that measured up to Lucida Grande in usefulness and that would give this site a unique feel, I might be tempted to pony up.

The closest thing I could find was a font called Liberation, which is a favorite in the Linux world. At this writing, those without Lucida Grande will see that font (unless you’re using Internet Explorer). It’s OK, but the text is actually a little smaller for the same font size. That certainly is annoying. I haven’t looked at the text on enough different screens to know for sure, but I think right now the lettering is too small.

How’s it looking for you, my windows-using readers? Do you have any favorite fonts? I think with screen resolutions improving, it’s even possible to consider a serifed font these days.

Cascading Style Sheets (CSS) and PHP

Often when dealing with Cascading Style sheets, or CSS, I find myself wishing that the CSS mechanism included variables. This is especially true when dealing with colors, since you want the same color applied to lots of different things. It can be a real pain to go back through an old style sheet and find the code for the color you want. I was quietly surprised that no one making up how CSS worked had addressed something like this.

Then, a while back I was giving a buddy of mine a few exercises to introduce him to the exciting world of Web programming, touching on CSS, HTML, PHP and MySQL. I gave him pretty much no guidance; I just thought up plans that would introduce him to the concepts and gave him a list of my favorite references. (I’ll be posting those exercises here in the nearish future.)

Anyway, without me to tell him how to do things, he went and dug around and one of the first style sheets he sent me for evaluation had a .php extension rather than .css.

Bingo! Once you see it in action, it’s obvious. PHP can be used to generate CSS files just as easily as it can be used to generate HTML files. Now my style sheets can change based on external conditions or can simply define a set of colors that all the style definitions share. Why did it take me so long to figure this out? It seems like this technique should be a lot more common than it is.

Here’s a quick code snippet for those who want to try it for themselves:

<?php
	header('Content-Type: text/css');
 
	$header_back_color = '#dddddd';
?>
 
#corner_table th {
	background-color:<?php echo $header_back_color ?>;
	text-align: center;
}

A couple of notes: the <?php MUST be the very first thing in the file. No empty lines, no spaces. The reason is that the next line, with the header() function, has to be called before the server sends any page content. (Once the server starts sending content back to the browser, it’s too late to be fiddling with the headers. Any whitespace outside the <?php tag will be considered content.) The header line is necessary because you need to tell your browser that what you are sending really is a css file.

In the <head> of the html file, you call the style sheet just like normal, but of course the file you fetch will have a php extension:

<link rel="stylesheet"
      href="http://yourdomain.com/css-tables.php"
      type="text/css"
      media="screen" />

That’s all there is to it. Why have I not done this with every css file?

Drupal and WordPress

There is a lot of talk at Drupalcon about how they stack up against the competition. We are in the weeding-out phase of the Web Content Management System market, when most of the myriad contenders will fall by the wayside. Those who make a living building and using Drupal naturally want their platform to be among the survivors.

Drupal, according to their own assessment, powers about 1% of the Work Wide Web. The Drupalistas estimate that WordPress accounts for just north of 8%. There is another system called Joomla that is roughly even with Drupal. These three look to be the survivors in the Great-Web-site-in-a-box sweepstakes.

Honestly, I was a little surprised that Drupal considered WordPress to be a competitor. Sure, they both want to be used for more and more of the Web, but does Lego consider Tonka to be a competitor? Here’s the deal: Drupal says WordPress is the most popular Content Management System (CMS). I say WordPress is not a CMS at all.

That’s not to say WordPress isn’t a fine tool, in fact, this blog uses WordPress. But would I use WordPress for my current paying gig? No. Honestly I dread the day when WordPress becomes a big, fancy CMS like Drupal. That’s not what it’s for. There is a reason WordPress is the big dog, and it’s not because you can build sophisticated Web applications with it, it’s because you can install WordPress, find a nice skin, and get your stuff on the Web in an attractive and intuitive way. WordPress is a publishing platform, and a pretty good one at that.

Drupal, on the other hand, begins to shine at the next level up in terms of sophistication. It is the Lego to WordPress’ Tonka. There is considerably more design up front, and much critical functionality must be added as external modules that don’t come with the main (“core”) code. (Some of these things will be added to core in Drupal 7.) Maintenance of a Drupal site is more labor-intensive as well, as updating the parts is more complicated than with WordPress.

In exchange for the added complexity, you get a lot more flexibility. That’s not to say that WordPress can’t be used to make sophisticated Web sites, but generally speaking WordPress is optimized in putting a defined sort of information (like blog posts) on the screen in a very flexible way. There are hundreds of ways to add other pre-defined data types (for instance, there are shopping cart plugins), and all that works really well and most people are going to be happy with that.

Drupal is the tool you use when the data types in WordPress won’t do it for you. In Drupal you are building a Web application, where with WordPress you are using a Web application. Step one in building a site with Drupal is designing the data and the relationships between the various data types. Drupal allows you to design your data without having to design your database. Some of the ways Drupal implements your data design are pretty hokey, but it works. You can create pretty sophisticated data models without knowing a thing about how a database works – or even what kind of database you’re using. (In fact, you’re better off not knowing how the data is structured, because things can move unexpectedly as you tweak your design.) You are also presented with an almost dizzying set of options to decide who is allowed to see, edit, or create each little piece of each data type you define.

Once you get your content types defined, then you can move on to how to get actual content into the system (handled pretty much automatically), and how to present specific subsets of your data on the screen. To get at the data one often uses views, which are built using a tool that generates (frustratingly limited) database queries and then processes the results with a gratifying set of options tailored to each data type.

Then it comes time to put stuff on the screen. To control where things go and when, there are regions, blocks, panels, panes, pages, and so forth in a nonintuitive overlapping of roles. Blocks and regions and pages are built in, but the profusion of other options is a testament to the limited way they work together. For all the flexibility of Drupal, GUI building is still clumsy, though getting better (I’m told).

At last we come to the task of making the output pretty. For this purpose Drupal uses a maze of performance-sucking php template files that are invoked using a system of names that allows one to set up the display of information at just about any level of granularity. Many of these templates go hand-in-hand with specially-named preprocessor functions that allow you to customize how data is prepared for presentation.

Drupal separated the preparation of data and the presentation of data to allow people with different skill sets to do the different tasks. The template files can be done with only a minimal amount of php, while the preprocessors are where the real logic is implemented, unencumbered by HTML and CSS. This also has the effect of putting the risky code out of reach of those who aren’t expert in Web security. All good things.

I used the phrase “performance-sucking” above, and I meant it. The designers of Drupal made a conscious decision to emphasize good architecture and flexibility over fast execution. This was the same decision Google faced a few years back, as they developed ever-more-sophisticated pattern matching algorithms. While competitors kept things simple to reduce server load, the folks at Google decided that the cost of processing cycles and storage was tending toward free, and chose to emphasize the quality of the information they provided instead. Similarly, Drupal has decided to make things in a structurally sound way and spend the processor cycles and disk reads necessary to support that.

Drupal 7 will be even slower, but will be more scalable (they say). What that means is that although the software is not as fast as it could be, its behavior is predictable as demand increases, and it is easer to scale up your site as things go huge. Good structure pays greater and greater dividends as things get bigger.

All that stuff Drupal has makes it a more complicated to get up and running, and for a simple site (or even one of moderate complexity but with a relatively straightforward data model), WordPress is going to get you to the promised land with a lot less pain.

I am led to believe that the WordPress community feels it needs to compete with Drupal just as much as Drupal thinks they need to compete with WordPress. Toward this end WordPress 3.0 will have new features that answer some of Drupal’s flexibility advantages. All I can say is “PLEASE, WordPress, don’t try to be everything Drupal is.” That WordPress is not everything Drupal is constitutes its greatest advantage. Stay with your market, WordPress!

Drupalcon Day 1 – notes from the floor

I’m working on a project right now that is based on a Web development platform called Drupal. I have a long editorial episode building in my head concerning Drupal and competing platforms for Web development, but today I’m going to write this episode assuming you already know what Drupal is and how it works. (This is also how the documentation for Drupal is written.)

Once I arrived and registered, I looked over the program to decide which seminars I would attend. I looked down the list and realized I already knew what most of the seminars were discussing. Some of the seminars, I could have been the presenter. I realized that the cross-section of stuff I know about Drupal probably qualifies me as a Drupal expert now.

Still, there’s always new stuff to learn. I decided to dedicate my day to security. There are a lot of ways to break a Web site these days.

Before the security sessions there was the keynote address by the guy who invented Drupal in his dorm room in Antwerp ten years ago. He is still holder of the vision for the project, and hearing him speak I have to say that the project is in good hands. He knows there are challenges ahead, and he was an excellent cheerleader for open source, and for encouraging everyone who uses Drupal to give back to the community. Currently they are trying to release the next major upgrade, something they absolutely must have, and soon (more on that in a bit). “There are 114 critical bugs to fix,” he said (or something like this), “If we break into teams right now we can have them fixed by the end of the day. So, we’re locking the doors…”

There was a laugh, but his point was a good one. Rather than wait eagerly for the release, the Drupal community should be actively making it happen.

He also mentioned that 1% of the Web is now powered by Drupal. That’s pretty dang impressive (until you compare it to WordPress). It’s difficult to call his methods for estimating that 1% as scientific, but whatever that number is, I can tell you that it could be a lot higher except for one thing: This software induces more WTF? moments than any other development platform I’ve ever used. Novices who come to the platform install the software, stare blankly at the screen, click things, and give up and move on to a more intuitive product. It would be impossible to measure how many adoptions they have lost because of that initial Now What? moment, but it’s significant I promise you.

On a related note, employment opportunities for Drupal experts is on the rise. People who have worked their way through the WTF to where they can be productive with the platform are in demand. I can now navigate and decode the documentation (I think some of the writers of the documentation are so steeped in the Drupal Way that they don’t even realize they are writing in code), and that puts me in good position to find work. When the out-of-box experience is improved (a major thrust of Drupal 7), my “expert” status will be less lucrative.

Speaking of the Drupal Way: At the risk of being overly general, these guys are more sensitive than even Mac people when it comes to hearing criticism about their platform. Also, there were easily more Macs in evidence than Windows laptops. Perhaps that is because we are on Apple’s home turf here, but I think that Mac, with its handy Unix underpinnings, is finding a sweet spot in the Web design world, with cachet among the designers as well as unix (the os of the Web) for the übergeeks. (The only apple I brought to the convention, I ate. I have Ol’ Pokey charging up, however, to see if it’s game for one last field trip before its ten-year-old video system gives up entirely.)

Back to the keynote: Mr. Buytaert, while talking about the future of Drupal, mentioned that as they got bigger, there would be people for whom using Drupal would be a day job! They wouldn’t be using it just for the love of it, they would think of it as just another tool to get their job done. Mr. Buytaert, welcome to 2008. Those people are now your market, if you want to meet your stated goals for growth.

One thing I’ll say for the guy, he really seems driven by the simple desire to make Drupal the best. He’s probably wealthy now, but commercial success just doesn’t seem to be what motivates him. He wants to make his baby better and better and world domination is simply a way to measure how well he’s doing. It’s refreshing to hear from someone like that.

As for the security sessions, I think this best sums it up (this link was given in one of the seminars):

xkcd 327

A note of explanation for the less-geeky (which you can skip): When a programmer is careless, people can put a string in any field on a site and cause database commands to be executed. In the comic, the name “Robert’); DROP TABLE Students;–” will cause the database the table named ‘Students’, obliterating their records. The ‘); tells the database that the command to add the name is finished, then the rest of the text is treated as a new command. Aren’t you glad you asked?

I also learned just what a risk it is to link to an image the way I just did. The owner of that site can now attack my blog. Ah, the irony.

I did learn some useful stuff in the seminars, and just in time, too. I’m really glad I went.

1

New Features here at MR&HBI!

First, allow me to call your attention to the episode immediately before this one. You might notice the little icon is a camera. “huh,” you might be saying to yourself, “I don’t remember seeing that one before.” Very observant, Buckaroo! It’s for a new category, Photography, that I added. “But,” the even more observant amongst you might say, “There are already a handful of episodes in that category.” Right again, Wisenstein! I recategorized a couple episodes that were under The Great Adventure and found a couple in Idle Chit-Chat that were better filed under the new category. I expect there are plenty more; the trick is finding them.

The icon is actually my camera sitting on an opened unabridged dictionary. That may seem staged, but that’s actually where we keep the camera these days. Yes, we have an unabridged dictionary open on a stand at all times. No, that does not make us geeks.

Second, way down at the bottom of the sidebar, there’s a section called Other Muddled Stats (or something like that). That’s a wordpress widget I made that counts all the words in all the episodes, and keeps a tally of how many comments there have been as well. I plan to add other stats as well next time I have the hood open. Perhaps the number of times I’ve said “You don’t have to thank me,” or the number of times I’ve blamed the Chinese for things. (Hm… haven’t done that in a while…) Anything you’d like to know? The number of letters typed? Words in comments? Most prolific commenters? If it’s on these pages, I can count it.

The WordPress plugin itself is hand-crafted by yours truly. I started by downloading a different word-counting plugin, but it counted the words on every page load and didn’t have a sidebar widget. All it was was a database query and a loop. My version only counts when the relevant value changes – it only counts words when a new episode is posted, for instance. Once I tidy it up I’ll be adding it to the WordPress repository, so others can also gather useless stats about their blogs. It’s all about sharing the love.