Archive for ‘Rumblings from the Secret Labs’

« Older Episodes
2
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

Evil Flash Cookies

August 30th, 2010
Otherwise known as LSO's, these things allow Flash to watch you - behind your back. Just how worried should you be?

For a long time now we’ve been aware of browser cookies. These are little bits of data that a Web developer can set on your computer to keep track of your visits, or which ads you’ve seen, and things like that. Cookies are regulated by your browser and you can set up rules to reduce the amount that other people learn about your habits.

Way back in the day the makers of Flash realized that it would be handy to store little bits of information on the user’s computer as well. They developed LSO’s, otherwise known as Flash Cookies, to do that. This site uses an LSO so the banner animation doesn’t run every time you change pages. (See my rant about html.)

Advertisers and less benign sites also use LSO’s, and this has people worried. There are fewer restrictions on what LSO’s can do compared to cookies, and management of these little bits of information is not done through the browser. Many people out there in the Wild Wild World of the Web have said “There’s no way to manage them! aaaah! AAAAAAH!” in mildly hysterical voices, but that is simply not true.

So, just how worried should you be? If you do nothing to manage the cookies on your browser, then you probably don’t need to get too worked up about their somewhat-more-evil cousins. You’re already telling the trackers all they want to know. The potential for outright evil is higher with Flash LSO’s, but not that much.

Still, it’s a good idea to control who leaves what on your computer, and LSO’s are a good place to start. There are two complimentary strategies – control what gets saved, and clean up after.

Control What Gets Saved

First, let’s look at how to keep most of the unwanted items from being saved in the first place. This is done by managing the settings of your Flash Player. You do this through a control panel on the Macromedia Web site. You can access this panel any time by right-clicking any Flash on your page (including the banner of this site) and choosing “Global Settings…”. This control panel is written in Flash and when you make changes it will save your settings to your computer – in an LSO file.

Let’s look at what’s already on your computer. Choose the Web Storage Settings panel:

Flash Web Storage Settings Panel

The Web Storage Settings Panel

I cleaned everything out recently, but you can see that since then I’ve been to two places that put Flash cookies on my machine. Only muddledramblings.com is actually storing anything; www.kfox.com had stored something on my machine, but it has since been cleaned up. Even after you clean up a site’s cookies, Flash will remember you were there, and if you set special rules for that site, it will remember them, too.

So at this point the easiest thing to do is probably to make Flash forget everything and clear out all the cookies stored on your machine. If you’re curious you can go down the list and see what’s there. “Delete all sites” is probably your best bet, however.

Now your Flash Player has totally forgotten where you’ve been. It’s a good time to set rules for how Flash should behave when you encounter a new site. Click the “Global Storage” tab:

Flash Global Storage Settings Panel

The Global Storage Settings Panel

Note: these settings will not affect sites that already have storage allocated.

There are two things you can do to limit who puts stuff on your computer. The first is to move the slider to 0 KB. This will force any flash animation to ask permission before storing something on your machine. If you check Never Ask Again, you have effectively turned off all Flash cookies from everywhere. That’s pretty drastic, and may break some of your favorite sites, though.

The second thing you can do is uncheck “Allow third-party blah blah blah”. That allows the Web site you’re visiting to store stuff, but no one else. For instance, let’s say that on this page I had advertising. This setting would allow only Flash from muddledramblings.com to save stuff, but an ad served from eviladvertisinggiant.com would not be allowed. Basically, only Flash that comes from the domain showing in your browser is allowed to store stuff. That way my site will still work correctly but others won’t be able to track you.

Note that in a few cases, Web sites put their own Flash stuff on different servers (there are good reasons for doing this), and this setting might break those sites. You can turn off the restriction temporarily and allow that site to run, then turn the restriction back on. There is no way that I know of to say set the “Allow third-party…” value for a particular site.

OK, now that you’re keeping most of the drek off your machine, it’s time to tackle the other prong in our battle for privacy: cleaning up unneeded LSO’s.

Periodic Cleanup

In general, benign LSO’s only need to save stuff while you’re on the site. When you go back, it’s not going to harm anything if previous data has been deleted. Any information they do store from session to session might just be snooping. For the most part, then, we can just empty out the stored data and never notice a thing.

What NOT to delete
There are two kinds of LSO’s – those set by the flash animation, and those set by the Flash Player to store information about a site. Deleting the second type can actually undermine your security if you’ve made special restrictions for specific sites using the control panel above. Also, for some few sites (pandora.com, in my case), you want Flash to remember your info between visits. As you decide on a cleanup strategy, keep that stuff in mind. There is one LSO used to store the settings you made from the control panel above, and I strongly recommend that you NOT delete it. Both the cleanup methods I mention below preserve that file by default.

Having said all that, don’t let the decisions stop you from moving forward. In the following techniques just using the defaults will work just fine for almost everyone.

BetterPrivacy
This is by far the best solution — If you use Firefox. Users of Firefox have access to BetterPrivacy, which provides lots of options for which LSO’s to clean up when, controlled from a pretty nice user interface. By default BetterPrivacy leaves the Flash Player preferences alone, but if you make different settings for specific Web sites, BetterPrivacy will delete those unless you tell it not to. If you do put special restrictions or grant special permissions to a site, be sure to protect the settings.sol file for that site.

If you keep your browser open pretty much all the time, you can set BetterPrivacy to clean up the LSO’s periodically.

I don’t use Firefox that much. What am I to do? A Web search will tell you there’s a Mac application called Flush, but don’t bother. As of this writing, it’s completely broken. I didn’t find a good solution for non-Firefox Mac users out there, so I made one. You don’t have to thank me, it’s what I do.

Jer’s LSO Cleanup script for Mac
First I set up a simple cron task that just deleted the folders where LSO’s live. That was too ham-fisted, however, since it also deleted beneficial LSO’s, as mentioned above. Then I wrote a little script. I used Python to write it simply because I’d never used Python before, and it seemed more appropriate than php. You are welcome to use the script as well, but there’s a little fiddling involved. Nothing major, but you’ll be using the terminal.

Right-clidk to download lsoclean.py here.

OK, now for the fiddling:

  1. Download the file and put it somewhere that won’t clutter up your life. (I used /usr/local/bin/)
  2. (optional) Edit the file to choose your paranoia level and what sites you don’t want to clean up. The default leaves LSO’s from your own computer (for Flash developers) and from pandora.com
  3. Tell the OS that the file is actually a script that it can run. To do this open Terminal.app and type chmod +x /path/to/lsoclean.py. TIP: if you just type chmod +x (with a space after the “x”), then drag the file from wherever you put it into the terminal window, it will automatically fill in the path. Neat!
  4. (optional) The script can now be activated by typing the full path to the file in the terminal, but that’s not very convenient. Better to set up a way to have the thing run every so often. There are plenty of ways, like using AppleScript (ptui) or iCal (which would have been clever of me), but the simplest (if geekiest) is to set up a cron task. You can use CronniX to avoid editing the crontab file directly. Here’s what I did:
    1. Download CronniX here.
    2. Run it. It’s a little… incomplete. Start by clicking “New”
    3. Choose the “Simple” tab
    4. Check the boxes next to Month, Day of Month, Hour, and Day of Week. Leave Minute unchecked and set to 0.
    5. In the command field, put the full path to the script file. (You can copy it out of the terminal window where you dragged the file before.)
    6. Click Apply, then Save
    7. Quit CronniX

    You have now set up the task to execute the script once an hour.

  5. Test: Visit some sites that use flash, and look in ~/Library/Preferences/Macromedia/Flash Player/#SharedObjects/8JA5UY2L (the last bit is random) to see the .sol files.
  6. After an hour, go back and see that they are gone! Hooray!

CronniX

The CronniX UI when everything is set to go.

That was a quite a bit of fiddling for those not versed in the ways of cron, but now you can forget it ever happened. If you find yourself having to reenter information in a Flash-based Web site and it annoys you, add the domain for that site to the list in the script.

If anyone wants to take this little script create a reduced-fiddling version with automator or whatever, I’d love to provide that for download here.

Conclusion

You’ve just struck a blow against invasive advertisers! Hooray! Now the ads you see will be less focussed on what you are interested in. That’s OK, because it nobody’s damn business what you’re interested in. Now you can carry on with your life like none of this ever happened.

Tags: , ,
Posted in Rumblings from the Secret Labs | 9 Comments »

0
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

Appreciating Fonts

August 27th, 2010
Good ones are rare.

The look of this blog when viewed on a Windows machine has always subtly annoyed me. I’ve been using the default font setup for WordPress, which uses Lucida Grande first, and if that is not available it uses Verdana. Verdana to me looks, I don’t know, thin or stretched or something. Loose. Unfortunately most Windows boxes don’t come with Lucida Grande, so Verdana is what most people experience. Today I decided to do something about it.

It’s possible now to tell a broswer to load a font from the Web when displaying a particular page. I could quite easily put @font-face directives in my files, load copies of Lucida Grande onto the server, and I’d be done (except for Internet Explorer, and those people can get by with Verdana). Unfortunately, although technically pretty simple, that course of action would not be legal.

There’s a font on Windows called Lucida Sans Unicode (or something like that) which is very similar to Lucida, but is not nearly as good for italics and bold face. This will be my fall-back solution.

For a while today, however, I thought I might go look for a new font, something that caught the spirit of this blog, yet was easy to read on a screen and had a nice ink density. On top of that, it had to be free or at least reasonably priced, and it had to include good italic and bold versions, and it had to include the wacky Czech diacriticals for those few episodes where I use them, plus the full range of punctuation including a variety of dashes, copyright symbols, and stuff like that.

I came up empty. Making a good font is not at all simple, and the people who make the great ones quite understandably want to be paid for their work. If I found one that measured up to Lucida Grande in usefulness and that would give this site a unique feel, I might be tempted to pony up.

The closest thing I could find was a font called Liberation, which is a favorite in the Linux world. At this writing, those without Lucida Grande will see that font (unless you’re using Internet Explorer). It’s OK, but the text is actually a little smaller for the same font size. That certainly is annoying. I haven’t looked at the text on enough different screens to know for sure, but I think right now the lettering is too small.

How’s it looking for you, my windows-using readers? Do you have any favorite fonts? I think with screen resolutions improving, it’s even possible to consider a serifed font these days.

Tags: , , ,
Posted in Rumblings from the Secret Labs | 9 Comments »

0
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

Cascading Style Sheets (CSS) and PHP

May 30th, 2010
Obvious in retrospect.

Often when dealing with Cascading Style sheets, or CSS, I find myself wishing that the CSS mechanism included variables. This is especially true when dealing with colors, since you want the same color applied to lots of different things. It can be a real pain to go back through an old style sheet and find the code for the color you want. I was quietly surprised that no one making up how CSS worked had addressed something like this.

Then, a while back I was giving a buddy of mine a few exercises to introduce him to the exciting world of Web programming, touching on CSS, HTML, PHP and MySQL. I gave him pretty much no guidance; I just thought up plans that would introduce him to the concepts and gave him a list of my favorite references. (I’ll be posting those exercises here in the nearish future.)

Anyway, without me to tell him how to do things, he went and dug around and one of the first style sheets he sent me for evaluation had a .php extension rather than .css.

Bingo! Once you see it in action, it’s obvious. PHP can be used to generate CSS files just as easily as it can be used to generate HTML files. Now my style sheets can change based on external conditions or can simply define a set of colors that all the style definitions share. Why did it take me so long to figure this out? It seems like this technique should be a lot more common than it is.

Here’s a quick code snippet for those who want to try it for themselves:

<?php
	header('Content-Type: text/css');
 
	$header_back_color = '#dddddd';
?>
 
#corner_table th {
	background-color:<?php echo $header_back_color ?>;
	text-align: center;
}

A couple of notes: the <?php MUST be the very first thing in the file. No empty lines, no spaces. The reason is that the next line, with the header() function, has to be called before the server sends any page content. (Once the server starts sending content back to the browser, it’s too late to be fiddling with the headers. Any whitespace outside the <?php tag will be considered content.) The header line is necessary because you need to tell your browser that what you are sending really is a css file.

In the <head> of the html file, you call the style sheet just like normal, but of course the file you fetch will have a php extension:

<link rel="stylesheet"
      href="http://yourdomain.com/css-tables.php"
      type="text/css"
      media="screen" />

That’s all there is to it. Why have I not done this with every css file?

Tags: , , ,
Posted in Rumblings from the Secret Labs | No Comments »

0
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

Drupal and WordPress

April 21st, 2010

There is a lot of talk at Drupalcon about how they stack up against the competition. We are in the weeding-out phase of the Web Content Management System market, when most of the myriad contenders will fall by the wayside. Those who make a living building and using Drupal naturally want their platform to be among the survivors.

Drupal, according to their own assessment, powers about 1% of the Work Wide Web. The Drupalistas estimate that WordPress accounts for just north of 8%. There is another system called Joomla that is roughly even with Drupal. These three look to be the survivors in the Great-Web-site-in-a-box sweepstakes.

Honestly, I was a little surprised that Drupal considered WordPress to be a competitor. Sure, they both want to be used for more and more of the Web, but does Lego consider Tonka to be a competitor? Here’s the deal: Drupal says WordPress is the most popular Content Management System (CMS). I say WordPress is not a CMS at all.

That’s not to say WordPress isn’t a fine tool, in fact, this blog uses WordPress. But would I use WordPress for my current paying gig? No. Honestly I dread the day when WordPress becomes a big, fancy CMS like Drupal. That’s not what it’s for. There is a reason WordPress is the big dog, and it’s not because you can build sophisticated Web applications with it, it’s because you can install WordPress, find a nice skin, and get your stuff on the Web in an attractive and intuitive way. WordPress is a publishing platform, and a pretty good one at that.

Drupal, on the other hand, begins to shine at the next level up in terms of sophistication. It is the Lego to WordPress’ Tonka. There is considerably more design up front, and much critical functionality must be added as external modules that don’t come with the main (“core”) code. (Some of these things will be added to core in Drupal 7.) Maintenance of a Drupal site is more labor-intensive as well, as updating the parts is more complicated than with WordPress.

In exchange for the added complexity, you get a lot more flexibility. That’s not to say that WordPress can’t be used to make sophisticated Web sites, but generally speaking WordPress is optimized in putting a defined sort of information (like blog posts) on the screen in a very flexible way. There are hundreds of ways to add other pre-defined data types (for instance, there are shopping cart plugins), and all that works really well and most people are going to be happy with that.

Drupal is the tool you use when the data types in WordPress won’t do it for you. In Drupal you are building a Web application, where with WordPress you are using a Web application. Step one in building a site with Drupal is designing the data and the relationships between the various data types. Drupal allows you to design your data without having to design your database. Some of the ways Drupal implements your data design are pretty hokey, but it works. You can create pretty sophisticated data models without knowing a thing about how a database works – or even what kind of database you’re using. (In fact, you’re better off not knowing how the data is structured, because things can move unexpectedly as you tweak your design.) You are also presented with an almost dizzying set of options to decide who is allowed to see, edit, or create each little piece of each data type you define.

Once you get your content types defined, then you can move on to how to get actual content into the system (handled pretty much automatically), and how to present specific subsets of your data on the screen. To get at the data one often uses views, which are built using a tool that generates (frustratingly limited) database queries and then processes the results with a gratifying set of options tailored to each data type.

Then it comes time to put stuff on the screen. To control where things go and when, there are regions, blocks, panels, panes, pages, and so forth in a nonintuitive overlapping of roles. Blocks and regions and pages are built in, but the profusion of other options is a testament to the limited way they work together. For all the flexibility of Drupal, GUI building is still clumsy, though getting better (I’m told).

At last we come to the task of making the output pretty. For this purpose Drupal uses a maze of performance-sucking php template files that are invoked using a system of names that allows one to set up the display of information at just about any level of granularity. Many of these templates go hand-in-hand with specially-named preprocessor functions that allow you to customize how data is prepared for presentation.

Drupal separated the preparation of data and the presentation of data to allow people with different skill sets to do the different tasks. The template files can be done with only a minimal amount of php, while the preprocessors are where the real logic is implemented, unencumbered by HTML and CSS. This also has the effect of putting the risky code out of reach of those who aren’t expert in Web security. All good things.

I used the phrase “performance-sucking” above, and I meant it. The designers of Drupal made a conscious decision to emphasize good architecture and flexibility over fast execution. This was the same decision Google faced a few years back, as they developed ever-more-sophisticated pattern matching algorithms. While competitors kept things simple to reduce server load, the folks at Google decided that the cost of processing cycles and storage was tending toward free, and chose to emphasize the quality of the information they provided instead. Similarly, Drupal has decided to make things in a structurally sound way and spend the processor cycles and disk reads necessary to support that.

Drupal 7 will be even slower, but will be more scalable (they say). What that means is that although the software is not as fast as it could be, its behavior is predictable as demand increases, and it is easer to scale up your site as things go huge. Good structure pays greater and greater dividends as things get bigger.

All that stuff Drupal has makes it a more complicated to get up and running, and for a simple site (or even one of moderate complexity but with a relatively straightforward data model), WordPress is going to get you to the promised land with a lot less pain.

I am led to believe that the WordPress community feels it needs to compete with Drupal just as much as Drupal thinks they need to compete with WordPress. Toward this end WordPress 3.0 will have new features that answer some of Drupal’s flexibility advantages. All I can say is “PLEASE, WordPress, don’t try to be everything Drupal is.” That WordPress is not everything Drupal is constitutes its greatest advantage. Stay with your market, WordPress!

Tags: , ,
Posted in Rumblings from the Secret Labs | 5 Comments »

1
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

Drupalcon Day 1 – notes from the floor

April 20th, 2010

I’m working on a project right now that is based on a Web development platform called Drupal. I have a long editorial episode building in my head concerning Drupal and competing platforms for Web development, but today I’m going to write this episode assuming you already know what Drupal is and how it works. (This is also how the documentation for Drupal is written.)

Once I arrived and registered, I looked over the program to decide which seminars I would attend. I looked down the list and realized I already knew what most of the seminars were discussing. Some of the seminars, I could have been the presenter. I realized that the cross-section of stuff I know about Drupal probably qualifies me as a Drupal expert now.

Still, there’s always new stuff to learn. I decided to dedicate my day to security. There are a lot of ways to break a Web site these days.

Before the security sessions there was the keynote address by the guy who invented Drupal in his dorm room in Antwerp ten years ago. He is still holder of the vision for the project, and hearing him speak I have to say that the project is in good hands. He knows there are challenges ahead, and he was an excellent cheerleader for open source, and for encouraging everyone who uses Drupal to give back to the community. Currently they are trying to release the next major upgrade, something they absolutely must have, and soon (more on that in a bit). “There are 114 critical bugs to fix,” he said (or something like this), “If we break into teams right now we can have them fixed by the end of the day. So, we’re locking the doors…”

There was a laugh, but his point was a good one. Rather than wait eagerly for the release, the Drupal community should be actively making it happen.

He also mentioned that 1% of the Web is now powered by Drupal. That’s pretty dang impressive (until you compare it to WordPress). It’s difficult to call his methods for estimating that 1% as scientific, but whatever that number is, I can tell you that it could be a lot higher except for one thing: This software induces more WTF? moments than any other development platform I’ve ever used. Novices who come to the platform install the software, stare blankly at the screen, click things, and give up and move on to a more intuitive product. It would be impossible to measure how many adoptions they have lost because of that initial Now What? moment, but it’s significant I promise you.

On a related note, employment opportunities for Drupal experts is on the rise. People who have worked their way through the WTF to where they can be productive with the platform are in demand. I can now navigate and decode the documentation (I think some of the writers of the documentation are so steeped in the Drupal Way that they don’t even realize they are writing in code), and that puts me in good position to find work. When the out-of-box experience is improved (a major thrust of Drupal 7), my “expert” status will be less lucrative.

Speaking of the Drupal Way: At the risk of being overly general, these guys are more sensitive than even Mac people when it comes to hearing criticism about their platform. Also, there were easily more Macs in evidence than Windows laptops. Perhaps that is because we are on Apple’s home turf here, but I think that Mac, with its handy Unix underpinnings, is finding a sweet spot in the Web design world, with cachet among the designers as well as unix (the os of the Web) for the übergeeks. (The only apple I brought to the convention, I ate. I have Ol’ Pokey charging up, however, to see if it’s game for one last field trip before its ten-year-old video system gives up entirely.)

Back to the keynote: Mr. Buytaert, while talking about the future of Drupal, mentioned that as they got bigger, there would be people for whom using Drupal would be a day job! They wouldn’t be using it just for the love of it, they would think of it as just another tool to get their job done. Mr. Buytaert, welcome to 2008. Those people are now your market, if you want to meet your stated goals for growth.

One thing I’ll say for the guy, he really seems driven by the simple desire to make Drupal the best. He’s probably wealthy now, but commercial success just doesn’t seem to be what motivates him. He wants to make his baby better and better and world domination is simply a way to measure how well he’s doing. It’s refreshing to hear from someone like that.

As for the security sessions, I think this best sums it up (this link was given in one of the seminars):

xkcd 327

A note of explanation for the less-geeky (which you can skip): When a programmer is careless, people can put a string in any field on a site and cause database commands to be executed. In the comic, the name “Robert’); DROP TABLE Students;–” will cause the database the table named ‘Students’, obliterating their records. The ‘); tells the database that the command to add the name is finished, then the rest of the text is treated as a new command. Aren’t you glad you asked?

I also learned just what a risk it is to link to an image the way I just did. The owner of that site can now attack my blog. Ah, the irony.

I did learn some useful stuff in the seminars, and just in time, too. I’m really glad I went.

Tags: ,
Posted in Rumblings from the Secret Labs | 4 Comments »

0
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

New Features here at MR&HBI!

March 30th, 2010
Not only that, but you help decide how one of them works!

First, allow me to call your attention to the episode immediately before this one. You might notice the little icon is a camera. “huh,” you might be saying to yourself, “I don’t remember seeing that one before.” Very observant, Buckaroo! It’s for a new category, Photography, that I added. “But,” the even more observant amongst you might say, “There are already a handful of episodes in that category.” Right again, Wisenstein! I recategorized a couple episodes that were under The Great Adventure and found a couple in Idle Chit-Chat that were better filed under the new category. I expect there are plenty more; the trick is finding them.

The icon is actually my camera sitting on an opened unabridged dictionary. That may seem staged, but that’s actually where we keep the camera these days. Yes, we have an unabridged dictionary open on a stand at all times. No, that does not make us geeks.

Second, way down at the bottom of the sidebar, there’s a section called Other Muddled Stats (or something like that). That’s a wordpress widget I made that counts all the words in all the episodes, and keeps a tally of how many comments there have been as well. I plan to add other stats as well next time I have the hood open. Perhaps the number of times I’ve said “You don’t have to thank me,” or the number of times I’ve blamed the Chinese for things. (Hm… haven’t done that in a while…) Anything you’d like to know? The number of letters typed? Words in comments? Most prolific commenters? If it’s on these pages, I can count it.

The WordPress plugin itself is hand-crafted by yours truly. I started by downloading a different word-counting plugin, but it counted the words on every page load and didn’t have a sidebar widget. All it was was a database query and a loop. My version only counts when the relevant value changes – it only counts words when a new episode is posted, for instance. Once I tidy it up I’ll be adding it to the WordPress repository, so others can also gather useless stats about their blogs. It’s all about sharing the love.

Tags: , , , ,
Posted in Rumblings from the Secret Labs | 9 Comments »

0
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

A little help?

March 21st, 2010

I’m getting the sweet-o-meter reinstalled. The old one broke and there was confusion about versions and so forth and in the end I decided to go with another plugin that seems to be more actively maintained. This one is fancier, but the creator really didn’t imagine all the different ways people might want to customize something like this. As a result I’ve been tinkering under the hood.

To get things right I need to see what it looks like when someone besides me has voted for an episode. Can someone out there click the “sweet” button at the top of this episode? Thanks!

Tags:
Posted in Rumblings from the Secret Labs | 14 Comments »

1
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

Lost in Translation?

March 5th, 2010
The mere existence of the message implies a history, as well.

Even if you’re not a programmer, take a look at the following lines of code:

public function sendCommunication($oCommunication)
{
    if (self::emailMode != EMAIL_TEST_MODE_NONE) {
        if (self::emailMode == EMAIL_TEST_MODE_LOGGED_IN_ONLY) {
            // DO NOT COMMENT OUT THE FOLLOWING LINES
            // EVER
            // FOR ANY REASON
            // INSTEAD CHECK THE TEST MODE AND SET THE ADDRESS FIELDS ACCORDINGLY
            $oCommunication->to = $oCommunication->from;
            $oCommunication->cc = '';
        }

Now, I ask you, even if you’re not a programmer, you know there’s one thing you would never, ever, do to the above code. Right? Now let’s say you are a programmer, a professional, being paid because of your ability to find solutions to problems and express them in an abstract language.

Now further imagine that changing the above code can lead to the customers of the people paying for this work getting spammed with confusing emails with our client’s name on them.

Yeah, you guessed it.

Tags: , , ,
Posted in Rumblings from the Secret Labs | 6 Comments »

0
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

Jer’s Software Hut Falls Silent

February 28th, 2010
Shutting the doors (at least for now).

The shadowy, misshapen minions have all gone home; the vast underground chamber that once rang with their chants as they turned the giant wooden capstans has fallen silent. The river of lava flows unimpeded, the precarious rope bridges spanning it falling into disrepair. Above, the streets of Sky City Research Facility, once teeming with antigravity cars, are empty, the crystalline architecture acquiring a layer of eagle guano and dust that is transformed into gritty runoff when it rains.

The crudely-crafted Web site at jerssoftwarehut.com no longer accepts payment for Jer’s Novel Writer software licenses, and bears the following statement:

Well, it’s happened; I have a regular job. As I slave away working for the man I often wonder if things might have been different had I only worked harder at making Jer’s Software Hut a business rather than a hobby. Probably now we will never know. It was a good run but it’s time to ackowledge that development is stalled and customer service around here has been really awful.

That pretty much says it all; despite thousands of happy users, some of whom even paid for the software, when it came time to have a steady income again I took the safer path of working for someone else. (The ironic twist to this narrative I will leave for another time.)

It was a good run, and as I get my work life under control I hope soon to at least return to using Jer’s Novel Writer for its intended purpose – as a writing tool that helps me create fiction. Until I do that I can’t even consider opening the shutters on the Hut and throwing the big switch that raises the lightning rod into the violent midnight thunderstorm, while sparks fly and the turbines spin faster and faster, the needles on their gauges creeping ominously into the red. Maybe someday, though. Maybe someday.

Tags: ,
Posted in Rumblings from the Secret Labs | 3 Comments »

0
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

Haloscan comments to WordPress – the nitty gritty.

February 5th, 2010
... or, what I did last night.

As I mentioned in the previous episode, I recently had to move more than 8000 comments from my old comment system, Haloscan, and import them into WordPress. Haloscan served me well back in the day, but they are going away, and all my more recent comments are in the WordPress system anyway. Nice to have them all in one place.

The process turned out to be pretty easy. I found a script for importing comments from a different system, modified it, modified it some more, found a fundamental problem with it, fixed that, and in the end not much of code remained from the example, except the part where the WordPress logo is displayed on the screen. I assume that part came from the code the guy copied to make the code that I copied.

Along the way I learned a couple of things. PHP is a pretty flexible language, but running a loop that sets up 8500 data structures and runs 25500 database queries exposes PHP’s primary weakness: memory management. The whiz kids who invented PHP designed it for a load/compile/execute/exit-and-clean-up flow. Memory allocated during execution is cleaned up when the program is done running (usually when the Web page is delivered). When you try to do heavy lifting with PHP, you have to start paying attention to getting your memory back before the traditional clean-up time.

The code I started with did a direct database query to add the comment to the comments table, but that got things out of sync with other tables. (The posts table keeps track of the number of comments that apply to it, presumably for performance reasons.) I dug into the core WordPress code and found the method they call to post comments, and I made my code call that function. I have no idea what all the bookkeeping chores are that function does, and really I don’t care as long as they get done.

I didn’t worry about performance too much at first (after all, it only has to run once), but one of the database queries I did was really expensive (scanning all the posts for a specific set of characters). Even running on my local server it was slow, and I knew that if I tried something like that on my actual Web host alarms would go off and they’d shut me down for a while. I did a little optimization on that front, and it was enough.

The following script has some Muddle-specific code in it, but it might come in handy for others who need to move Haloscan comments to a new system. The part that parses Haloscan XML is pretty generic and would work for anyone, the part that saves the comments might be useful as a guide as well. The main difference others will have to deal with is where to get proper post_id based on the thread field in the XML. In my case I had a link in each blog episode back to the Haloscan thread.

The HTML bit in the middle of the file is not essential; but it puts a nice WordPress logo on the screen when the script starts up. I inherited that from the script I started with.

NOTE: While this script has code in it specific to me, I am available to customize it for others who need to move their code from Haloscan into another environment, or, for that matter, from any structured source into WordPress. Drop me a line!

<?php
 
if (!file_exists('../wp-config.php')) die("There doesn't seem to be a wp-config.php file. You must install WordPress before you import any comments.");
require('../wp-config.php');
 
function saveCommentToWP($comment, $dbRef, &$postThreads) {
    //echo "here's where the comment save happens <br/><br />";
    $thread = $comment['thread'];
    $postID = $postThreads[$thread];
    if (!isset($postThreads[$thread])) {
        $query = "SELECT * FROM wp_posts WHERE post_content LIKE '%".$thread."%' AND post_status='publish'";
        $postID = $dbRef->get_var($query, 0);
        $postThreads[$thread] = $postID ? $postID : 0;
        if ($postThreads[$thread] == 0)
            echo ("<br />Thread $thread has no post!");
        else
            echo "<br />Thread $thread";
        flush();       // got to have real-time updates!
    }
 
    if ($postID && $postID != 0) {
        $userId = $comment['email'] == 'vikingjs@mac.com' ? 1 : 0;
 
        //set up the data the way wp_insert_comment expects it.
        $wp_commentData = array();
        $wp_commentData['comment_post_ID'] = (int) $postID;
        $wp_commentData['user_id'] = (int) $userId;
        $wp_commentData['comment_parent'] = 0;
        $wp_commentData['comment_author_IP'] = $comment['ip'];
        $wp_commentData['comment_agent'] = 'Haloscan';
        $wp_commentData['comment_date'] = $comment['datetime'];
        $wp_commentData['comment_date_gmt'] = $comment['datetime'];
        $wp_commentData['comment_approved'] = '1';
        $wp_commentData['comment_content'] = $comment['text'];
        $wp_commentData['comment_author'] = $comment['name'];
        $wp_commentData['comment_author_email'] = $comment['email'];
        $wp_commentData = wp_filter_comment($wp_commentData);
 
        $comment_ID = wp_insert_comment($wp_commentData);
 
        //echo ("<strong>saved comment $comment_ID</strong>");
    }
 
    // try to reclaim some memory
    unset($wp_commentData);
    unset($comment);
}
 
header( 'Content-Type: text/html; charset=utf-8' );
?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<title>WordPress &rsaquo; Import Comments from RSS</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<style media="screen" type="text/css">
    body {
        font-family: Georgia, "Times New Roman", Times, serif;
        margin-left: 20%;
        margin-right: 20%;
    }
    #logo {
        margin: 0;
        padding: 0;
        background-image: url(http://wordpress.org/images/logo.png);
        background-repeat: no-repeat;
        height: 60px;
        border-bottom: 4px solid #333;
    }
    #logo a {
        display: block;
        text-decoration: none;
        text-indent: -100em;
        height: 60px;
    }
    p {
        line-height: 140%;
    }
    </style>
</head><body> 
<h1 id="logo"><a href="http://wordpress.org/">WordPress</a></h1> 
 
<?php
 
// Bring in the data
$reader = new XMLReader();
if ($reader->open('export-8.xml')) {
    $postThreads = array();
    $thread = '';
    while ($reader->read()) {
        //echo "<br />read node type: ".$reader->nodeType.';     '.$reader->name.': '.$reader->value;
        if ($reader->nodeType == XMLReader::ELEMENT && $reader->name == 'thread') {
            $thread = $reader->getAttribute('id');
        }
        if ($thread) {
            if ($reader->nodeType == XMLReader::ELEMENT && $reader->name == 'comment') {
                // begin building comment
                $comment = array('thread' => $thread);
                $reader->read();
                while ( !($reader->nodeType == XMLReader::END_ELEMENT && $reader->name == 'comment') ) {
                    if ($reader->nodeType == XMLReader::ELEMENT) {
                        $property = $reader->name;
                        $reader->read(); // assumes text element following element tag has the data
                        $comment[$property] = $reader->value;
                    }
                    $reader->read();
                }
                saveCommentToWP($comment, $wpdb, $postThreads);
            }
        }
    }
    $reader->close();
}
 
?>
 
 
</body>
</html>

Tags: , , ,
Posted in Rumblings from the Secret Labs | 6 Comments »

0
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

In with the Old

February 5th, 2010

I got a message today that Haloscan is closing down. That is the service that provided refreshingly spam-free comments on my old blog. A year ago I finally abandoned iBlog for WordPress, and I’m glad I did. At the time, however, I didn’t want to tackle moving the old comments over into the new system. In my conversion I embedded a link into each of the old episodes to the legacy comment system, and left it at that.

It is fortunate I found out about Haloscan when I did. Another week and 8500 comments would have been lost forever. That’s a big part of the underlayer of this blog, the part people sink gradually into as they hang around more, and they realize that this isn’t just about me. There are some pretty interesting conversations, observations, poems, and even stories in those comments. With the timer running I set to work to get the comments out of Haloscan and into WordPress.

The move turned out to be pretty straightforward. (Simpler, perhaps, than it had been to put the links into the posts.) I’ll go into the technical details in an episode tomorrow, but for now, why don’t you pop into the archives for 2004 or so and find an old episode with good comments? Maybe you’ll find something interesting someone said once. Maybe you’ll see the name of someone you haven’t thought of in a while. Maybe you’ll see something you want to comment on, even.

Tags: , , ,
Posted in Rumblings from the Secret Labs | No Comments »

0
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

A Job I’m Glad I Don’t Have

January 24th, 2010
Even cushy jobs can suck.

As you might be able to tell from the paucity of episodes here at MR&HBI, I’ve rejoined the ranks of the employed. My writing has taken a real beating, so today I’m going to spend some time writing about work. You don’t have to thank me, it’s what I do.

I don’t mind writing software; I’m pretty good at it and I can make pretty decent money doing it. I would much rather write code than dig ditches, for instance, and luckily for me the world has decided that making Web sites is worth more than roadside drainage. (Before you go and say, “that’s because it takes skill and training to make a Web site, but anyone could dig a ditch’, ask yourself – could you dig ditches for a living? If the economy were turned upside-down, that ditch-digger living in his nice house would say, ‘anyone can make a living sitting on their ass in front of a computer, but I dig ditches. I’m glad things are the way they are, is all I’m saying.)

My current job sends me dangerously into territory I don’t much like, however, and that’s the area known as Information Technology. It’s not really a good name for the job, which is about setting up computers and keeping them running. It’s less about making things and more about making things work.

Last night, for instance, I moved the Web product I’m working on to a different server and it didn’t work. Naturally I assumed the problem was in my code (it had worked on that server in the past), so it was several hours later that I discovered that for reasons I still don’t know, the server failed when it tried to compress very large messages. Just *poof* no response beyond the number 500 (something went wrong). To make things more fun the server was specifically set up to not write out a lot of error messages to its log. I turned off the compression feature (with a hammer) and things worked again. Five hours or so spent to add seven characters to a PHP file, to make things work the same way they already did on other servers. Welcome to the world of IT.

I think the original intention of the phrase information technology referred to the the information that would be stored, manipulated, and distributed by machines. What the I really stands for is the vast store of arcane crap you have to know to do that job well. What line of the php.ini file to modify if you want zlib output buffering and utf-8 character encoding. How to set up all the computers in an office to use a local domain name server first. That’s the information in IT.

The worst thing about having an IT job is this, however: When you’re doing a good job, no one notices. When a company is running smoothly, that’s a sign that the IT department can be downsized. There are no problems! What are those guys doing all day? Having things not happen as part of your job description makes for tricky times when you do your job well. Of course, when something does go wrong people know just where to find you.

So if you work in a company that has people on payroll working to keep your technology humming along, cut them a little slack. Someone’s got to do that stuff; be glad it’s not you. I do enough IT now to know that I’d rather let someone else have the pleasure.

Tags: , ,
Posted in Rumblings from the Secret Labs | 4 Comments »

0
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

MacPorts and GIMP

January 9th, 2010
MacPorts is an awesome innovation for getting all the little pieces to work together. Usually.

Setting up a new computer can be a tedious task; there are all kinds of settings and programs and files and whatnot that need to be passed from old to new. When you work with Web development things can get even more cumbersome, as one finds oneself descending deep into the world of IT. There are programs to install that all have to talk to each other, and configuration files to be tweaked. Many of the applications that are required have no user interface of their own, they simply run in the background and answer requests from other applications.

I found myself facing (for the third time in three months) the need to install the latest Apache (and set up virtual hosts), PHP, Pear, MySQL server, PHP email addons, Propel, and on and on. For many of these items, the instructions for installations go something like:

1) Download the source code
2) Configure the build
3) Compile the application

And the instructions go on from there. For most of the above there are shortcuts, and probably-recent-enough versions of some things come built-in with the Mac OS, but when you install it yourself you can get everything where you want it and avoid conflicts. Still, this can be a long, tedious, pain in the butt to get going. And when you install something and it doesn’t play well with the others, finding that one line in the secret config file that’s causing the problem can be a real pain.

Enter MacPorts. MacPorts is a project that has developed a system that does all the steps of the installation for you, and puts everything in standard places so everything else installed with MacPorts can find it and talk to it. There’s still some configuration to do (tell PHP where the database server’s socket is, and set passwords for instance), but overall things are much simpler, and there are very good instructions out there for tweaking and troubleshooting MacPort installs. Since the person writing the instructions knows where all the files are in the standard install, instructions can be much more specific.

With MacPorts installing php 5.3.1 was a simple matter of typing “sudo port install php5″ and letting the MacPort system do the rest. Hooray! Setting up a server is suddenly much simpler.

As an aside, MySQL didn’t work when I used MacPorts to install it on a previous machine. Don’t know why. Ran the install, followed the instructions, nothing. After a few hours banging my head against it, I went and got the excellent binary installer. It worked without a hitch. This time around I didn’t bother with the MacPorts version at all.

Anyway, thanks to MacPorts, I was able to get a complete development system up with nary a hitch, in a fraction of the time. Knowing where all the config files were this time around helped as well. I found several useful links on the Web, particularly from HiveLogic and my new buddy Danilo Stern-Sapad who took a little grammar rant from me with grace.

On that note, I read the phrase “How to setup…” so many times in so many places it’s amazing I still have teeth.

Last night I realized I hadn’t installed GIMP on this computer yet. GIMP is an open-source graphics program that wishes it rivaled Photoshop but it doesn’t really. It’s free, however, and when you consider the incredible amount of work that went into it, you have to be impressed. I don’t do a whole lot with graphics, so GIMP is usually adequate for my needs.

I went to the Web site for GIMP there were a couple of options for installation, including MacPorts. Just type “sudo port install gimp” into the terminal and that’s that. Pretty sweet.

One thing about a program like GIMP: it’s really a collection of a bazillion smaller parts. Many of those parts require other bits to work. When you run a traditional installer, all the parts are already there and they’re already tied together in a neat bundle. MacPorts does things a little differently; each part knows what parts it depends on to work, so when you say “install gimp” it first looks at the parts gimp needs, then at the parts those parts depend on, and so forth. You get to watch (if you choose to pay attention) the parade of all the little pieces as they’re installed, each the product of an individual or small group of people who have allowed their work to be exploited for free.

For GIMP, the list of dependencies goes very deep. I watched as X11 was installed. X11 is already on my computer, but ok, this is MacPorts and they keep their own realm and that way they can update parts without worrying about how that will affect non-MacPort installations. It’s redundant, but that’s why God made big hard drives.

Then I saw Python 2.6 install. Later, Python 2.5 went by. One of the little pieces seems to depend on an earlier version of Python. This probably means the person in charge of that bit just hasn’t updated it recently. On the installation went. After a while the Gnu Compiler Collection came down the pipe. gcc is a collection of compilers for building programs, and I watched as gcc was built… using the gcc already installed on my machine. Hm. And what’s this? Fortran! Yep, somewhere in the great tree of dependencies (maybe ‘root system’ would be a better term), someone decided that a Fortran compiler was necessary to run GIMP.

Actually, that’s not quite fair; the piece that loaded the Fortran compiler might need it for other tasks not related to GIMP. Just because GIMP uses a library doesn’t mean that’s the only use for it. Still, I ended up with a lot of stuff I don’t need. It’s all invisible and I’d never know it was there if I hadn’t watched the install (which took hours), so it’s not a disaster or anything. And next time I need Fortran I’m ahead of the curve!

Time slipped past, the install continued. I went to bed. When I got up this morning to check if the build was finished, I discovered there had been an error. Yep, the MacPort version of gimp-app doesn’t currently compile on 64-bit operating systems. All that other stuff that was installed? Python 2.6 and Python 2.5, the Gnu Compiler Collection (including Fortran), libgnome and libbonoboui and tkl and tk and gd2 and dozens of other things I don’t know what are, they’re still there, waiting to be useful in some way.

Tags: ,
Posted in Rumblings from the Secret Labs | 3 Comments »

1
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

Filling a Need

October 26th, 2009
A little diversion becomes a big hit.

I dragged my sorry butt out of bed just before 7 a.m. Big meeting. I put on my fuzzy bathrobe and plunked down in front of the computer while my sweetie made me tea. Then she went back to bed. We stayed up way too late last night.

Skype lit up and away we went. The meeting was more about my employer and the evolution of their corporate character than about technology, deadlines, or the transient issues of the day. How do you make a company run smoothly when you span from San Jose to Moscow? How do you make sure everyone is having a good time while you’re at it?

At one point in the conversation I was asked for bio data for my employer to put up on their Web site. I thought I would throw them a link to the bio page here here first, as a joke and also to let them learn a little more about who I am (or who I want to be, at least). I popped over here and saw… that this site had been suspended by my Web host. I checked my email. No notice. I checked my account on MMHosting’s site. Suspended, no reason given.

Some Mondays are Mondayer than others.

I sent off an urgent help request and spent the rest of the day bouncing between databases, php server code, and Flex client code, generally trying to be smart enough to deserve what they are paying me. Eventually I got a message back from MMHosting.

First, they said they had in fact sent an email. I searched everywhere I know how to search, and I couldn’t find it. No matter; they also turned the site back on. The guy said that some of my php files (code that runs on the server and builds these pages) was loading bazillions of times and slowing down the server. Uh, oh! Looks like some plugin I’m using ran amok. The people sharing the server with me probably weren’t happy.

That’s what I thought until I started looking at the numbers, anyway. Runaway software? Not at all! It turns out this table I wasted way too much time on got mentioned in a prominent place, and twitter and digg took care of the rest. When you look at the graph, remember that the site was down for much of the time during that traffic spike. Holy Schnikies!

Traffic for the last month. The bulge at the beginning is from Cyberspace Open traffic. Things have been slow since I started working - until today!

Traffic for the last month. The bulge at the beginning is from Cyberspace Open traffic. Things have been slow since I started working - until today!


I really did put a lot of work into that dang table, so I’m glad people are picking up on it. Maybe there are other CSS3 features I could tote up – transform and shadow come to mind.

Boy, I sure wish I had a killer episode at the top of the blog to hook some of these visitors. Oh, well.

Tags: , ,
Posted in Rumblings from the Secret Labs | 1 Comment »

1
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

Employed!

October 11th, 2009
Working for the man again...

Well, now I’ve gone and done it! I have a short-term job, helping to program up a fancy Web-based learning management system for a couple of companies in the medical industry. I didn’t really look for the job, but it found me anyway.

This month is going to be pretty crazy; and I won’t have nearly the time to get ready for the World Fantasy Convention now. On the other hand, I’ll be able to pay rent. That’s pretty important, what with this being the rainy season and all. Don’t know what’s going to happen to the ol’ blog here; I was a little bit employed at the start of my homeless tour and still managed to keep things alive, so let’s hope.

Long ago, when I was dabbling in radio, a mentor told me “if you have a safety net, you will fall into it.” I had a safety net, a job programming computers. I fell into it. I’m going to have to be very, very careful that I don’t allow that to happen again.

Tags:
Posted in Rumblings from the Secret Labs | 1 Comment »

« Older Episodes