Posts Tagged ‘geek’

« Older Episodes
0
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

An Online Community that I can Get Behind

February 2nd, 2012

Since there are others using my server now, I thought it would be a good idea to upgrade my backup practices. I looked around a bit, hoping for a solution that was free, butt-simple to set up, and automatic, so I would never have to think about it again. I don’t like thinking when I don’t have to.

I came across CrashPlan, the backup solution my employer uses. Turns out their software is free to chumps like me; they make their cash providing a place for you to put that valuable information.

There are two parts to any backup plan: you must gather your data together and you must put it somewhere safe that you can get to later. The CrashPlan software handles the gathering part, making it easy, for instance, to save all my stuff to the external hard drive sitting on my desk, but if the house burns down that won’t do me much good.

Happily CrashPlan also makes it easy to talk to remote computers, provided they have the software installed. I put CrashPlan on my server in a bunker somewhere in Nevada, and now this site and a couple of others are saved automatically to my drive in California as well. Easy peasy! Any computer signed up under my account can make backups to any other.

But wait! There’s more! The cool idea CrashPlan came up with was letting friends back each other up. I give you a special code and you can put backups of your stuff on my system. I can’t see what you saved, it’s all encrypted. But unless both our houses burn down at the same time, there’s always a safe copy.

Sure, if you pay you get more features and they will store your stuff in a safe place where you don’t have to wait if I happen to be on vacation, but for free that’s not bad at all. The idea of friends getting together and forming a backup community appeals to me as well. It’s a great way for geeks to look out for one another.

Tags: ,
Posted in Rumblings from the Secret Labs | No Comments »

2
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

Step-by-Step LAMP server from scratch with MacPorts

January 26th, 2012
A works-every-time guide to getting everything installed and configured.

Getting Apache, PHP, and MySQL installed and talking to each other is pretty simple — until something doesn’t come out right. This guide takes things one step at a time and checks each step along the way.

January 14, 2011
January 14, 2011

Using MacPorts to build a LAMP server from scratch

About this tutorial:

There are other step-by-step guides out there, and some of them are pretty dang good. But I’ve never found one that I could go through and reach the promised land without a hitch. (Usually the hitches happen around MySQL.) Occasionally key points are glossed over, but I think mostly there are things that have changed, and the tutorials haven’t updated. Now however I’ve done this enough times that there are no hitches anymore for me. Since MacPorts occasionally changes things, I’ll put up at the top of this page the last time this recipe was last used exactly as written here.

This guide breaks things down into very small steps, but each step is simple. I include tests for each stage of the installation, so problems can be spotted while they're easy to trace. We get each piece working before moving on to the next. I spend a little time telling you what it is you accomplish with each step, because a little understanding can really help when it’s time to troubleshoot, and if things are slightly different you have a better chance of working through them.

Audience: This guide is designed to be useful to people with only a passing familiarity with the terminal. More sophisticated techno-geeks may just want to go through the sequence of commands, and read the surrounding material only when something doesn't make sense to them. The goal: follow these steps and it will work every damn time.

MacOS X versions: Tiger, Leopard, Snow Leopard, Lion. Maybe others, too. The beauty of this method is it doesn’t really matter which OS X version you have.

The advantages of this approach

There are multiple options for setting up a Mac running OS X to be a Web server. Many of the necessary tools are even built right in. Using the built-in stuff might be the way for you to go, but there are problems: It’s difficult to customize (Search on install apc Mac and you’ll see what I mean), you don’t control the versions of the software you install, and when you upgrade MacOS versions things could change out from under you.

Also simple to set up is MAMP, which is great for developing but not so much for deployment. For simple Web development on your local machine, it’s hard to beat.

But when it comes right down to it, for a production server you want control and you want predictability. For that, it’s best to install all the parts yourself in a known, well-documented configuration, that runs close to the metal. That’s where MacPorts comes in. Suddenly installing stuff gets a lot easier, and there’s plenty of documentation.

Holy schnikies! A new timing exploit on OpenSSL! It may be months before Apple's release fixes it. I want it sooner!

What you lose:

If you’re running OS X Server (suddenly an affordable option), you get some slick remote management tools. You’ll be saying goodbye to them if you take this route. In fact, you’ll be saying goodbye to all your friendly windows and checkboxes.

Also, I have never, ever, succeeded in setting up a mail server, MacPorts or otherwise, and I’ve tried a few different ways (all on the same box, so problems left over from one may have torpedoed the next), and no one I've ever met, even sophisticated IT guys, likes this chore. If serving mail is a requirement, then OS X Server is probably worth the loss of control. Just don’t ever upgrade your server to the next major version. (Where’s MySQL!?! Ahhhhhh! I hear frustrated sys admins shout.)

So, here we go!

Document conventions

There are commands you type, lines of code you put in files, and other code-like things. I've tried to make it all clear with text styles.

This is something you type into the terminalDo not type the ; it's just to represent the prompt in your own terminal window. Once you've typed the text (or pasted it in from here), hit return.
This is a line of code in a file.Either you will be looking for a line like this, or adding a line like this.
This is a reference to a file or a path.

Prepare the Box

  1. Turn off unneeded services on the server box. Open System Preferences and select Sharing.
    • turn on remote login
    • (optional) turn on Screen Sharing
    • Turn off everything else - especially Web Sharing and File Sharing
  2. Install XCode. This provides tools that MacPorts uses to build the programs for your machine. You can get XCode for free from the app store. It’s a huge download. Note that after you download it, you have to run the installer. It may launch XCode when the install is done, but you can just Quit out of it.
  3. Install MacPorts. You can download the installer from http://www.MacPorts.org/install.php (make sure you choose the .dmg that matches the version of MacOS you are running). Run the installer and get ready to start typing.
  4. Now it’s time to make sure MacPorts itself is up-to-date. Open terminal and type
    1. sudo port selfupdate
    2. password: <enter your admin password>
    MacPorts will contact the mother ship and update itself.
    • If you're not familiar with sudo, you will be soon. It gives you temporary permission to act as the root user for this machine. Every once in a while during this process you will need to type your admin password again.
  5. May as well get into the habit of updating the installed software while we’re at it. Type
    1. sudo port upgrade outdated
    and you will most likely see a message that looks like an error but really says that there was nothing to upgrade. No biggie.
    • Make a habit of running these commands regularly. One of the reasons you're doing this whole thing is to make sure your server stays up-to-date. This is how you do it.

Install Apache

  1. Now it’s time to get down to business. All the stuff we’ve installed so far is just setting up the tools to make the rest of the job easier. Let’s start with Apache!
    1. sudo port install apache2
    • This may take a little while. It’s actually downloading code and compiling a version of the server tailored to your system. First it figures out all the other little pieces Apache needs and makes sure they’re all installed correctly. Hop up and grab a sandwich, or, if you're really motivated, do something else productive while you wait.
  2. When the install is done, you will see a prompt to execute a command that will make Apache start up automatically when the computer is rebooted. Usually you will want to do this. The command has changed in the past, so be sure to check for the message in your terminal window. As of this writing, the command is:
    1. sudo port load Apache2
  3. Create an alias to the correct apachectl. apachectl is a utility that allows you to do things like restart Apache after you make changes. The thing is, the built-in Apache has its own apachectl. To avoid confusion, you can either type the full path to the new apachectl every time, or you can set up an alias. Aliases are commands you define. In this case you will define a new command that executes the proper apachectl.
    1. In your home directory (~/) you will find a file called .profile - if you didn’t have one before, MacPorts made one for you. Note the dot at the start. That makes the file invisible; Finder will not show it. In terminal you can see it by typing
      1. ls -a ~/
      You will get back a list of all the files in your home directory, including the hidden ones that start with ..
    2. Edit ~/.profile and add the following line:
      1. alias apache2ctl='sudo /opt/local/apache2/bin/apachectl'
      • Edit how? See below for a brief discussion about editing text files and dealing with file permissions.
      • ~/.profile isn't the only place you can put the alias, but it works.
    3. You need to reload the profile info for it to take effect in this terminal session.
      1. source ~/.profile
    4. Now anywhere in the docs it says to use apachectl, just type apache2ctl instead, and you will be sure to be working on the correct server.
  4. Start Apache:
    1. apache2ctl start
    You might see a warning or two, probably a notification about the server's name. That's fine.
  5. Test the Apache installation. At this point, you should be able to go to http://127.0.0.1/ and see a simple message: “It works!”
  6. MILESTONE - Apache is up and running!

Install PHP

  1. Use MacPorts to build PHP 5:
    1. sudo port install php5 +pear
    • You could install the MySQL extensions to PHP now (sudo port install php5-mysql), but that will cause MySQL to be installed as well. It’s no biggie, but I like to make sure each piece is working before moving on to the next. It makes problem-solving a lot easier. So, let’s hold off on that.
    • +pear adds an industry-standard way to load other PHP addons later.
  2. Choose your php.ini file. There are a couple of different options that trade off security for convenience (error reporting and whatnot). As of this writing there is php.ini-development (more debugging information, less secure) and php.ini-production. Copy the one you want to use and name it php.ini:
    1. sudo cp /opt/local/etc/php5/php.ini-development /opt/local/etc/php5/php.ini
    You will be editing this file a little bit later, but mostly it’s just a bunch of settings you’ll never need to understand.
  3. Test the PHP install
    1. On the command line, type
      1. php -i
    2. A bunch of information will dump out. Hooray!
  4. Now it’s time to get Apache and PHP talking to each other. Apache needs to know that PHP is there, and when to use it. There’s a lot of less-than-ideal advice out there about how to do this.
    1. httpd.conf is the heart of the Apache configuration. Mess this up, Apache won’t run. It’s important, therefore, that you MAKE A BACKUP (there’s actually a spare copy in the install, but you never rely on that, do you?)
      1. cd /opt/local/apache2/conf
      2. sudo cp httpd.conf httpd.conf.backup
    2. First run a little utility installed with Apache that supposedly sets things up for you, but actually doesn’t do the whole job:
      1. cd /opt/local/apache2/modules
      2. sudo /opt/local/apache2/bin/apxs -a -e -n "php5" libphp5.so
    3. The utility added the line in the Apache config file that tells it that the PHP module is available. It does not tell Apache when to use it. There is an extra little config file for that job, but it’s not loaded (as far as I can tell), and it’s not really right anyway. Let's take matters into our own hands.
      • It won't let me save! See below for a brief discussion about editing text files and dealing with permissions.
    4. Time to edit! Open /opt/local/apache2/conf/httpd.conf with permission to edit it. We need to add three lines; one to tell it that PHP files are text files (not strictly necessary but let’s be rigorous here), and two lines to tell it what to do when it encounters a PHP file.
      1. Search for the phrase AddType in the file. After the comments (lines that start with #) add:
        1. AddType text/html .php
      2. Search for AddHandler (it’s just a few lines down) and add:
        1. AddHandler application/x-httpd-php .php
        2. AddHandler application/x-httpd-php-source .phps
        The second of those is just to let you display PHP source code in a Web page without actually running it.
      3. Finally, we need to tell Apache that index.php is every bit as good as index.html. Search in the config file for index.html and you should fine a line that says DirectoryIndex index.html. Right after the html file put index.php:
        • Before:
          1. DirectoryIndex index.html
        • After:
          1. DirectoryIndex index.html index.php
      4. (Optional) As long as we’re in here, let’s make one more change for improved security. Search for the line that specifies the default options for Apache and remove Indexes:
        • Before:
          1. Options Indexes FollowSymLinks
        • After:
          1. Options FollowSymLinks
        This prevents outsiders from seeing a list of everything in a directory that has no index file.
      5. Save the file.
    5. Check the init file syntax by typing
      1. /opt/local/apache2/bin/httpd -t
      You will probably get a warning about the server’s name again, but that’s OK, as long as you see the magical Syntax OK message. If there is an error, the file and line number should be listed.
    6. Restart Apache:
      1. apache2ctl restart
  5. Test whether PHP and Apache can be friends. We will modify the “It Works!” file to dump out a bunch of info about your PHP installation.
    1. Currently the default Apache directory is /opt/local/apache2/htdocs
    2. Start by renaming index.html to index.php:
      1. cd /opt/local/apache2/htdocs
      2. sudo mv index.html index.php
    3. Edit the file, and after the It Works! bit add a PHP call so the result looks like this:
      1. <html>
      2.     <body>
      3.         <h1>It works!</h1>
      4.         <?php echo phpinfo(); ?>
      5.     </body>
      6. </html>
    4. Save the file
    5. Go to http://127.0.0.1 - you should see a huge dump of everything you wanted to know about your PHP but were afraid to ask.
  6. MILESTONE - Apache and PHP are installed and talking nice to each other.

Install and configure MySQL

  1. Use MacPorts to install MySQL database and server and start it automatically when the machine boots:
    1. sudo port install mysql5-server
    2. sudo port load mysql5-server
  2. Now we get to the trickiest part of the whole operation. There's nothing here that's difficult, but I've spent hours going in circles before, and I'm here so you won't find yourself in that boat as well. MySQL requires some configuration before it can run at all, and it can be a huge bother figuring out what’s going on if it doesn’t work the first time. We start by running a little init script:
    1. sudo -u _mysql mysql_install_db5
  3. As with Apache, you can create a set of aliases to simplify working with MySQL. There are some commands you will run frequently; things get easier if you don’t have to type the full path to the command every time. Open up ~/.profile again and add the following three lines:
    1. alias mysqlstart='sudo /opt/local/share/mysql5/mysql/mysql.server start'
    2. alias mysql='/opt/local/lib/mysql5/bin/mysql'
    3. alias mysqladmin='/opt/local/lib/mysql5/bin/mysqladmin'
    When you're done, save and
    1. source ~/.profile
  4. Start MySQL server:
    1. mysqlstart
  5. Next we need to deal with making the database secure and setting the first all-important password. The most complete way to do this is running another utility that takes you through the decisions.
    1. /opt/local/lib/mysql5/bin/mysql_secure_installation
    The script offers to delete some test users and databases that in my experience are totally useless anyway. Take the advice offered and get rid of all that junk.
    Remember the password you set for the root user!
    • You now have a MySQL account named root which is not the same as the root user for the machine itself. When using sudo you will use the machine root password (as you have been all along), but when invoking mysql or mysqladmin you will enter the password for the database root account.
  6. As with PHP above, MySQL has example config files for you to choose from. The config file can be placed in a bunch of different places, and depending on where you put it, it will override settings in other config files. If you follow this install procedure, you don’t actually need to do anything with the config files; we’ll just be using the factory defaults. But things will work better down the road if you choose a config that roughly matches the way the database will be used.
    1. Find where the basedir is. As of this writing it’s /opt/local, and that’s not likely to change anytime soon, but why take that for granted when we can find out for sure? Let's make a habit of finding facts when they're available instead of relying on recipes like this one.
      1. mysqladmin -u root -p variables
      2. password: <enter MySQL root user's password>
      A bunch of info will spew across your screen. At this moment, there are two interesting nuggets: basedir and socket. Make a note of them for later.
    2. Now it’s time to choose which example config file you want to start with. The examples are in /opt/local/share/mysql5/mysql/, and each has a brief explanation at the top that says what circumstances it’s optimized for. You can read those, or just choose one based on the name. If you have no idea how big your database is going to be, medium sounds nice. You can always swap it out later.
      1. sudo cp /opt/local/share/mysql5/mysql/my-medium.cnf <basedir>/my.cnf
      Fill in <basedir> with the basedir you learned in the previous step.
  7. Test MySQL
    1. On the command line, type
      1. mysql -u root -p
      2. password: <enter MySQL root user's password>
      and enter the MySQL root user password when prompted. No errors? Cool. We’re done here. Type
      1. exit
      at the prompt.
  8. MILESTONE - MySQL server is running and happily talking to itself.

Teach PHP where to find MySQL

  1. The database is up and running; now we need to give PHP the info it needs to access it. There's a thing called a socket that the two use to talk to each other. Like a lot of things in UNIX the socket looks like a file.

    The default MySQL location for the socket is in /tmp, but MacPorts doesn’t play that way. There are a couple of reasons that /tmp is not an ideal place for the socket anyway, so we’ll do things the MacPorts way and tell PHP that the socket is not at the default location. To do this we edit /opt/local/etc/php5/php.ini.

    There are three places where sockets are specified, and they all need to point to the correct place. Remember when you saved the socket variable from MySQL before? Copy that line and then search in your php.ini file for three places where is says default_socket:

    1. pdo_mysql.default_socket = <paste here>
    2. . . .
    3. mysql.default_socket = <paste here>
    4. . . .
    5. mysqli.default_socket = <paste here>

    In each case the whatever = part will already be in the ini file; you just need to find each line and paste in the correct path.

  2. While we’re editing the file, you may want to set a default time zone. This will alleviate hassles with date functions later.
  3. Finally, we need to install the PHP module that provides PHP with the code to operate on MySQL databases.
    1. sudo port install php5-mysql
  4. Restart Apache:
    1. apache2ctl restart
  5. Test the connection.
    1. Typing
      1. php -i | grep -i 'mysql'
      Should get you a list of a few mysterious lines of stuff.
    2. Second test: The whole bag of marbles. You ready for this?
      1. In the Apache’s document root (where the index.php file you made before lives), create a new file named testmysql.php
      2. In the file, paste the following: 
        1. <?php
        2. $dbhost = 'localhost';
        3. $dbuser = 'root';
        4. $dbpass = 'MYSQL_ROOT_PASSWRD';
        5. $conn = mysql_connect($dbhost, $dbuser, $dbpass);
        6. if ($conn) {
        7.     echo 'CONNECT OK';
        8. } else {
        9.     die ('Error connecting to mysql');
        10. }
        11. $dbname = 'mysql';
        12. mysql_select_db($dbname);
      3. Edit the file to replace MYSQL_ROOT_PASSWRD with the password you set for the root database user.
      4. Save the file.
    3. In your browser, go to http://127.0.0.1/testmysql.php
    4. You should see a message saying “Connection OK”
  6. MILESTONE - Apache, PHP, and MySQL are all working together. High-five yourself, bud! You are an IT God!

Set up virtual hosts.

Finally, we will set up virtual hosts. This allows your server to handle more than one domain name. Even if you don't think you need more than one domain, it's a safe bet that before long you'll be glad you took care of this ahead of time.

We will create a file that tells Apache how to decide which directory to use for what request. There is an example file already waiting for us, so it gets pretty easy.

  1. Tell Apache to use the vhosts file. To do this we make one last edit to httpd.conf. After this, all our tweaks will be in a separate file so we don’t have to risk accidentally messing something up in the master file.
    1. In /opt/local/apache2/config/httpd.conf, find the line that says
      1. #Include conf/extra/httpd-vhosts.conf
      and remove the #.
    2. The # told Apache to ignore the include command. Take a look at all those other files it doesn’t include by default. Some of them might come in handy someday...
    3. Save the file and restart Apache
    4. Test by going to your old friend http://127.0.0.1
    5. Forbidden! What the heck!?! Right now, that's actually OK. The vhosts file is pointing to a folder that doesn't exist and even if it did it would be off-limits. All we have to do is modify the vhosts file to point to a directory that actually does exist, and tell Apache it's OK to load files from there.
  2. Before going further, it's probably a good idea to figure out where you plan to put the files for your Web sites. I've taken to putting them in /opt/local/www/domain.com/public/ - not through any particular plan, but /opt/local/www is the default location for phpMyAdmin and I just went with it. The public part is so you can have other files associated with the site that are not reachable from the outside.
  3. Set up the default host directory
    1. Open /opt/local/apache2/conf/extra/httpd-vhosts.conf for editing.

    2. You will see two example blocks for two different domains. Important to note that if Apache can’t match any of the domains listed, it will default to the first in the list. This may be an important consideration for thwarting mischief.

      The examples provided in the file accomplish one of the two things we need to get done — they tell Apache what directory to use for each domain, but they do nothing to address what permissions Apache has in those directories. A lot of people put the permissions stuff in the main httpd.conf, but why not keep it all in one place and simplify maintenance while we reduce risk?

      Here's an example:

      1. <VirtualHost *:80>
      2.     ServerAdmin [email protected]
      3.     DocumentRoot "/opt/local/www/mydomain.com/public"
      4.     ServerName mydomain.com
      5.     ServerAlias www.mydomain.com
      6.     ErrorLog "logs/mydomain.com-error_log"
      7.     CustomLog "logs/mydomain.com-access_log" common
      8.     <Directory "/opt/local/www/mydomain.com/public">
      9.         Options FollowSymLinks
      10.         AllowOverride None
      11.         Order allow,deny
      12.         Allow from all
      13.     </Directory>
      14. </VirtualHost>

      You can see where it sets what directory to go to, where it says to treat www.mydomain.com the same as mydomain.com, and then in the Directory block it sets permissions. The actual permissions instructions are pretty arcane. The most important thing to note is the line

      1. AllowOverride none
      This is not typical, but it's better, as long as you don't forget you did it.

      Here's the skinny: A lot of web apps like WordPress and Drupal need to set special rules about how certain requests are handled. They use a file called .htaccess to set those rules. By setting AllowOverride none you're telling Apache to ignore those files. Instead, you can put those rules right in the <Directory> blocks in your vhosts file. It saves Apache the trouble of searching for .htaccess files on every request, and it's a more difficult target for hackers. .htaccess is for people who don't control the server. You do control the server, so you can do better.

      1. If others will be putting sites on the server and you don't want them fiddling with the config files, you can allow .htaccess to override specific parameters. Read up in the Apache docs to learn more.
      2. If you are using SSL, you also need to set up a VirtualHost entry for port 443. That entry will also include the locations of the SSL certificates.
    3. Add further blocks that match the domains you will be hosting.
    4. Restart Apache and test your setup. http://127.0.0.1 should go to your default directory. Testing the domains is trickier if you don’t have any DNS entries set up for that server. I’ll write up a separate document about using /etc/hosts to create local domains for this sort of test.
  4. MILESTONE - You have done it. A fully operational LAMP environment on your Mac, suitable for professional Web hosting.

(Optional) Install phpMyAdmin

phpMyAdmin makes some database operations much easier. There have been security issues in the past, so you might reconsider on a production machine, but on a development server it can be a real time saver.

    1. sudo port install phpMyAdmin
  2. Update your Virtual Hosts with the domain you want to use to access phpMyAdmin, which is by default at /opt/local/www/phpmyadmin/
  3. test - log in as root.
  4. Configure - configuring phpMyAdmin fills me with a rage hotter than a thousand suns. It just never goes smoothly for me, whether I use their helper scripts or hand-roll it while poring over the docs. Maybe if I do it a few more times I’ll be ready to write a cookie-cutter guide for that, too. In the meantime, you’re better off getting advice on that one elsewhere.

Wrapping Up

I hope this guide was useful to you. I'm he kind of guy who learns by doing, and I've made plenty of mistakes in the past getting this stuff working. Funny thing is, when it goes smoothly, you wonder what the big deal was. Hopefully you're wondering that now.

If you find errors in this guide, please let me know. Things change and move, and I'd like this page to change and move with them.

Keep up to date: One of the big advantages of this install method is that updates to key software packages get to your server faster. Use that power. Run the update commands listed in step one regularly.

  1. The script that tests the PHP-MySQL connection is based on one I found at http://www.pinoytux.com/linux/tip-testing-your-phpmysql-connection

Appendices

Appendix 1: A brief explanation of sudo

In the UNIX world, access to every little thing is carefully controlled. There's only one user who can change anything they want, and that user is named root.

When you log in on a Mac, you're not root, and good thing, too. But as an administrator, you can temporarily assume the root role. You do this by preceding your command with sudo. (That's an oversimplification, and you will have earned another Geek Point when you understand why. In the meantime, just go with it. sudo gives you power.)

When you use sudo, you type your password and if the system recognizes you as an administrator it will let you be root for that command.

For convenience, you only have to type your password every five minutes, but you do need to repeat 'sudo' for each command.

Just remember, as root you can really mess things up.

Appendix 2: On editing text files and permissions

Jerry told me to edit the file, you lament, but he didn't say how. Kind of strange, considering the minute detail of the rest of the guide. The thing is, there's not one easy answer.

Let's start with the two kinds of text editors. There are editors like vim and pico that run right in terminal. They are powerful, really useful for editing files on a remote box, and if you know how to use them you're not reading this footnote. The other option is a windowed plain-text editor. TextEdit is NOT a plain-text editor. There are a lot of plain-text editors out there, and they all have their claims to fame. You can use any of them to edit these files.

Whoops! That brings us to the gotcha: permissions. In UNIX, who can change what is tightly controlled. Many of the files we need to edit are owned by root, the God of the Machine, so we need to get special permission to save our changes. Many of the plain-text editors out there will let you open the file, but when it comes time to save... they can't. You don't have permission.

Some editors handle this gracefully, however, and let you type your admin password and carry on. BBEdit and its (free) little brother TextWrangler give you a chance to type your password and save the file. I'm sure there are plenty of others that do as well.

BBEdit and TextWrangler also allow you to launch the editor from the command line, so where I say above edit ~/.profile, you can actually type edit ~/.profile and if you have TextWrangler installed, it will fire right up and you'll have taken care of the permissions issue. (If you decided to pay for BBEdit, the command is bbedit ~/.profile.) I'm sure there are plenty of other editors that do that too.

I'm really not endorsing BBEdit and TextWragler here; they just happen to be the tools I picked up first. Over time I have become comfortable with their (let's call them) quirks. Alas, finding your text editing answer is up to you. If you're starting down this path, it's only a matter of time before you pick up rudimentary vim or pico skills; eventually you'll be using your phone to tweak files while you're on the road. It's pretty empowering. But is now the time to start learning that stuff? Maybe not. It's your call.

Tags: ,
Posted in Rumblings from the Secret Labs | 5 Comments »

1
Thanks!
The Working LIfeThe Working LIfe

A Visit From Steve

January 1st, 2012
Does this dream make my geek look big?

Steve Jobs came to visit me in a dream last night. He was a younger version with badly-bleached hair that turned out on the orange side. He was very animated as we discussed the best way to add advanced table features to Safari. Steve was as intense as people say he was when he was alive, and we got along great.

Tags:
Posted in The Working LIfe | 1 Comment »

2
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

How This Blog Works

August 20th, 2011
Were it not for a couple of bad Web hosts, I might never have learned all this stuff.

Over the years, the technology behind this blog has gone from cave-dwelling stone-knives-and-bearskin static pages to cloud-city jet-packs-and-lightsaber dynamic yumminess. That transformation starts with WordPress but does not end there. Not by a long shot.

I started the Muddled Media Empire using a tool called iBlog, because it was free and worked with Apple’s hosting service, which I was already paying for. iBlog’s claim to fame was that it didn’t require a database – every time you made a change it went through and regenerated all pages that were affected. Toward the end, that was getting to be thousands of pages in some cases, each of which had to be uploaded individually. When iBlog’s support and development faltered, it was already past time for me to move on.

WordPress is an enormously popular Web-publishing platform. It comes in two flavors: you can host your blog on their super-duper servers and accept their terms of service and the slightly limited customization options, or you can install the code on your own server and go nuts. I chose the latter, mainly because I wanted to be able to touch the code. I’m a tinkerer.

So I signed up for a cheap Web host and set to work building what you see now. At first things were great, but after a while the host started having issues, and the once-great customer service withered up and vanished. So much for LiveRack. I think they just didn’t want to be in the hosting business anymore. I moved to iPage.

iPage was cheap, but I was crammed onto a server with a bunch of other people and sometimes my blog would take an agonizing time to load. Like, almost a minute. Then there was the time a very popular Geek site linked to my CSS border-radius table and iPage shut me down because the demand on the server was too much. Ouch! My moment in the sun became my moment at the bottom of a well.

I set out to find ways to make this blog more server-friendly and more user-friendly at the same time. Step 1: caching. WordPress doesn’t store Web pages, it stores data and the instructions on how to build a Web page. So, every time you ask to load a page here, WordPress fires up a program that reads from the database and assembles all the parts to the page. The thing is, that takes longer than just finding the requested file and sending it back, the way iBlog did. Caching is a way for the server to say, “hey, wait a minute – I just did this page and nothing’s changed. I’ll just send the same thing I did last time.” That can lead to big savings, both in time and server load.

I looked at a few WordPress cacheing programs and eventually chose W3 Total Cache, because it does far more than just cache data. For instance, it will minify scripts and css files (remove extra spaces and crunch them down) and combine the files together so the browser only has to make one request. It will zip the data, meaning fewer 1′s and 0′s moving down the pipe, and it does a few other things as well, one of which I will get to shortly.

I installed W3 Total Cache, and although some settings broke a couple of javascripts (for reasons I have yet to figure out – I’ll get to that someday), the features I could turn on definitely made a difference. Hooray!

But Muddled Ramblings and Half-Baked Ideas was still way too slow. I continued my search for ways to speed things up. I also began a search for a host that sucked less than iPage. (iPage was also starting to have outages that lasted a day or more. Not acceptable.) I decided I was willing to pay extra to be sure I wasn’t on an overwhelmed machine.

I’m not sure which came first – new server or Amazon Simple Storage Service. S3 is a pretty basic concept – you put your stuff on their super-duper servers, and when people need it they will get it really quickly. Things that don’t change, like images and even some scripts, can live there and your server doesn’t have to worry about them.

This is where W3 Total Cache earned my donation to their cause. You see, you can sign up for Amazon S3, and then put your account info into the proper W3TC panel and Bob’s Your Uncle. W3TC goes through your site, finds images and whatnot, puts them in your S3 bucket, and automatically changes all the links in your Web pages to point to your bucket instead of your own server. (Sometimes I find I have to copy the image to my S3 bucket manually, but that’s a small price to pay.)

Now a lot of the stuff on my blog, like the picture of me with the Utahraptors the other day, sits on a different, high-performance server out there somewhere, and no matter how overwhelmed my server happens to be at the moment those parts will arrive to you lickety-split. Amazon S3 is not free, however – each month I get an invoice for two or three cents. Should Muddled Ramblings suddenly become wildly popular, that number would increase.

About that server – the next stop on my quest for a good host was a place called Green Geeks. I wanted to upgrade to a VPS, which means I get a dedicated slice of a server that acted just like it was my very own machine. There is a lot to like about those, but my blog just wouldn’t run in the base level of RAM they offered. I upgraded and reorganized so that different requests would not take up more ram than they needed. Still, I had outages. Sometimes the server would just stop freeing up memory and eventually choke and die. Since it was a virtual server in a standard configuration, logic says it was caused by something I was doing, but all my efforts to figure it out were fruitless, and Green Geeks ran out of patience trying to help me figure it out.

The server software itself is Apache. At this point I considered using nginx (rhymes with ‘bingin’ ex’) instead. It’s supposedly faster, lighter, and easier to configure. But, I already know Apache. I may move to nginx in the future, but it’s not urgent anymore.

During the GreenGeeks era I came across another service that improves the performance of Web sites while reducing the load on the servers. I recently wrote glowingly about CloudFlare, but I will repeat myself a bit here for completeness. CloudFlare is a service that has a network of servers all over the world, and they stand between you the viewer and my server. They stash bits of my site all around the world, and much of the time they will have a copy of what you need on hand, and won’t even need to trouble my server with a request. About half of all requests to muddledramblings.com are magically and speedily taken care of without troubling my server at all. They also block a couple thousand bogus requests to my server each day, so I don’t have to deal with them (or pay for the bandwidth). It’s sweet, and the base service is free.

Unfortunately, it was not enough to keep my GreenGeeks server from crashing. Once more I began a search for a new host. I found through word of mouth a place called macminicolo. Apple employees get a discount, but I wasn’t an Apple employee yet. It was still a bargain. For what turned out to be the same monthly cost of sharing part of a machine at GreenGeeks, I get an entire server, all to myself, with plenty of RAM. I’ve set up several servers on Mac using MacPorts, and I knew just how to get things up and running well. It costs less than half what a co-located server costs anywhere else I have found (Mac, Windows, or Linux). (Co-location has up-front costs, but in the long term saves money.) So I have that going for me.

The only thing missing is that at GreenGeeks I had a fancy control panel that made it much simpler to share the machine with my friends. I do miss that, but I’m ready now to host friend and family sites at a very reasonable cost.

So there you have it! This is just your typical Apache/WordPress/W3 Total Cache/Amazon S3/CloudFlare site run off a Mac mini located somewhere in Nevada. Load times are less than 5% of what they were a year ago. Five percent! Conservatively. Typically it’s more like 1/50th of the load time. Traffic is up. Life is good.

Now I have no incentive at all to learn more about optimization.

Tags: , , , , , ,
Posted in Rumblings from the Secret Labs | No Comments »

2
Thanks!
ObservationsObservations

Must be a Very Long Train

May 13th, 2011
Or maybe a quantum train?

I’m traveling to beautiful scenic Lawrence, Kansas this summer, and I thought I’d see if taking the train was an option. On the plus side, the Southwest Limited passes right through town; if I flew I’d have to arrange transport from Kansas City. On the minus side, traveling by rail in this country is pricey. Back on the plus side, a stop in Santa Fe for a few days is trivial – the train goes through Lamy.

As I perused my options I came upon this table:

The Southwest Chief

Note that, depending on how I reach Los Angeles, the Southwest Chief departs at different times. The back end of the train catches up with the front end over the course of the journey; the arrival time is almost the same.

Sometimes a movie maker will see a shot in a film and have to ask, “how did they do that?” Most of the time, a question like that is a compliment. But here I am, a Web/database guy, asking, “how did they do that?” and it’s with a disbelieving shake of the head. Who on this planet would design a system that allowed such inconsistency? Trust me, it takes extra work to get system behavior like that.

Don’t tell the people signing my time sheets every week, but this stuff is not that hard.

Tags: , ,
Posted in Observations | No Comments »

0
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

Securing Dropbox

April 20th, 2011

As I mentioned recently, Dropbox is awesome. When using it, however, it’s important to think about security.

Think of it this way: You’re on a cruise ship, and you have a priceless diamond tiara (don’t we all?). You know it’ll be much safer in the ship’s vault than in your cabin. The ship’s purser is only too happy to watch over your valuables in their very strong safe. Now you can rest easy.

Except… there’s someone else who can open the vault. What if the government serves the purser with a warrant (or some other constitutionally-questionable writ) and takes your tiara? What if someone fools the purser into handing over your tiara? For most things, trusting the purser is fine, but that tiara is really something special. What you need, then, is a special box with a really strong lock. You give the purser the box and neither he nor anyone else can even see what’s inside, and you can make it a really strong box, so even if the purser hands over the keys to his vault, your stuff is still safe.

The same principle applies with Dropbox. It’s really convenient and pretty darn secure, but someone else is holding the keys. For most things, like my writing, no further security is necessary. Yet I have a few files that I don’t want to leave to someone else to protect, but I still want the convenience and data backup Dropbox provides. On my mac I’ve set up a very simple system that allows me to see my most secret files whenever I need to on any of my machines, but protects them from prying eyes. It’s actually pretty simple, and there’s almost certainly a direct analog on Windows.

The disk utility that comes with Macs can create an encrypted disk image using pretty dang strong encryption. If you put that image file in your dropbox, then any files you add to that virtual disk will encrypted and saved to your Dropbox when you unmount the disk. Here are the steps:

  1. Fire up Disk Utility (it’s in the Utilities folder).
  2. Click New Image
  3. Decisions, decisions….
    • name your new disk. If you name it “secret stuff” that will just make people curious.
    • Size: For reasons I’ll go into shortly, I’d advise not making this any bigger than you really need. If you’re protecting text files, it can be pretty small. The 100MB setting is probably more than enough for most people.
    • Format: Just use the default
    • Encryption: I say, what the heck. Go for the maximum unless you’ll be using a really old machine.
    • Partition: just use the default.
    • Image Format: sparse disk image – this will keep the size of the actual disk file down.
  4. Save. You will be asked for a password. You’ll not need to remember it, so make it good and strong, nothing like any password you’ve used anywhere else. Keep the “save in keychain” option selected. (If you need it later, you can find it with Keychain Access.) – Remember: this is the secret that protects all your other secrets.
  5. Voila! Put the disk image in your Dropbox folder. When you open the image file, a new hard drive will appear in finder. Anything you put on the drive will be added to the .dmg file you created.
  6. “Eject” the drive on that machine and open the .dmg on any other machines you want to share the information. While you remember your crazy password, get it saved in the keychains of your various machines.

A couple of notes:

  • The .dmg file will only update when you “eject” the drive. So I advise you not keep it mounted most of the time. Open it, add/access the files inside, and close it again. If you open it on two machines at the same time, you will end up with two versions in your Dropbox folder.
  • I advised saving your password on your keychain, but remember that anyone who can access your computer can also access your secrets. That’s pretty much true whatever you do, but you might want to consider not putting the password on your laptop, for instance, if you think it might fall into the wrong hands.
  • Since your secret files are saved as a single blob of data, you won’t have automatic backups of individual files. If you need to recover one, you’ll have to find the right version of the image file.
  • Since your information is saved as a big ol’ blob, if you make a huge .dmg file it will eat up space in your Dropbox and burn up unnecessary bandwidth each time your save.
  • It’s possible to set things up to protect individual files, but it’s complicated. Hopefully it won’t always be.
  • Important! If you only store the password on one machine – Save it somewhere else also!. If you lose that password (if your hard drive crashes or your computer is stolen, for instance), you’re not getting into your strongbox. Ever. That was the whole point, after all.

Tags: ,
Posted in Rumblings from the Secret Labs | 5 Comments »

0
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

Excel 2011 for Mac, UNIX Time, and Visual Basic for Applications

March 23rd, 2011
Because it wasn't quite complicated enough.

Note to people looking for a formula: Yes, the code is here (for Mac and Windows, even). I tend to go long-winded even in technical articles, but if you’re dealing with converting UNIX time to Excel time, the answers lie below. You can skip ahead or read my brilliant and entertaining *cough* analysis first.

Microsoft Excel uses a method to represent time that is both smart and frustrating. How do they manage this? They take a good engineering solution, then fiddle with it. First some background:

Long before Y2K people who knew what they were doing had already abandoned the practice of using strings of text to represent dates in a computer. Using strings like “3/10/2011″ to represent a date has plenty of drawbacks, from cultural (is that March 10th or October 3rd?) to performance-related (sort 3/10/2011, 4/2/1902, 3/8/2012). Therefore long ago people who were smarter than I am came up with other ways to represent time. Happliy, time is nice and linear. All you really need is a number line. Remember them? A number line stretches from zero to infinity in both directions. To measure time all you need to do is decide on a zero point and then any point in the history of the universe can be represented by some number of time units from that instant.

My first exposure to a more rational way to measure time was in the old MacOS. I don’t remember anymore exactly when the zero point was, or even what the units chosen were. I do remember that the number gets too big for the computer to handle some time in 2014. Ancient Macs will have a problem then. I blame the Aztecs.

The UNIX boys count seconds from January 1, 1970 at 00:00. You get special Geek Cred if you went to a party to celebrate second 1234567890 of the Unix epoch. 32-bit computers that use the Unix epoch will break in 2038, when the number of seconds gets too big to fit in 32 bits. (Note also that you can only go back a finite distance before the negative number goes out of the range the processor can handle.

That’s all well and good, but I’m here to discuss Microsoft Excel today, and in particular Microsoft Excel for Mac. Excel counts in days, but allows fractional values. 12.5 represents noon twelve days after the zero point. I haven’t checked, but I think this system works for dates far, far into the future. So good on Microsoft for coming up with it. (As long as you don’t need dates before the zero time. As far as I can tell, Excel doesn’t handle them).

Of course, there are a couple of caveats. First: the historical oddity. In Excel, the day February 29, 1900 exists. Alas, there never was such a day. Microsoft included this error because they wanted to be compatible with Lotus, who simply messed up. To change it now would cause problems, because the zero point for the Microsoft time is January 1, 1900. Every date in every spreadsheet would suddenly be off by one. A thousand years from now we may still be calculating time based on the insertion of a bogus day.

Oh, except that Microsoft time doesn’t always start in 1900, and here’s where things start to get squirrely. If you’re using Excel for Mac, the default day zero is January 1, 1904. Except when it isn’t. More on that in a bit.

I descended into Excel recently to write a macro that does fancy formatting based on data I dump from a Web-based tracking tool I’m building. The dates in my data are based on the UNIX epoch, so I need to convert them. I dump the raw data into one sheet and then display it correctly converted and formatted on the main sheet that people actually look at. Here’s the code I use in a cell of the spreadsheet that needs to show a converted date:

    =DATE(1970,1,1)+import!Z3/(60*60*24)

where the unix time is in cell Z3 of the ‘import’ sheet. This divides the unix time by the number of seconds in a day, which gives me the number of days that have passed since the UNIX zero time. The formula then adds on the number of days from the spreadsheet zero to the UNIX zero time. (I could just say 86,400 instead of 60*60*24, but this way I can tell at a glance I’m dealing with days, and speed will not be an issue.) Happily, this formula will work on both Mac and Windows versions of Excel, because the DATE function will return the right value for the start of the unix epoch based on whichever version of Excel is running.

So, no problem, right?

Well… except. I also have some more fancy work to do that requires scripting. The good news: Mac Excel 2011 uses Visual Basic for Applications (VBA), which while imperfect is a zillion times better than AppleScript. So away I went, coding with a twinkle in my eye and a song in my heart. To convert dates, I naturally followed the same plan I did in the sheet’s cells: get the value for 1/1/1970, then add the unix epoch days.

And the dates came out different. Yep, when scripting, Excel always uses the Windows zero time, even when the spreadsheet in question uses the Mac zero time. Dates calculated in cells in the sheet are four years different than dates calculated using the same method in a script.

Aargh. Of course, once I realized what the problem was, it was not too difficult to work around it. I just lost some of the portability of my code, because now it has to be tweaked based on what the zero date of the spreadsheet is.

An aside here: If you’re here to resolve some date confusion in your own Mac spreadsheet, I strongly recommend you start by going to Preferences->Calculation->Workbook Options and uncheck the “use 1904 date” option. Unfortunately it will not recalculate the dates already entered in your sheets, so if that’s a problem then it’s too late for you, bucko. Read on.

Here’s some not-as-portable-as-it-could-be code. You need to choose one of the first two lines based on whether your sheet uses mac dates or windows dates:

'excelZeroDate = DateSerial(1904, 1, 1)              ' mac zero
excelZeroDate = 0      ' DateSerial(1900, 1, 1)     ' windows zero
unixZeroDate = DateDiff("d", excelZeroDate, DateSerial(1970, 1, 1))

Then if I have a date in the ‘import’ sheet to convert, I can write something like:

startDate = DateAdd("s", Worksheets("import").Cells(dateRow, dateColumn), unixZeroDate)

The nice part is that these functions handled converting seconds and days for me. Overall it’s not a bad system if you overlook the part where a single application gives two different answers to the same question.

Tags:
Posted in Rumblings from the Secret Labs | 4 Comments »

1
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

Well, THAT Sucked

November 21st, 2010
It's particularly frustrating when things go wrong and there's nothing you can do about it.

The last few days my Web host has been having a tough time. I don’t know the exact nature of the problem and I doubt I ever will, but this site has been broken. For a while it would not load at all, and then it was in ‘read-only mode’, Which meant that it was still performing terribly and I couldn’t even put up a notice that I knew things weren’t going well but the solution was out of my hands. Not a good situation when my credibility as a programmer is an important asset.

I couldn’t even make a backup.

Things seem to be getting back to normal (though they are not there yet – the site is still quite slow). There’s even a chance that I’m running on a brand-new server that is not being shared with as many other people. Or at least a brand-new server. Unfortunately, however, while I have come to appreciate iPage the company, which was very helpful and patient getting me up and running, iPage the service has not been so great.

I have vowed that the next move I make will be to a server that I control completely, so I can choose who shares it with me. I’m looking at Co-location deals now, though I might wimp out ant take the middle road. A VPS (virtual private server) gives me all the control of having my own machine, but in fact it’s an illusion — I still share physical hardware with an unknown number of others.

Tags: , , ,
Posted in Rumblings from the Secret Labs | 2 Comments »

1
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

The Drupal Attitude

November 13th, 2010
Drupal's biggest problem is cultural.

I’ve been doing some geekery with Drupal lately. Drupal is a free, open-source server application that makes it easier to build really complex Web sites. It allows you to create complex data types and establish relationships and do fancy database stuff… without actually touching the database. That’s not too shabby. Drupal is rapidly becoming more popular, but there are a few things standing between Drupal and world domination. At the top of the list is the Drupal Attitude.

I will illustrate with an example. Things will get geeky for a while as I set the stage, then mellow out as I focus on the human interactions between various groups.

From a technical standpoint, Drupal’s biggest flaw is that it sucks when it comes to many-to-many relationships. Imagine I have a data type called “shirt” and another called “color”. It is very easy for me to set up “shirt” so that it can have several colors. So, when I look at a specific shirt in my database I can see that it has red and yellow in it. That’s all pretty straightforward.

The catch comes when I want a list of all shirts with yellow in them. If I had direct control over the database, many-to-many relationships like this are trivial and do not diminish the performance of the server. Drupal has no built-in way to get a list of all shirts with yellow in them.

But wait! Drupal is open source, and better yet has been built to be easy to extend by outside programers. Into this glaring hole in Drupal several folks have stepped forward with modules that solve the problem in a variety of different ways. Some of these methods are clever (one uses the indexes built by the search engine, for instance), but all have trade-offs and weaknesses.

So, you’re a Drupal developer, and you want a list of shirts with yellow in them. Which module do you use? Each module works differently, each requires some installation and fiddling to get working. Then there are the two modules by the same guy that are for similar but different purposes, yet the actual differences are not spelled out very clearly. What would help a lot would be some concrete examples of when to use which.

Now we’re getting closer to the Drupal Attitude. Remember as I rant about this that all the modules I’m evaluating are free, posted by geeks who wanted to contribute to make Drupal better. So, some slack-cutting is in order. BUT…

I had already spent more time than I had available trying to figure out which module to use, when I found a question posted by a guy asking “can I use this module for x”, where x was very similar to what I needed. “Aha!” thought I, “Now we’ll get a definitive answer!” Except that the response to the question was, “In this discussion (the article was about the differences between two modules) we want to focus on generalities, not specific applications. You should download both modules and fiddle with them for a few hours to determine which is right for you.” Or something like that. Notably absent from the answer was a pointer to where specific questions would be answered.

The guy who asked the question responded a bit harshly, pretty much saying, “Would it kill you to just answer my question? I don’t want to spend hours learning something you already know and could tell me in fifteen seconds.”

Well, this is just the sort of uppity user that the Drupal community loves to hate. Several people piled on in defense of the developer who had refused to answer the question. “He’s doing this for free, he’s helping the community, you should be grateful, blah, blah, blah.” None of them deigned to answer the original question either. There is a real, entrenched cadre in the Drupal community that says, “we learned things the hard way, and you should too.” Who needs documentation when you can read the source code?

Let’s step back for a moment and ask ourselves, “Why did the developer give this code back to the Drupal community?” The obvious answer, the one everyone talks about, is that he wants to make things easier for other Drupal users. That is a noble motivation and one I wholeheartedly support. He wants to be useful. Perhaps he just isn’t aware that a huge part of utility of software lies in the documentation. Perhaps he isn’t aware that a few choice examples of what his modules are meant to accomplish would have cost him an hour of his time and improved the acceptance of his work dramatically. He’s a coder, after all, not a marketer or a technical writer.

Even with all that, however, when someone, in the form of a question, contributes to the documentation by providing a specific example, he didn’t answer the question. No light came on that even if that was not the place for the question, then spending five minutes creating an FAQ would have helped the community far more than adding a new feature to his software. So an opportunity to spend just a few seconds and make his contribution to the community better went completely ignored. His supporters congratulated him for not capitulating to the demands of his potential users for more clarity.

Any of them could have stepped up and helped the newbie, probably in ten words or less, but none did. None of them wanted improved documentation. “We had to learn it the hard way, so you should too,” with a side order of “we make lots of money because we’ve figured all this stuff out.” Ladies and gentlemen, the Drupal Attitude.

If the guy posted his module but doesn’t seem interested in making it useful, then why did he post it? Well, he’s certainly getting lots of love from the people who figured out his work the hard way. They can all feel good about how smart they are.

And in the end, should I be thankful this guy shared his work with the rest of us? Actually, no. In my case, the presence of his modules ultimately had negative value. They cost me time, and never getting an answer about which was appropriate for my task, I went with a module developed by someone else.

So, Drupal contributors: If you don’t want to document your module, and you don’t want to answer straightforward questions from people who need to get a job done in limited time, don’t bother posting your fucking module at all. I don’t have time for endless fiddling and I sure as hell don’t have time for the Drupal Attitude.

Tags: , , ,
Posted in Rumblings from the Secret Labs | 9 Comments »

1
Thanks!
ObservationsObservations

Night of the Busy Brain

November 11th, 2010

I couldn’t sleep last night. My brain just wouldn’t quiet down. Kept thinking of stuff. Sometimes those times are productive, however. Last night I thought of:

  • Why my algebraic attempts to calculate the point on a sloped line where the two halves of the shape had the same area were turning out so complex
  • How to make money off PeoplePost (ten years too late).
  • What to call my next version of PeoplePost
  • One of the reasons Tomcat won’t run as a daemon on my machine
  • There was a WordPress thing, too. What was it?
  • Sometimes a weasel with a hammer… um… maybe that wasn’t so productive.

Anyway, eventually I fell asleep. That was about two hours ago. I’m more convinced than ever that alarm clocks are the bane of our civilization.

Tags: ,
Posted in Observations | 1 Comment »

0
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

Things I Learned while Moving to a new Web Host

September 5th, 2010
I expected hiccups, and found a couple.

You probably can’t tell, but this site is now being served by a different host. The reasons I switched were many, but once MMHosting got hacked I decided it was time to move. Then when a particular PHP library was not on their servers (one that allows WordPress to read the date of an uploaded image), I actually did the move.

After quite a bit of looking around in which all the dang hosts started to look the same, I chose iPage. They are not quite the cheapest, but they purchase carbon offsets for wind-generated power. They also had a stronger emphasis on security.

At this writing, I still don’t know if my new host has the needed php library. I’ll be finding out when the dust settles. If not, I can install it in my site myself, I suppose, but let’s keep fingers crossed.

So, I learned a few things, and remembered a few others.

  1. Among sftp clients, RBrowser may have my favorite interface but it’s glacially slow with multiple files.

  2. Without ssh access to my site (none of the big hosting companies allow that on their cheap plans), I had to use phpMyAdmin to copy my databases over. Here’s an interesting bit of trivia: if your data has the phrase ‘drop database’ anywhere in it, phpMyAdmin will stop executing the import right there and then. This is to protect you from SQL injection attacks, where people sneak malicious data into your database that later gets executed as an instruction. ‘Drop database’ can be pretty devastating, so the software simply refuses to complete the import, even if the phrase is safe in the text of a post.

    The way phpMyAdmin is configured at my new host, however, when it stops, it doesn’t say why. It doesn’t even admit that anything went wrong, or indicate in any way that not all the data was imported. This can be inconvenient when you have a bulletin board for a product that has a drag-and-drop database feature. (Now it has a drag-and-drop&nbsp;database.)

  3. You can’t tell Safari not to uncompress zip archives it downloads (that I could find), but the original zip files can be found in the trash.
  4. jerssoft_phpb5 and jerssoft_phpbb5 are not the same thing, no matter how many hours you spend banging your head on them.

So iPage has been great (although their control panel is not completely Safari-friendly when it comes to processing payments). I’ve interacted with them in three different ways now — phone, chat, and a support ticket. Two of the interactions were due to the afore-mentioned payment glitch, and once for technical support trying to get my files copied and the site ready to go before I switched the domain registry. Although I didn’t get the answers I was hoping for, the tech was competent and knew what she was doing.

If I continue to be pleased with iPage, I will provide a link for those looking for a Web host. Because we all need Web hosts these days, don’t we?

Tags: , ,
Posted in Rumblings from the Secret Labs | No Comments »

3
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

Evil Flash Cookies

August 30th, 2010
Otherwise known as LSO's, these things allow Flash to watch you - behind your back. Just how worried should you be?

For a long time now we’ve been aware of browser cookies. These are little bits of data that a Web developer can set on your computer to keep track of your visits, or which ads you’ve seen, and things like that. Cookies are regulated by your browser and you can set up rules to reduce the amount that other people learn about your habits.

Way back in the day the makers of Flash realized that it would be handy to store little bits of information on the user’s computer as well. They developed LSO’s, otherwise known as Flash Cookies, to do that. This site uses an LSO so the banner animation doesn’t run every time you change pages. (See my rant about html.)

Advertisers and less benign sites also use LSO’s, and this has people worried. There are fewer restrictions on what LSO’s can do compared to cookies, and management of these little bits of information is not done through the browser. Many people out there in the Wild Wild World of the Web have said “There’s no way to manage them! aaaah! AAAAAAH!” in mildly hysterical voices, but that is simply not true.

So, just how worried should you be? If you do nothing to manage the cookies on your browser, then you probably don’t need to get too worked up about their somewhat-more-evil cousins. You’re already telling the trackers all they want to know. The potential for outright evil is higher with Flash LSO’s, but not that much.

Still, it’s a good idea to control who leaves what on your computer, and LSO’s are a good place to start. There are two complimentary strategies – control what gets saved, and clean up after.

Control What Gets Saved

First, let’s look at how to keep most of the unwanted items from being saved in the first place. This is done by managing the settings of your Flash Player. You do this through a control panel on the Macromedia Web site. You can access this panel any time by right-clicking any Flash on your page (including the banner of this site) and choosing “Global Settings…”. This control panel is written in Flash and when you make changes it will save your settings to your computer – in an LSO file.

Let’s look at what’s already on your computer. Choose the Web Storage Settings panel:

Flash Web Storage Settings Panel

The Web Storage Settings Panel

I cleaned everything out recently, but you can see that since then I’ve been to two places that put Flash cookies on my machine. Only muddledramblings.com is actually storing anything; www.kfox.com had stored something on my machine, but it has since been cleaned up. Even after you clean up a site’s cookies, Flash will remember you were there, and if you set special rules for that site, it will remember them, too.

So at this point the easiest thing to do is probably to make Flash forget everything and clear out all the cookies stored on your machine. If you’re curious you can go down the list and see what’s there. “Delete all sites” is probably your best bet, however.

Now your Flash Player has totally forgotten where you’ve been. It’s a good time to set rules for how Flash should behave when you encounter a new site. Click the “Global Storage” tab:

Flash Global Storage Settings Panel

The Global Storage Settings Panel

Note: these settings will not affect sites that already have storage allocated.

There are two things you can do to limit who puts stuff on your computer. The first is to move the slider to 0 KB. This will force any flash animation to ask permission before storing something on your machine. If you check Never Ask Again, you have effectively turned off all Flash cookies from everywhere. That’s pretty drastic, and may break some of your favorite sites, though.

The second thing you can do is uncheck “Allow third-party blah blah blah”. That allows the Web site you’re visiting to store stuff, but no one else. For instance, let’s say that on this page I had advertising. This setting would allow only Flash from muddledramblings.com to save stuff, but an ad served from eviladvertisinggiant.com would not be allowed. Basically, only Flash that comes from the domain showing in your browser is allowed to store stuff. That way my site will still work correctly but others won’t be able to track you.

Note that in a few cases, Web sites put their own Flash stuff on different servers (there are good reasons for doing this), and this setting might break those sites. You can turn off the restriction temporarily and allow that site to run, then turn the restriction back on. There is no way that I know of to say set the “Allow third-party…” value for a particular site.

OK, now that you’re keeping most of the drek off your machine, it’s time to tackle the other prong in our battle for privacy: cleaning up unneeded LSO’s.

Periodic Cleanup

In general, benign LSO’s only need to save stuff while you’re on the site. When you go back, it’s not going to harm anything if previous data has been deleted. Any information they do store from session to session might just be snooping. For the most part, then, we can just empty out the stored data and never notice a thing.

What NOT to delete
There are two kinds of LSO’s – those set by the flash animation, and those set by the Flash Player to store information about a site. Deleting the second type can actually undermine your security if you’ve made special restrictions for specific sites using the control panel above. Also, for some few sites (pandora.com, in my case), you want Flash to remember your info between visits. As you decide on a cleanup strategy, keep that stuff in mind. There is one LSO used to store the settings you made from the control panel above, and I strongly recommend that you NOT delete it. Both the cleanup methods I mention below preserve that file by default.

Having said all that, don’t let the decisions stop you from moving forward. In the following techniques just using the defaults will work just fine for almost everyone.

BetterPrivacy
This is by far the best solution — If you use Firefox. Users of Firefox have access to BetterPrivacy, which provides lots of options for which LSO’s to clean up when, controlled from a pretty nice user interface. By default BetterPrivacy leaves the Flash Player preferences alone, but if you make different settings for specific Web sites, BetterPrivacy will delete those unless you tell it not to. If you do put special restrictions or grant special permissions to a site, be sure to protect the settings.sol file for that site.

If you keep your browser open pretty much all the time, you can set BetterPrivacy to clean up the LSO’s periodically.

I don’t use Firefox that much. What am I to do? A Web search will tell you there’s a Mac application called Flush, but don’t bother. As of this writing, it’s completely broken. I didn’t find a good solution for non-Firefox Mac users out there, so I made one. You don’t have to thank me, it’s what I do.

Jer’s LSO Cleanup script for Mac
First I set up a simple cron task that just deleted the folders where LSO’s live. That was too ham-fisted, however, since it also deleted beneficial LSO’s, as mentioned above. Then I wrote a little script. I used Python to write it simply because I’d never used Python before, and it seemed more appropriate than php. You are welcome to use the script as well, but there’s a little fiddling involved. Nothing major, but you’ll be using the terminal.

Right-click to download lsoclean.sh here.

OK, now for the fiddling:

  1. Download the file and put it somewhere that won’t clutter up your life. (I used /usr/local/bin/)
  2. (optional) Edit the file to choose your paranoia level and what sites you don’t want to clean up. The default leaves LSO’s from your own computer (for Flash developers) and from pandora.com
  3. Tell the OS that the file is actually a script that it can run. To do this open Terminal.app and type chmod +x /path/to/lsoclean.py. TIP: if you just type chmod +x (with a space after the “x”), then drag the file from wherever you put it into the terminal window, it will automatically fill in the path. Neat!
  4. (optional) The script can now be activated by typing the full path to the file in the terminal, but that’s not very convenient. Better to set up a way to have the thing run every so often. There are plenty of ways, like using AppleScript (ptui) or iCal (which would have been clever of me), but the simplest (if geekiest) is to set up a cron task. You can use CronniX to avoid editing the crontab file directly. Here’s what I did:
    1. Download CronniX here.
    2. Run it. It’s a little… incomplete. Start by clicking “New”
    3. Choose the “Simple” tab
    4. Check the boxes next to Month, Day of Month, Hour, and Day of Week. Leave Minute unchecked and set to 0.
    5. In the command field, put the full path to the script file. (You can copy it out of the terminal window where you dragged the file before.)
    6. Click Apply, then Save
    7. Quit CronniX

    You have now set up the task to execute the script once an hour.

  5. Test: Visit some sites that use flash, and look in ~/Library/Preferences/Macromedia/Flash Player/#SharedObjects/8JA5UY2L (the last bit is random) to see the .sol files.
  6. After an hour, go back and see that they are gone! Hooray!

CronniX

The CronniX UI when everything is set to go.

That was a quite a bit of fiddling for those not versed in the ways of cron, but now you can forget it ever happened. If you find yourself having to reenter information in a Flash-based Web site and it annoys you, add the domain for that site to the list in the script.

If anyone wants to take this little script create a reduced-fiddling version with automator or whatever, I’d love to provide that for download here.

Conclusion

You’ve just struck a blow against invasive advertisers! Hooray! Now the ads you see will be less focussed on what you are interested in. That’s OK, because it nobody’s damn business what you’re interested in. Now you can carry on with your life like none of this ever happened.

Tags: , ,
Posted in Rumblings from the Secret Labs | 9 Comments »

3
Thanks!
Idle Chit-ChatIdle Chit-Chat

None of Your Damn Business

August 13th, 2010

After reading a post in my buddy’s blog (and the articles that post links to) about National Security Letters I started to get more and more irate. Apparently, our government sends out thousands and thousands of letters to Libraries, Web hosts, and the like, saying, “We’re the government, we’re fighting terrorists, so give us everything you have about this person. Also, you’re not allowed to tell anyone about this, not even your lawyer.” This is not like a search warrant, because there is no judicial oversight.

The FBI’s use of national security letters to get information on Americans without a court order increased from 16,804 in 2007 to 24,744 in 2008. The 2008 requests targeted 7,225 U.S. people.

Read More

Those are all requests for personal information with no warrant, no need for probable cause, and no right to legal counsel even for the people who are not themselves under investigation. I’m not a trained legal scholar, but good lord, this can’t possibly be constitutional.

Well, it’s not like I have anything to hide, but if my ISP got served with one of these letters, would they turn over the information, or would they fight? Would Google protect the emails mouldering in that account that I rarely check? What about my Web hosting provider? I would love to see each entity that has my personal information publicly state that they will not turn over information without due process.

The only way this governmental bullying will be stopped is if everyone agrees not to be intimidated.

On a personal level, I’ve decided to start a policy of encrypting my emails. Not because there’s anything incriminating in there, but because if only secrets are encrypted, then everyone knows where the secrets are. And really, it’s nobody’s business but mine what I put in private correspondence. If everyone encrypted all their messages, the constitutional rape called National Security Letters would be pointless.

Toward that end I have installed Gnu Privacy Guard, which is based on OpenPGP (Pretty Good Privacy), a system which can withstand any attacks feasible at this time (naturally as computing power increases, the encryption must be made ever-more sophisticated).

It takes two to pass a secret message, however. I’m not able to encrypt messages to people who do not also have GPG or PGP installed, and who do not have my public key. The system works with a pair of keys – one I keep secret and another that everyone can see. When the message is encoded using one key, it can be decoded using the other. So if I have your public key, I can encode a message that only you can read.

It’s a bit of a hassle to set up GnuPG (available for Mac, Windows, and Linux), but once you have your key generated and all the pieces in place, it’s pretty transparent to use. My public key is now available on many servers, so once you have the plugin to your email program installed, it’s easy to load.

You can learn more through the link called “Jerry’s public key” on the sidebar in the top section. Please join me in taking the teeth out of National Security Letters and the bullying bureaucrats that use them.

Addendum: Comcast’s privacy policy states that they will not disclose information without a subpoena, warrant, or “other valid legal process”. Then they go on to say they will also disclose information if they think “the disclosure of information is necessary to prevent financial loss“. That means they reserve the right to sell out their customers if they think they could end up incurring legal costs to protect them. Sigh.

Tags: , ,
Posted in Idle Chit-Chat | 11 Comments »

1
Thanks!
ObservationsObservations

Google Calculator

June 1st, 2010

Here’s a little trick I discovered by accident a while back: Go to Google, and put 250-(tan(4pi/6))^2 into the search field. Hey presto! It does the math. This came in handy yesterday as I was evaluating things like

E.x = Origin.x + sqrt(1/((rv2/rh4)(tan2θ) + 1/rv2)

To find the point on an ellipse where the tangent is at a given angle. My little calculator widget was not remotely up to the task.

Tags: ,
Posted in Observations | 6 Comments »

0
Thanks!
Rumblings from the Secret LabsRumblings from the Secret Labs

Cascading Style Sheets (CSS) and PHP

May 30th, 2010
Obvious in retrospect.

Often when dealing with Cascading Style sheets, or CSS, I find myself wishing that the CSS mechanism included variables. This is especially true when dealing with colors, since you want the same color applied to lots of different things. It can be a real pain to go back through an old style sheet and find the code for the color you want. I was quietly surprised that no one making up how CSS worked had addressed something like this.

Then, a while back I was giving a buddy of mine a few exercises to introduce him to the exciting world of Web programming, touching on CSS, HTML, PHP and MySQL. I gave him pretty much no guidance; I just thought up plans that would introduce him to the concepts and gave him a list of my favorite references. (I’ll be posting those exercises here in the nearish future.)

Anyway, without me to tell him how to do things, he went and dug around and one of the first style sheets he sent me for evaluation had a .php extension rather than .css.

Bingo! Once you see it in action, it’s obvious. PHP can be used to generate CSS files just as easily as it can be used to generate HTML files. Now my style sheets can change based on external conditions or can simply define a set of colors that all the style definitions share. Why did it take me so long to figure this out? It seems like this technique should be a lot more common than it is.

Here’s a quick code snippet for those who want to try it for themselves:

<?php
	header('Content-Type: text/css');
 
	$header_back_color = '#dddddd';
?>
 
#corner_table th {
	background-color:<?php echo $header_back_color ?>;
	text-align: center;
}

A couple of notes: the <?php MUST be the very first thing in the file. No empty lines, no spaces. The reason is that the next line, with the header() function, has to be called before the server sends any page content. (Once the server starts sending content back to the browser, it’s too late to be fiddling with the headers. Any whitespace outside the <?php tag will be considered content.) The header line is necessary because you need to tell your browser that what you are sending really is a css file.

In the <head> of the html file, you call the style sheet just like normal, but of course the file you fetch will have a php extension:

<link rel="stylesheet"
      href="http://yourdomain.com/css-tables.php"
      type="text/css"
      media="screen" />

That’s all there is to it. Why have I not done this with every css file?

Tags: , , ,
Posted in Rumblings from the Secret Labs | No Comments »

« Older Episodes