Redfin Developers\' Blog » Dan Fabulich

How to Set Up Hot Code Replacement with Tomcat and Eclipse

Dan Fabulich — Wed, 30 Sep 2009 19:28:32 +0000

This blog post will guide you through setting up Tomcat hot code replacement (also called hotswap debugging) in Eclipse.

What Is “Hot Code Replace”?
What’s the Catch?
What About JavaRebel?
Configuring Your Web Application in Eclipse
1. Download Eclipse “JEE” Edition
2. Switch to the “Java EE” Perspective
3. Configure Your WAR Project
4. Create a New Server
5. Magic Setting: Disable “Auto Reloading” on Each Project in the Server
6. Performance Tip: Adjust Memory Settings
7. Start Your Tomcat Server in Debug Mode
Why Disable Auto Reloading?
Disable Auto Reloading but Enable Auto Publishing
Finding the tmp0 Fake Tomcat Directory
Exorcising the tmp0 Directory
Troubleshooting: What Do I Do If I Still Can’t Get It to Work?

What Is “Hot Code Replace”?

“Hot Code Replace” (HCR) lets you make modifications to a Java class and see the effect immediately in a running JVM, without restarting your application. HCR is part of the Java Platform Debugger Architecture (JPDA) and is available on all modern JVMs.

Consider this ordinary application:

public class Sample {
  public static void main(String[] args) {
    String foo = "unchangeable";
    foo += blah();
    System.out.println(foo);
  }

  public static String blah() {
    String bar = "bar";
    bar += "blah";
    return bar;
  }

}

If you debug this class in Eclipse, you can make changes to it, on the fly, without restarting the JVM. For example, try setting a breakpoint on the second line of blah(). Next, change the literal blah to quz. Save the file and the program will continue running with the new code.

You can do this with Tomcat web applications in Eclipse, but it’s a lot trickier.

What’s the Catch?

There are a few limitations when using hot code replace. You can’t use JPDA HCR to change the signature of a class (add/remove methods or fields) or to add new classes on the fly. Additionally, some method calls (”stack frames”) can’t be modified, including the main method or any method invoked via reflection, that is, by using java.lang.reflect.Method.invoke().

Here’s what happens if you try to replace the unchangeable variable in the main method of Sample.java above.

What About JavaRebel?

JavaRebel is a hot code replacement system that’s a little better than JPDA HCR. (Maybe a lot better.)

With JavaRebel you can add/remove methods and classes without restarting Tomcat. However, JavaRebel costs $149 per developer per year, so it may or may not be worthwhile for you.

Configuring Your Web Application in Eclipse

Download Eclipse “JEE” edition

Most developers already use this, since it’s the first option available on the Eclipse download page, but if you’re using “Eclipse IDE for Java Developers” (92MB), you’ll need to go back and download “Eclipse IDE for Jave EE Developers” (189MB). It’s worth it, I promise!

Note: The difference between the regular Java IDE and the Java EE IDE is that the JEE edition comes with the Eclipse “Web Tools Project” (WTP), which includes “Web Server Tools” (WST). The terms are sometimes used interchangeably; keep an eye out for this if you need to search for them.
Switch to the “Java EE” Perspective
Make sure you’re in the “Java EE” perspective, not the “Java” perspective. If it’s not in the upper-right corner of your Eclipse window, you might need to enable it manually (Window menu -> Open Perspective -> Other…). If “Java EE” doesn’t appear on this list, you’ve probably downloaded the wrong package of Eclipse; go back and download the Java EE version.
Configure Your WAR Project
From scratch: From the New menu, select “Dynamic Web Project”. Configure your source and output folders, as well as your “Content directory”, which will contain your JSPs, your WEB-INF directory, etc.

Via Maven: Use Maven to create a WAR project. For example:

mvn archetype:create -DarchetypeArtifactId=maven-archetype-webapp -DartifactId=YOURNAMEHERE -DgroupId=YOURNAMEHERE

Modify your pom.xml to include an explicit reference to maven-eclipse-plugin, like this:
```
    
    
        
        
            maven-eclipse-plugin
            
                2.0
            
        
    
```
Now generate an Eclipse project from the command line, like this:

mvn eclipse:eclipse

Here’s an example Maven project you can use. Just download it, extract it, and run mvn eclipse:eclipse to generate your Eclipse project. (If this is your first time using Maven with Eclipse, you’ll also need to add an M2_REPO classpath variable in your Eclipse workspace preferences that points to your Maven repository, typically $HOME/.m2/repository or %USERPROFILE%\.m2\repository.)
Create a New Server
From the New menu, select Other… -> Server -> Server. For your server type, expand the “Apache” folder and select the version of Tomcat you intend to use. Choose “Next” and then specify the path to your Tomcat installation directory, e.g. c:\tools\tomcat-6.0. Add your web project as a “resource” to this server.
Magic Setting: Disable “Auto Reloading” on Each Project in the Server
You now have a weird pseudo-project called “Servers” in your Project Explorer. In the explorer, your server looks like a folder called something like “Tomcat v6.0 Server at localhost-config” …but that’s not what you want. You need to interact with your server using the “Servers” tab. (Eclipse calls these tabs “Views,” but everybody else just calls them “tabs.”)

The “Servers” tab should be exposed by default, but in case it isn’t, you can expose it via Window -> Show View -> Servers. From there you can double-click on your server to configure it.

Note that the configuration panel for your server has two tabs, “Overview” and “Modules”, down at the bottom of the window. Click on the “Modules” tab to bring up the list of projects associated with the server.

Select your project in the list and click on “Edit”. You’ll see the magic secret checkbox: “Auto reloading enabled”. It’s checked by default. For the love of Pete, uncheck it!

(It’s interesting to note that JavaRebel also requires disabling auto reloading to work properly in Eclipse.)
Performance Tip: Adjust Memory Settings
Before you start up your server, I strongly recommend adjusting your server’s -Xmx memory settings; otherwise, it will constrain itself to the default value, 64 MB, which is just not enough!

Double-click on your server in the “Servers” tab and switch to the “Overview” tab. Click on the “Open launch configuration” link. Switch to the Arguments tab; there you can add relevant memory settings to the “VM Arguments” section. For example, you might add -Xmx512m to the end of the existing arguments.
Start Your Tomcat Server in Debug Mode
Now you can right-click on the Server in your Servers tab and choose “Debug”. Changes you make to your JSPs or Java classes will be instantly hotswapped into your running WAR.

Why Disable Auto Reloading?

Auto reloading is a feature of Tomcat that allows you to replace Java classes at runtime without using JPDA. In this mode, Tomcat uses Java classloaders to try to unload classes and reload them; whenever it reloads, it also tries to reinitialize your application, re-launching any servlets that are marked load-on-startup in your web.xml file.

As a result, Tomcat auto reloading may not save you any time at all if your webapp has a lot of startup code. For example, if your load-on-startup code needs to warm up Hibernate database caches, Spring/Guice dependency injection configuration, etc., it may take almost as long to restart your webapp as it does to restart Tomcat.

Worse, an app that has been auto reloaded can behave strangely, and can quickly run out of PermGen memory due to frequent unloading/reloading of classes. When this happens, restarting Tomcat typically fixes the problem. If you spend even five minutes investigating a weird auto reloading problem, you’ve just wasted all the time you were hoping to save by avoiding a restart. (Not to mention your stress and frustration!)

By disabling auto reloading and using JPDA hot code replace instead, you get a more reliable code replacement system.

Disable Auto Reloading but Enable Auto Publishing

In the screenshot above you can see how to disable auto reloading on the “Modules” tab of the Tomcat server; auto reloading is bad for JPDA debugging. But there’s another setting called “Automatically publish when resources change” on the “Overview” tab of the Tomcat server. It’s hidden by default, collapsed under the “Publishing” section. You can see it if you expand that section; you want to make sure auto publishing is enabled while auto reloading is disabled.

To understand the difference between auto publishing and auto reloading, we’ll have to go into how exactly Eclipse WTP works. When you create a “Server” in Eclipse, the IDE creates a fake Tomcat directory, complete with the conf, logs, temp, webapps and work directories. When you configured the server, you told Eclipse where to find Tomcat, but it doesn’t use any of your settings files or any data from your webapps directory. Instead, Eclipse launches Tomcat with special command-line arguments, indicating where to find the fake Tomcat directory.

“Publishing” means to populate the fake Tomcat directory with all of your code. It copies your JSPs, JARs up your Java, regenerates settings files, etc.

If you turn off auto publishing, then you have to right-click on your Server and “Publish” your changes manually every time you save your Java code. Additionally, auto publishing allows Tomcat to reload JSPs automatically, regardless of whether you use JPDA or not.

Finding the `tmp0` Fake Tomcat Directory

Sometimes it can be helpful to look inside the fake Tomcat directory and see what’s going on in there. Eclipse tells you where it put the Tomcat directory in the “Server Locations” section of your “Tomcat” server configuration panel. (Double-click on your Server in the “Servers” tab to open the configuration panel.) Typically, Eclipse says that your server is in .metadata/.plugins/org.eclipse.wst.server.core/tmp0; for this reason I typically call it the tmp0 directory (pronounced “tempo”).

The .metadata folder is inside your Eclipse workspace directory. (You can find your Eclipse workspace directory by going to File -> Switch Workspace; the default value is your current workspace directory.) In the worst case, you can always just search your hard drive for tmp0. It’s there somewhere!

Inside, you can see all the folders Eclipse has created. Check out the generated server.xml file in tmp0/conf. Examine generated .java files in tmp0/work. Your tmp0/webapps directory is probably empty; Eclipse has probably generated your webapp in wtpwebapps.

Exorcising the `tmp0` Directory

Unfortunately, sometimes Eclipse gets a little confused about what to put in your WAR file, and you need to perform various stages of exorcism depending on how badly your tmp0 directory is messed up.

Try republishing your tmp0 directory. Open the “Servers” tab, right-click on your server and select “Clean…” (not “Clean Tomcat Work Directory…”). Then select “Publish.” That should completely rebuild your tmp0 directory.
Try restarting Eclipse. This works more often than I’d like to admit.
Try completely deleting and recreating your server. Follow this ritual:
1. Open the “Servers” tab, right-click on the server and select “Delete”.
2. Make sure “Delete unused server configuration(s)” is checked, then click OK.
3. Look at your “Servers” pseudo-project; make sure the folder for your server is gone. If it isn’t, right-click on it and Delete it.
4. Quit Eclipse.
5. Go find your tmp0 directory (if it’s still present) and delete it from your file system.
6. Launch Eclipse and recreate your server from scratch.
Try creating a new workspace. File -> Switch Workspaces: specify an empty directory. Create your server from scratch.

Troubleshooting: What Do I Do If I Still Can’t Get It to Work?

My project works in regular Tomcat, but doesn’t work in Tomcat under Eclipse
Try using Eclipse to generate a WAR file for Tomcat. Right-click on your web project and select Export -> WAR file, and install it in Tomcat by dropping the exported WAR into your Tomcat webapps directory.

If the exported WAR file doesn’t work, then you now have two WARs: one working WAR generated by your build script, and one non-working WAR generated by Eclipse. WAR files are just zips; extract them both, find the difference, and fix it! Right-click on your web project and select “Properties”. The problem is somewhere in here.

On the other hand, if the exported WAR file does work, then you know that the problem has to do with the way Eclipse is launching Tomcat. Find your tmp0 directory (described above) and poke around. Does everything look OK in there? Be sure to check your server.xml file, as well as your webapp itself in wtpwebapps. Make sure to note your WEB-INF/lib directory, typically in tmp0/wtpwebapps/YOURAPP/WEB-INF/lib.
Tomcat is throwing NoClassDefFoundError or ClassNotFoundException
First, double-check whether this problem happens in regular Tomcat. See My project works in regular Tomcat, but doesn’t work in Tomcat under Eclipse above.

If this problem occurs in the exported WAR file under regular Tomcat, then your webapp is probably missing JARs. See My exported WAR is missing JARs below.

If the exported WAR works but your webapp is still broken under Tomcat, you may need to perform an exorcism. (See Exorcising the tmp0 Directory above.) If this happens to you often, double-check that you haven’t accidentally disabled auto publishing. (See Disable Auto Reloading but Enable Auto Publishing above.)
My exported WAR is missing JARs
Right-click on your web project and select “Properties.” The problem is somewhere in here. Make sure you see your JAR listed under “Java Build Path” in the Properties dialog.

Beware: not every JAR in “Java Build Path” gets exported to the WAR. The list of JARs for the WAR is under “Java EE Module Dependencies.” If a JAR/project is unchecked on that list, it won’t appear in your WAR file.
My tmp0 directory is missing an important configuration file
Eclipse will publish files that it finds in the “Tomcat” folder of the “Servers” pseudo-project to your tmp0/conf directory; if you’re missing files, you can add them here.
My server.xml file is messed up
That file is copied from the “Tomcat” folder in your “Servers” pseudo-project to your tmp0/conf directory. But note that the server.xml file is at least partially autogenerated by Eclipse. If you make direct changes to the file, Eclipse will do its best to try to incorporate your changes, but it often gets confused and does the wrong thing. When possible, it’s better to find the appropriate Eclipse settings and change them there instead of modifying the server.xml file directly.

Note that one of the most common problems with server.xml is an incorrect path attribute on your webapp’s element, causing your webapp to appear on a non-standard URL. See the following question about 404 errors for more details about this problem.
Tomcat is giving me strange 404 errors
First, double-check whether this problem happens in regular Tomcat. (See My project works in regular Tomcat, but doesn’t work in Tomcat under Eclipse) If it happens in regular Tomcat too, then it’s probably a bug in your code.

If the problem only happens in Eclipse, then it’s probably a server.xml configuration problem. Check your tmp0/conf/server.xml file’s element; check the path attribute. The path attribute indicates the virtual directory of your webapp. For example, if your Context/path is “examplePath”, then your webapp will appear at http://localhost:8080/examplePath. If it’s misconfigured, your webapp may not be available at the URL you expect.

The path attribute is auto-generated based on settings in the properties of your WAR project. Right-click on your web project, select “Properties” and go to the “Web Project Settings” section. There’s only one setting here; it’s called “Context root”. Specify the context you intend to use here. If you want your project to appear in the root directory, you’ll need to put / as your context root (since you aren’t allowed to leave it blank).
Tomcat times out when starting under Eclipse (”Server [...] was unable to start within 45 seconds”)
The Eclipse developers, in their infinite wisdom, have added a timeout to Tomcat startup. If Tomcat doesn’t declare a successful startup in 45 seconds, it kills Tomcat automatically. (Gee, thanks, guys!)

You can increase that timeout. Open the “Servers” tab and double-clicking on your server to open the server configuration panel. Make sure the panel’s “Overview” tab is selected. Expand the “Timeouts” section and increase the Start timeout to something reasonable for your server.
I had Tomcat working under Eclipse, but now it’s broken and I can’t figure out why
You may need to perform an exorcism. (See Exorcising the tmp0 Directory above.)
My web project starts up fine, but when I save .java files in Eclipse, it doesn’t take effect until I restart
- Did you make sure to launch the server in Debug mode, as opposed to Run mode? JPDA only works when you Debug the server.
- Is your server configured to auto publish? (See Disable auto reloading but Enable auto publishing above.)
- Did you change something that broke JPDA? (See What’s the catch? above.) If you make large changes to your classes, JPDA may be unable to replace the code; if you choose to “Continue” past that point, further changes will have no effect.
My web project starts up fine, but when I save .jsp files in Eclipse, it doesn’t take effect until I restart
This is typically due to disabled auto publishing. Double-check that your server is configured to auto publish. (See Disable auto reloading but Enable auto publishing above.)

If that doesn’t work, examine your tmp0 directory to make sure Tomcat is using the correct JSP. It should automatically begin consuming new JSPs as they are installed in the tmp0/wtpwebapps directory.
Whenever I save a .java file in Eclipse, Tomcat restarts
This is typically due to Tomcat auto reloading. Double-check that you correctly disabled auto reloading. (See Magic Setting above.)
Tomcat in Eclipse is much slower than regular Tomcat
Try increasing your memory settings as described above, if you haven’t already.

Try running Tomcat in “Run” mode (as opposed to “Debug”) mode. If that fixes the problem, then there may be nothing you can do about it. JPDA does have some overhead; you can turn it off, but while you’ve turned it off you won’t have access to hot code replacement.

Installing Beta Builds on iPhone

Dan Fabulich — Wed, 30 Sep 2009 01:16:02 +0000

Probably the hardest part of learning to code an iPhone app is figuring out how to get your app installed on a phone. Even after you’ve installed your app on your personal development phone, installing it on other people’s phones for beta testing is especially tricky.

Here’s the rough outline:

Create an “Ad Hoc” Provisioning Profile (a .mobileprovision file) for beta testing.
Find yourself a friendly beta tester.
Each beta tester needs to give you the Universal Device Identifier (UDID) of his/her device. (This number is not easily visible to users.)
Submit the UDID as a named “Device” to Apple.
Attach the device to your Provisioning Profile. Apple will generate (or regenerate) a .mobileprovision file for you to download. Always use the latest .mobileprovision file.
Build (or rebuild) your software in Xcode using the latest Provisioning Profile.
Zip up your application.
Deliver your finished .app together with the .mobileprovision to your beta tester.
The beta tester installs your provisioning profile, either by dragging it into iTunes or by using the iPhone Configuration Utility (iPCU).
Finally, the beta tester installs your app, using either iTunes or iPCU.

That’s a lot of steps; there’s a lot of room for things to go wrong. Here’s a few gotchas we encountered on our way to releasing the Redfin iPhone app.

Some Zip Tools Don’t Work; Use `ditto`

Zipping your .app file is surprisingly tricky business. For example, if you use Apache Ant’s task to create your zipfile, it will ignore the UNIX permissions of the files it zips. When unsuspecting beta testers try to extract the app, they’ll find that it installs successfully under Windows (which doesn’t honor UNIX permissions) but if they install from OS X, the app will immediately crash on startup, with no crash log.

FYI, if you observe this permissions crash via the device console logs, it looks a little like this:

Tue Sep 29 17:47:04 unknown com.apple.launchd[1] : (UIKitApplication:com.redfin.redfin[0x43d3]) posix_spawn("/var/mobile/Applications/2B5E0CE3-362F-4FF0-80A2-C45DE3923C86/Redfin.app/Redfin", ...): Permission denied
Tue Sep 29 17:47:04 unknown com.apple.launchd[1] : (UIKitApplication:com.redfin.redfin[0x43d3]) Exited with exit code: 1
Tue Sep 29 17:47:04 unknown SpringBoard[24] : Failed to spawn Redfin. Unable to obtain a task name port right for pid 13432: (os/kern) failure
Tue Sep 29 17:47:04 unknown com.apple.launchd[1] : (UIKitApplication:com.redfin.redfin[0x43d3]) Throttling respawn: Will start in 2147483647 seconds
Tue Sep 29 17:47:04 unknown SpringBoard[24] : Application 'Redfin' exited abnormally with exit status 1

(Please excuse the scrolling; WordPress automatically screws up this log file entry.)

Even the standard command-line zip tool from Info-ZIP that comes with OS X doesn’t quite do the job. Insidiously, zip archives created with zip work fine for beta testing, but the App Store will reject them, claiming that the app signature is invalid.

You can check the validity of your app signature with codesign -vvvv YourAppName.app. (Yes, all four “v”s are necessary.) Valid apps should look like this:

$ codesign -vvvv Redfin.app
Redfin.app: valid on disk
Redfin.app: satisfies its Designated Requirement

It is very easy to disturb this signature. For example, if you copy the app with cp -r Redfin.app /tmp and then check the signature, you’ll get a message like this:

$ codesign -vvvv Redfin.app
Redfin.app: a sealed resource is missing or invalid
/private/tmp/Redfin.app/CodeResources: resource added

Unfortunately, cp isn’t the only command-line tool that can invalidate the signature. If you create your zip archive using zip -r yourappname YourAppName.app and then unzip it (either with Finder or with unzip), you’ll invalidate the signature.

You don’t get this problem if you use Finder to create the zip archive, by putting YourAppName.app in a folder called YourAppName, and right-clicking on the YourAppName folder and selecting the “Compress” option. To duplicate this functionality automatically, you’ll have to use ditto, which comes with OS X. ditto -c -k YourAppName YourAppName.zip should do the trick.

Don’t Use iTunes; Use the iPhone Configuration Utility

Some people are able to successfully install apps and provisioning profiles using iTunes; if iTunes works for you, great. But a lot of our beta testers weren’t so lucky. iTunes would often fail mysteriously, sometimes with no useful error messages.

The iPhone Configuration Utility (iPCU) allows users to generate log files with error messages; these error messages are critical for diagnosing problems with application installation. More generally, iPCU includes better error messages than iTunes all around; in many cases, switching to iPCU gave our beta testers enough information to solve the problem without the aid of developers.

There’s another good reason to use iPCU instead of iTunes for app installation: you can sync an iPhone with only one iTunes machine. If you want to install beta apps using any other machine, you have to use the iPhone Configuration Utility. In most cases, you can workaround even the trickiest app installation problems just by using iPCU to install from another machine.

To install an app using iPCU:

Download and install the iPhone Configuration Utility from Apple.
Connect your iPhone/iPod to your computer via USB cable.
Close iTunes, if it is open, before launching iPCU
Launch iPCU. You should see your device appear in the “Devices” section on the left side of the window.
Add the provisioning profile (mobile provision) to the iPCU Library: Click on “Provisioning Profiles”, then drag the .mobileprovision file into the Provisioning Profiles pane.
Add the application to the iPCU Library: Click on “Applications”, then drag the app into the Applications Pane. On Windows, the app will be a folder, named something like YourAppName.app. On Mac OS X, the app will be a single application file.
Access your device: Click on your device in the list of devices on the left. You should see five tabs appear in the main window: “Summary”, “Configuration Profiles”, “Provisioning Profiles”, “Applications”, and “Console”.
Install the Provisioning Profile: Click on the “Provisioning Profiles” tab of your device. You should see your Ad Hoc Provisioning Profile in the list of profiles. In the “Install” column you should see either a button that says “Install” or a button that says “Remove” (if you’ve installed this profile previously).
Install the app: Click on your device’s “Applications” tab. You should see your app in the list of applications. In the “Install” column you should see either a button that says “Install” or a button that says “Uninstall” (if you’ve already installed the app).

That’s all there is to it, assuming that nothing goes wrong.

Error Messages: What Could Go Wrong?

iPhone Config Utility: “Could not start session with device. Error: kAMDSessionActiveError”
Your device has turned off / fallen asleep. Push the power button, slide to unlock, and try it again.
iPhone Config Utility: I tried to install, and I got an error like “Could not install application on device. Error: -402620395.”.
This is almost certainly because the .mobileprovision didn’t install correctly. Clear out all provisioning profiles from your phone and from the iPCU Library (following directions above), drag your .mobileprovision file into the Library / Provisioning Profiles section, then click on Devices / Your Device / Provisioning Profiles. Make sure you see only one profile on the list, and make sure it’s installed.
iPhone Config Utility: I tried to install, and I got an error like “Could not install application on device. Error: -402620393.”.
We saw this error several times but never figured out the root cause. Installing from another machine worked around the problem.
iPhone Config Utility: I tried to install, and I got an error like “Could not transfer application to device. Error: kAMDUndefinedError.”.
A number of our beta users had this error, but we never deduced the root cause. In at least one case, re-installing the latest version of iPCU resolved the problem. In all known cases, installing from another machine worked around the problem.
iTunes: I tried to sync, and I got an error like “The application ‘YourAppName’ was not installed on the iPhone because an unknown error occurred (0xE8008017).”
This is the same as error -402620393 above. (Note that E8008017 is hexadecimal for the signed integer -402620393.) We saw this error several times but never figured out the root cause. Installing from another machine without iTunes (using iPCU) worked around the problem.
iTunes: I tried to sync, and I got an error like “The application ‘YourAppName’ was not installed on the iPhone because an unknown error occurred (0xE8008015).”
This is the same as error -402620395 above. (Note that E8008015 is hexadecimal for the signed integer -402620395.)This is almost certainly because the .mobileprovision didn’t install correctly. Try dragging the .mobileprovision again to Applications pane in iTunes and syncing. It might also help to delete all of your existing mobile provisions (see Clear Out the Phone Using the Phone below).

Troubleshooting

Still not working? Try this.

Using iTunes? Try the iPhone Configuration Utility.
Try installing from another machine. It’s often especially helpful to try installing from a Mac if you’re failing on Windows, or to try installing from a Windows box if you’re failing on a Mac. (If you’re on Windows and you don’t have access to a Mac, try switching to a Windows machine that does not have iTunes installed.)
Close iTunes. The iPhone Configuration Utility doesn’t work so well if iTunes is open. Note that iTunes auto-launches by default when you plug in an iPhone/iPod.
Try restarting your phone. Press and hold the power button at the top of the phone, slide to power off. Wait until the screen turns off, then press and hold the power button again to start it up again. No, seriously. This actually works, more often than you’d think.
Clear out the app and provisioning profiles correctly, as described below.
Delete your entire iPCU Library, as described below.
Using iPhone Configuration Utility? Try iTunes.
Still not working? Examine the iPCU Log. Directions below.

Clear Out the Phone Using the Phone

Both iTunes and iPCU provide the ability to remove apps and provisioning profiles from the phone, but they don’t work very reliably, especially when you have multiple versions of the same app/profile bouncing around.

The most reliable way to remove an app from the iPhone is to do it the normal way: on the Home screen, press and hold your finger on the app; an X will appear. Tap on the X to delete the app.

You can remove provisioning profiles using the iPhone “Settings” app. In the “General” section, scroll down to the “Profiles” section, select it, tap on the old provisioning profile, and tap “Remove”. (Ordinarily you shouldn’t need to delete provisioning profiles, but the best way to be sure that you’ve installed a .mobileprovision file is to remove all profiles and then install just the provision you need.)

Deleting the iPhone Configuration Utility Library

Removing the app and provisioning profile from your iPCU library should just be a simple matter of deleting them in iPCU. But if something goes really wrong, you may have to purge the iPCU library. (In at least one recorded case, purging the iPCU library not only fixed iPCU, but also fixed installing the app via iTunes.)

On OSX, the iPCU library is stored in your Home directory, in ./Library/MobileDevice. You can delete the entire folder.

On Windows, the library is stored in your local application data folder. To find your local app data folder:

On Vista, press the Windows key, type %localappdata% in the search box, and press Enter.
On XP, press the Windows key, select “Run” and type %userprofile%\Local Settings\Application Data

Go to the “Apple Computer” folder. (There may also be an “Apple” folder and an “Apple_Inc” folder; ignore those.) You should see an “iPhone Configuration Utility” folder here and a “MobileDevice” folder here; delete both of them.

Extracting the iPCU Log

To access your device console, click on your device in the “Devices” section on the left pane of iPCU. There is a “Console” tab on the far right side. Click on that and you should start seeing console messages. (It sometimes takes a few seconds for the messages to appear.) You can click “Save Log As…” to generate a text file that you can send out as an attachment.

Maven Reactor Tricks

Dan Fabulich — Mon, 28 Sep 2009 17:56:01 +0000

Not many people know this, but you can use Maven to resume a failed build from the middle, or to build just a subset of projects in a multi-module (a.k.a. “reactor”) build.

Example Reactor Project

Consider this complex multi-module reactor build:

my-root-project
|-- pom.xml
|-- barBusinessLogic
|   `-- pom.xml
|-- bazDataAccess
|   `-- pom.xml
|-- quz
|   |-- pom.xml
|   |-- quzAdditionalLogic
|   |   `-- pom.xml
|   `-- quzUI
|       `-- pom.xml
`-- fooUI
    `-- pom.xml

Suppose project fooUI depends on project barBusinessLogic, which depends on project bazDataAccess.

fooUI --> barBusinessLogic --> bazDataAccess

Furthermore, quzUI depends on quzAdditionalLogic, which depends on barBusinessLogic.

quzUI --> quzAdditionalLogic --> barBusinessLogic --> bazDataAccess

Ordinarily, when you run mvn install from my-root-project, you’ll build the projects in this order:

my-root-project (parent project)
bazDataAccess
barBusinessLogic
fooUI
quz (parent project)
quzAdditionalLogic
quzUI

Resuming the Build with `--resume-from`

Suppose you’re working on your code and you attempt to run mvn install from my-root-project, but you encounter a test failure in fooUI. You make additional changes to barBusinessLogic without changing bazDataAccess; you know that bazDataAccess is fine, so there’s no need to rebuild/test it. You can then use the --resume-from argument, like this:

mvn install --resume-from=fooUI

That will skip over bazDataAccess and barBusinessLogic and pick up the build where you left off in fooUI. If fooUI succeeds, it will go on to build quzAdditionalLogic and quzUI.

Specify a Subset of Projects with –projects

Suppose you’ve made some changes to fooUI and bazDataAccess and would like to rebuild just those two projects. You can use the --projects argument, like this:

mvn install --projects fooUI,bazDataAccess

That will automatically build just those two projects, saving you the trouble of running Maven in each directory separately.

Making `fooUI` Without Building `quz` Using `--also-make`

Suppose you’re a developer working on fooUI; you don’t want to work on quz right now, but just want to get a working build of fooUI. You can use --also-make, like this:

mvn install --projects fooUI --also-make

--also-make will examine fooUI and walk down its dependency tree, finding all of the projects that it needs to build. In this case, it will automatically build bazDataAccess, barBusinessLogic and fooUI without building quz.

Changing `barBusinessLogic` and Verifying You Didn’t Break Anything Using `--also-make-dependents`

Suppose you’ve made a change to barBusinessLogic; you want to make sure you didn’t break any of the projects that depend on you. You also want to avoid rebuilding/testing projects that you know you haven’t changed. In this case, you want to avoid building bazDataAccess. You can use --also-make-dependents, like this:

mvn install --projects barBusinessLogic --also-make-dependents

--also-make-dependents will examine all of the projects in your reactor to find projects that depend on barBusinessLogic; it will automatically build those and anything that depends on them.

Resuming a `make` Build

When you use --also-make or --also-make-dependents, you run a subset of projects, but that doesn’t mean stuff won’t fail halfway through the build. You can resume an --also-make build from the project that stopped the build by using --resume-from together with --also-make, like this:

mvn install --projects quz/quzUI --also-make --resume-from barBusinessLogic

At Redfin, our Maven reactor has 86 projects; avoiding unnecessary rebuilds is essential to our productivity!

Announcing SitemapGen4j 1.0

Dan Fabulich — Tue, 03 Feb 2009 21:15:39 +0000

Redfin is happy to announce SitemapGen4j 1.0. SitemapGen4j is a library to generate XML sitemaps in Java.

Download SitemapGen4j 1.0

What’s an XML sitemap?

Quoting from sitemaps.org:

Sitemaps are an easy way for webmasters to inform search engines about pages on their sites that are available for crawling. In its simplest form, a Sitemap is an XML file that lists URLs for a site along with additional metadata about each URL (when it was last updated, how often it usually changes, and how important it is, relative to other URLs in the site) so that search engines can more intelligently crawl the site.

Web crawlers usually discover pages from links within the site and from other sites. Sitemaps supplement this data to allow crawlers that support Sitemaps to pick up all URLs in the Sitemap and learn about those URLs using the associated metadata. Using the Sitemap protocol does not guarantee that web pages are included in search engines, but provides hints for web crawlers to do a better job of crawling your site.

Sitemap 0.90 is offered under the terms of the Attribution-ShareAlike Creative Commons License and has wide adoption, including support from Google, Yahoo!, and Microsoft.

Getting started

The easiest way to get started is to just use the WebSitemapGenerator class, like this:

WebSitemapGenerator wsg = new WebSitemapGenerator("http://www.example.com", myDir);
wsg.addUrl("http://www.example.com/index.html"); // repeat multiple times
wsg.write();

Configuring options

But there are a lot of nifty options available for URLs and for the generator as a whole. To configure the generator, use a builder:

WebSitemapGenerator wsg = WebSitemapGenerator.builder("http://www.example.com", myDir)
    .gzip(true).build(); // enable gzipped output
wsg.addUrl("http://www.example.com/index.html");
wsg.write();

To configure the URLs, construct a WebSitemapUrl with WebSitemapUrl.Options.

WebSitemapGenerator wsg = new WebSitemapGenerator("http://www.example.com", myDir);
WebSitemapUrl url = new WebSitemapUrl.Options("http://www.example.com/index.html")
    .lastMod(new Date()).priority(1.0).changeFreq(ChangeFreq.HOURLY).build();
// this will configure the URL with lastmod=now, priority=1.0, changefreq=hourly
wsg.addUrl(url);
wsg.write();

Configuring the date format

One important configuration option for the sitemap generator is the date format. The W3C datetime standard allows you to choose the precision of your datetime (anything from just specifying the year like “1997″ to specifying the fraction of the second like “1997-07-16T19:20:30.45+01:00″); if you don’t specify one, we’ll try to guess which one you want, and we’ll use the default timezone of the local machine, which might not be what you prefer.

// Use DAY pattern (2009-02-07), Greenwich Mean Time timezone
W3CDateFormat dateFormat = new W3CDateFormat(Pattern.DAY);
dateFormat.setTimeZone(TimeZone.getTimeZone("GMT"));
WebSitemapGenerator wsg = WebSitemapGenerator.builder("http://www.example.com", myDir)
    .dateFormat(dateFormat).build(); // actually use the configured dateFormat
wsg.addUrl("http://www.example.com/index.html");
wsg.write();

Lots of URLs: a sitemap index file

One sitemap can contain a maximum of 50,000 URLs. (Some sitemaps, like Google News sitemaps, can contain only 1,000 URLs.) If you need to put more URLs than that in a sitemap, you’ll have to use a sitemap index file. Fortunately, WebSitemapGenerator can manage the whole thing for you.

WebSitemapGenerator wsg = new WebSitemapGenerator("http://www.example.com", myDir);
for (int i = 0; i < 60000; i++) wsg.addUrl("http://www.example.com/doc"+i+".html");
wsg.write();
wsg.writeSitemapsWithIndex(); // generate the sitemap_index.xml

That will generate two sitemaps for 60K URLs: sitemap1.xml (with 50K urls) and sitemap2.xml (with the remaining 10K), and then generate a sitemap_index.xml file describing the two.

It’s also possible to carefully organize your sub-sitemaps. For example, it’s recommended to group URLs with the same changeFreq together (have one sitemap for changeFreq “daily” and another for changeFreq “yearly”), so you can modify the lastMod of the daily sitemap without modifying the lastMod of the yearly sitemap. To do that, just construct your sitemaps one at a time using the WebSitemapGenerator, then use the SitemapIndexGenerator to create a single index for all of them.

WebSitemapGenerator wsg;
// generate foo sitemap
wsg = WebSitemapGenerator.builder("http://www.example.com", myDir)
    .fileNamePrefix("foo").build();
for (int i = 0; i < 5; i++) wsg.addUrl("http://www.example.com/foo"+i+".html");
wsg.write();
// generate bar sitemap
wsg = WebSitemapGenerator.builder("http://www.example.com", myDir)
    .fileNamePrefix("bar").build();
for (int i = 0; i < 5; i++) wsg.addUrl("http://www.example.com/bar"+i+".html");
wsg.write();
// generate sitemap index for foo + bar
SitemapIndexGenerator sig = new SitemapIndexGenerator("http://www.example.com", myFile);
sig.addUrl("http://www.example.com/foo.xml");
sig.addUrl("http://www.example.com/bar.xml");
sig.write();

You could also use the SitemapIndexGenerator to incorporate sitemaps generated by other tools. For example, you might use Google’s official Python sitemap generator to generate some sitemaps, and use WebSitemapGenerator to generate some sitemaps, and use SitemapIndexGenerator to make an index of all of them.

Validate your sitemaps

SitemapGen4j can also validate your sitemaps using the official XML Schema Definition (XSD). If you used SitemapGen4j to make the sitemaps, you shouldn’t need to do this unless there’s a bug in our code. But you can use it to validate sitemaps generated by other tools, and it provides an extra level of safety.

It’s easy to configure the WebSitemapGenerator to automatically validate your sitemaps right after you write them (but this does slow things down, naturally).

WebSitemapGenerator wsg = WebSitemapGenerator.builder("http://www.example.com", myDir)
    .autoValidate(true).build(); // validate the sitemap after writing
wsg.addUrl("http://www.example.com/index.html");
wsg.write();

You can also use the SitemapValidator directly to manage sitemaps. It has two methods: validateWebSitemap(File f) and validateSitemapIndex(File f).

Google-specific sitemaps

Google can understand a wide variety of custom sitemap formats that they made up, including a Mobile sitemaps, Geo sitemaps, Code sitemaps (for Google Code search), Google News sitemaps, and Video sitemaps. SitemapGen4j can generate any/all of these different types of sitemaps.

To generate a special type of sitemap, just use GoogleMobileSitemapGenerator, GoogleGeoSitemapGenerator, GoogleCodeSitemapGenerator, GoogleCodeSitemapGenerator, GoogleNewsSitemapGenerator, or GoogleVideoSitemapGenerator instead of WebSitemapGenerator.

You can’t mix-and-match regular URLs with Google-specific sitemaps, so you’ll also have to use a GoogleMobileSitemapUrl, GoogleGeoSitemapUrl, GoogleCodeSitemapUrl, GoogleNewsSitemapUrl, or GoogleVideoSitemapUrl instead of a WebSitemapUrl. Each of them has unique configurable options not available to regular web URLs.

URL Query Parameters and HTML Entities: The Case of the Missing Semicolon

Dan Fabulich — Wed, 15 Oct 2008 02:50:23 +0000

What’s the difference between this HTML snippet:

    foo=0

and this?

    copy=0

Both of them look like simple Google searches (though they could have been anything; Google is just an example). One of them appends an extra “&foo=0″ to the end of the URL; the other appends “©=0″ instead.

Only the second snippet is valid in HTML 4.01 Strict, but that snippet doesn’t work the way you might expect. Neither snippet is valid in XHTML.

Give up? Click on these:

Two weird things are happening here.

Note that “©” is an HTML entity for the copyright symbol “©.” It would have been more obvious if the URL had used a semicolon, like this:
```
    copy;=0
```
or if we’d used a more traditional HTML entity like this:
```
    quot;=0
```
The second weird thing is a quirk in the HTML specification on character references:

Note. In SGML, it is possible to eliminate the final “;” after a character reference in some cases (e.g., at a line break or immediately before a tag). In other circumstances it may not be eliminated (e.g., in the middle of a word). We strongly suggest using the “;” in all cases to avoid problems with user agents that require this character to be present.

As a result, all modern browsers (FF3, IE7, Opera 9, Safari 3.1) will helpfully notice possible entities like “©” and “<” and replace them with “©” and “<” … they assume you forgot the semicolon. This applies to all of the HTML entities, even the obscure ones like &empty “∅”, ¬ “¬”, ® “®”, &sub “⊂”, and &lang “⟨”. (Bizarrely, &Copy is left alone as “&Copy” but © is replaced with “©”.)

We think there are two valuable lessons to learn from this story. The first lesson you may already know:

The correct way to write an URL with a query parameter is to HTML escape the URL, replacing all &s with & like this:
```
    copy=0
```
That’s also the only way to make the snippet XHTML compliant.
Don’t use URL query parameters whose names are HTML entities. Never create a web service that accepts a query parameter like “&lang=en”. After all, there’s no way to know when your users might want to copy & paste your URLs into a blog, forum, or HTML email. Even if developers are clever enough to HTML escape href links, not everyone will be, and you can save everybody some trouble by avoiding the dangerous entities altogether.

Lightweight Headless Test for Trailing Commas in JavaScript

Dan Fabulich — Tue, 16 Sep 2008 20:26:56 +0000

Yesterday one of my co-workers spent hours reproducing and tracking down a bug that turned out to be a stray comma.

Unfortunately, Microsoft Internet Explorer doesn’t support trailing commas in JavaScript arrays and object literals. Code like this won’t parse:

    var chord = ["do", "mi", "so",];
    var json = { truth:"beauty", beauty:true, };

Those final commas will cause syntax errors in IE.

“How many thousands of developer hours have been lost to random IE bugs like this?” I asked myself. I decided that there had to be a good way to detect this problem in an automated way, without firing up a copy of IE and running a full test suite.

It turns out that these and other syntax errors can be detected automatically from the Windows command line, using the Windows Scripting Host (WSH). On Windows XP and higher, the command-line tool “cscript.exe” can be used to run JavaScript (ahem, JScript) headlessly (outside of any browser).

Just create a file called “wsh-parser.js” like this:

var fso = new ActiveXObject( "Scripting.FileSystemObject" );

function parse(fname) {
    var file = fso.OpenTextFile( fname, 1 );
    ret = file.ReadAll();
    file.Close();
    try {
        eval("(function(){\\n"+ret+"\\n});");
    } catch (e) {
        WScript.Echo("Syntax error parsing " + fname);
        WScript.Echo("  " + e.message);
    }
}

function findJavaScript(folder) {
    for (var fc = new Enumerator(folder.files); !fc.atEnd(); fc.moveNext()) {
        var file=fc.item();
        if (/.js$/.test(file.Name)) {
            parse(file);
        }
    }
    for (var fc = new Enumerator(folder.subfolders); !fc.atEnd(); fc.moveNext()) {
        var subfolder = fc.item();
        if (subfolder.Name == ".svn") continue; // ignore .svn folders
        findJavaScript(subfolder);
    }
}

var rootPath = "src/main/javascript";
var rootFolder = fso.GetFolder(rootPath);

findJavaScript(rootFolder);

Then run it like this:

    cscript //E:javascript //nologo wsh-parser.js

The script will automatically examine every JavaScript file in the specified rootPath, reporting errors in any file that won’t parse correctly.

There’s a bit of magic going on here in the “parse” function to guarantee that we’re only verifying the syntax of our JavaScript, without actually running it. The magic line is:

    eval("(function(){\\n"+ret+"\\n});");

The “eval” command would normally execute the provided string, but here we wrap the code in an anonymous function declaration. We improve the performance and the maintainability of our test by declaring the function without actually using it.

Note that you can also use JsLint to detect trailing commas, but it will probably report a bunch of other problems with your code that you may or may not care about.

Automated validators like these should be used together with headless unit tests as well as live browser tests. (I would recommend Selenium for automated browser tests. There are a ton of excellent headless JavaScript unit test tools, including Schaible’s JsUnit, RhinoUnit, and DOH. Note that Hieatt’s JsUnit is not headless.)

But I recommend running automated validators first, because they’re fast and the errors they report are easy to debug/fix. Once the automated validators are happy, run your unit tests. When your unit tests pass, run your live browser tests.