Monday, October 29, 2012

IIS, Visual Studio and Razor

I'm working on my second project that uses MVC and Razor. While working on the first one, I noticed that any time I tried to paste something into a .cshtml file, Visual Studio hung for what seemed like an eternity (although it was probably 10-20 seconds). It was so annoying that I edited those files in a text editor and never opened them in Visual Studio.

I didn't notice this problem in my current project, until today. Not only that, but suddenly every time I tried to set a breakpoint in a .cshtml file, Visual Studio would freeze and I would end up killing it with task manager. Everything worked fine until today, and I was pretty pissed when it quit working. I restarted Visual Studio multiple times, I rebooted my computer, I changed my code back to how it was at the beginning of the day, all to no avail.

But then I remembered that earlier today I had changed the ASP.NET Impersonation setting for my application in IIS because I needed it to impersonate a specific user. On a whim, I changed that back to the default (Authenticated user). And guess what? Now Visual Studio plays nicely with my .cshtml files.

Then, on another whim, I changed my specific user to an Administrator account and tried it again, and it still works.

So the moral of the story is to use an Administrator account (on the development machine) if you need to have IIS impersonate a specific user. I guess I can kind of see how using a regular account might mess with the breakpoints, but cutting and pasting? Really, Microsoft?

Wednesday, February 1, 2012

Automating rsync on a Mac

For reasons that I won't go into here, I decided that I wanted to sync some of the folders on one of my internal hard drives to another internal drive. This is not a backup system, because old versions of files are not saved (I have Time Machine for that)-- it's just a way to mirror data.

Naturally I thought of using rsync, and the command to copy the files is easy:

sudo rsync -axES --delete /Volumes/MacA/Data /Volumes/MacAMirror/

However, I wanted this to run automatically. I tried using Automator, but I'd have to enter a password each time the thing ran, which I did not want to do. In fact, I wanted it to run at night when I wasn't even here.

My solution? Create a symbolic link to rsync and then edit the sudoers file so that I didn't need to enter a password when running that program. I did this instead of changing my access to the main rsync executable so that I didn't accidentally get myself in trouble some time with rsync.

So first I created a directory to put my symbolic link and shell script in.

~$ mkdir ShellScripts
~$ cd ShellScripts
~/ShellScripts$ ln -s /usr/bin/rsync nopass_sudo_rsync
~/ShellScripts$ ls -l
total 16
lrwxr-xr-x  1 chrisg  staff   14 Jan 27 11:44 nopass_sudo_rsync -> /usr/bin/rsync
-rwxr--r--  1 chrisg  staff  334 Jan 31 12:50

The script file ( looks like this:


logger 'Start MacA rsync'
sudo /Users/chrisg/ShellScripts/nopass_sudo_rsync -axES --delete /Volumes/MacA/Data /Volumes/MacAMirror/
logger 'Stop MacA rsync'

Then I edited the sudoers file:

~/ShellScripts$ sudo visudo

I added this line to the Cmnd alias specification section:

Cmnd_Alias JUSTRUN  = /Users/chrisg/ShellScripts/nopass_sudo_rsync

And this line to the User privilege specification


This creates a group of commands called JUSTRUN that chrisg is allowed to run with sudo without a password.

The last step was to create a plist file in /Library/LaunchDaemons.

~/ShellScripts$ cd /Library/LaunchDaemons
/Library/LaunchDaemons$ sudo vi org.chrisg.daily.rsync_data.plist

Here are the contents of org.chrisg.daily.rsync_data.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN"
<plist version="1.0">

This will run /Users/chrisg/ShellScripts/ daily at 20:05.

See here for some plist recipes.

Use launchctl to get the automation working immediately instead of waiting for the next reboot.

/Library/LaunchDaemons$ launchctl load org.chrisg.daily.rsync_data.plist

Monday, December 6, 2010

Randomly assigning classes to rows in a table

This morning a coworker came to me with a problem he wanted to solve using the Field Calculator in ArcMap. He had a table and he wanted to randomly assign rows in the table to different classes, given a set of probabilities for each class. So for example if class1 had a probability of 0.5, class2 of 0.3, class 3 of 0.1, and class4 of 0.1, then half of the rows in the table would be assigned class1, 30% of them would be class2, and the remaining rows would be split evenly between class3 and class4.

This is the solution I gave him. Make sure the Field Calculator is set to use the Python parser, and put the following code in the Codeblock area:

import numpy
probs = {}
probs[1] = .5
probs[2] = .3
probs[3] = .1
probs[4] = .1
def setclass():
    r = numpy.random.random()
    t = 0
    for key in probs.keys():
        t += probs[key]
        if t > r:
            return key

You can add as many classes as you want using the probs variable, but the probabilities should add up to 1. I used integers for the classes (1-4) but they could also be strings enclosed in quotes. Just make sure that the field being calculated is a string field in that case.

The last step is putting this in the calculator part of the Field Calculator dialog.


Wednesday, October 20, 2010

Using custom toolboxes in ArcToolbox

I don't care for the batch processor in ArcGIS. As far as I can tell, I need to manually type in the filenames for each of my zillion output files. Frankly, that's close to useless. Someone at ESRI should take a look at the batch processor in Erdas Imagine (which I haven't used in years, but I remember it was great). It allowed you to set up variables and automate things like output names.

So what does this have to do with custom toolboxes? Well, I wrote a Python script to do a bunch of stuff to a data set and created a tool that used my script. But my ultimate goal was to process a whole bunch of files, so then I created another tool that looped through all of the files in a directory, automatically created output names, and called the original tool to do the processing. This way I only needed to maintain the code in one place but had a tool that worked on just one file as well as one that could handle a ton of them at once.

I put both tools in a custom toolbox, but the batch tool couldn't find the original tool because the toolbox hadn't been imported. This makes sense to me when running a script outside of ArcMap, since the toolbox isn't loaded by default. But it seems to me that if a tool is running, the toolbox that it lives in should be automatically loaded. But it's not. And what's even weirder is that ListToolboxes() and ListTools() find the toolbox and tools in question, but they can't be used until ImportToolbox() is called.

That's fine, except ImportToolbox() takes a path to a toolbox file. I can hard code that in for myself, but what if I give the toolbox to a coworker? Unless they use the exact same directory structure as I do then the tool is broken. I don't want to make them go in and edit the script (that's just asking for trouble!). I couldn't find any way to solve this problem using the arcpy functions provided by ESRI. It would be nice if they had a way to get info, like filename, about a toolbox found with ListToolboxes(), but either I'm blind or it's not there.

Assuming that the script and the toolbox live in the same directory, there is a simple solution to the problem, however:

arcpy.ImportToolbox(os.path.join(os.path.split(sys.argv[0])[0], 'toolbox.tbx'))

This gets the path of the currently executing script (sys.argv[0]), gets the directory of that (os.path.split()[0]) and then appends the filename of the toolbox to that with os.path.join(). That is then passed to ImportToolbox() and everything is good. Until someone renames the .tbx file, that is...

Tuesday, August 31, 2010

Converting dbf to csv with Python

I wanted something in the ArcGIS toolbox that I could use to convert a dbf to a csv. I didn't see anything already there, and the scripts I found in the ESRI forums seemed needlessly complicated, so I wrote this little one and added it as a new script tool with two inputs of type File. The first one is an Input and is the dbf filename. The second one is an Output and is the csv filename.

This script uses the dbfpy module from

import csv
from dbfpy import dbf
import sys

dbf_fn = sys.argv[1]
csv_fn = sys.argv[2]

in_db = dbf.Dbf(dbf_fn)
out_csv = csv.writer(open(csv_fn, 'wb'))

names = []
for field in in_db.header.fields:

for rec in in_db:


Thursday, August 12, 2010

Smoother Pyramid Layers

So yesterday I was working with a couple of 1-band raster images. The original images looked fine but my processed ones looked pixelated when zoomed out.

The problem was that the pyramid layers were built using nearest neighbor (the default). I didn't see where to change that in ArcMap or IMAGINE right away, so I solved the problem with gdaladdo.

gdaladdo --config HFA_USE_RRD YES -r average all_utm.img 4 8 16 32 64 128 256 512

I later found the correct settings in ArcMap 10, but still have no idea where they are in IMAGINE 2010.

Pyramid options are in the Environment Settings in ArcMap 10

And here's what my final output looks like:

Friday, August 6, 2010

Creating an installer for a COM component for ArcGIS 10

I just wrote my first installer for an ArcGIS 10 COM component. I realize that ESRI has created this new thing called an add-in, but since I want my component to work with both 9.3 and 10, I figured it would be easiest to keep most things the same. So I built my base project, fired up ArcGIS 10, and my toolbar was right where I expected it to be. Then I built a setup program using the 9.3 instructions on the ESRI website. I built the setup program, ran it, started ArcGIS, and my toolbar was nowhere to be found. Hmmm....

Turns out that with version 10, ESRI decided not to use the regular Windows COM registration services and instead uses its own registration utility. This utility can be used to register COM components, or they suggest using the Add From File option on the Customize dialog in ArcMap. I didn't like that solution because that's asking a bit too much for some of our users (don't ask what they're doing using ArcMap!). I want them to just double-click a file icon.

So here's how I created my installer for ArcGIS 10:
  1. Open the solution that you want to create an installer for.
  2. Add an Installer Class to the project that you want to create the installer for.
  3. Add the following code to the new Installer class. Note that you may want to enclose things in try/catch blocks. I have it show an error message if it takes more than 5 seconds to unregister the component.
public override void Install(System.Collections.IDictionary stateSaver)
    string esriRegAsmFilename = Path.Combine(
    Process esriRegAsm = new Process();
    esriRegAsm.StartInfo.FileName = esriRegAsmFilename;
    esriRegAsm.StartInfo.Arguments =
        string.Format("\"{0}\" /p:Desktop", base.GetType().Assembly.Location);

public override void Uninstall(System.Collections.IDictionary savedState)
    string esriRegAsmFilename = Path.Combine(
    Process esriRegAsm = new Process();
    esriRegAsm.StartInfo.FileName = esriRegAsmFilename;
    esriRegAsm.StartInfo.Arguments =
        string.Format("\"{0}\" /p:Desktop /u", base.GetType().Assembly.Location);
    if (!esriRegAsm.WaitForExit(5000))
        MessageBox.Show("ESRI unregistration failed");
  1. Make sure the Specific Version property for the referenced assemblies in the project is set to False and rebuild the project.
  2. Add a Setup Project to your solution.
  3. Right-click the new Setup project in Solution Explorer and select Add | Project Output... 
  4. In the dialog box, select the project you want to deploy from the pulldown menu and then select Primary output and click OK.
  5. This will generate a list of detected dependencies. Right-click and exclude all of the ESRI assembles and stdole.dll.
  6. Open the Properties pane for the .tlb file in the dependency list, and change the Register property to vsdrfDoNotRegister.
  7. Right-click the Setup project in the Solution Explorer and select View | Custom Actions.
  8. In the Custom Actions window, right-click the Install folder and select Add Custom Action...
  9. In the dialog box, select File System on Target Machine from the pulldown menu and then double-click Application Folder.
  10. Double-click Primary output from <AssemblyName>.
  11. Repeat this for the Uninstall folder in the Custom Actions window.
  12. Build the setup project.