Monday, December 6, 2010

Randomly assigning classes to rows in a table

This morning a coworker came to me with a problem he wanted to solve using the Field Calculator in ArcMap. He had a table and he wanted to randomly assign rows in the table to different classes, given a set of probabilities for each class. So for example if class1 had a probability of 0.5, class2 of 0.3, class 3 of 0.1, and class4 of 0.1, then half of the rows in the table would be assigned class1, 30% of them would be class2, and the remaining rows would be split evenly between class3 and class4.

This is the solution I gave him. Make sure the Field Calculator is set to use the Python parser, and put the following code in the Codeblock area:

import numpy
probs = {}
probs[1] = .5
probs[2] = .3
probs[3] = .1
probs[4] = .1
def setclass():
    r = numpy.random.random()
    t = 0
    for key in probs.keys():
        t += probs[key]
        if t > r:
            return key

You can add as many classes as you want using the probs variable, but the probabilities should add up to 1. I used integers for the classes (1-4) but they could also be strings enclosed in quotes. Just make sure that the field being calculated is a string field in that case.

The last step is putting this in the calculator part of the Field Calculator dialog.


Wednesday, October 20, 2010

Using custom toolboxes in ArcToolbox

I don't care for the batch processor in ArcGIS. As far as I can tell, I need to manually type in the filenames for each of my zillion output files. Frankly, that's close to useless. Someone at ESRI should take a look at the batch processor in Erdas Imagine (which I haven't used in years, but I remember it was great). It allowed you to set up variables and automate things like output names.

So what does this have to do with custom toolboxes? Well, I wrote a Python script to do a bunch of stuff to a data set and created a tool that used my script. But my ultimate goal was to process a whole bunch of files, so then I created another tool that looped through all of the files in a directory, automatically created output names, and called the original tool to do the processing. This way I only needed to maintain the code in one place but had a tool that worked on just one file as well as one that could handle a ton of them at once.

I put both tools in a custom toolbox, but the batch tool couldn't find the original tool because the toolbox hadn't been imported. This makes sense to me when running a script outside of ArcMap, since the toolbox isn't loaded by default. But it seems to me that if a tool is running, the toolbox that it lives in should be automatically loaded. But it's not. And what's even weirder is that ListToolboxes() and ListTools() find the toolbox and tools in question, but they can't be used until ImportToolbox() is called.

That's fine, except ImportToolbox() takes a path to a toolbox file. I can hard code that in for myself, but what if I give the toolbox to a coworker? Unless they use the exact same directory structure as I do then the tool is broken. I don't want to make them go in and edit the script (that's just asking for trouble!). I couldn't find any way to solve this problem using the arcpy functions provided by ESRI. It would be nice if they had a way to get info, like filename, about a toolbox found with ListToolboxes(), but either I'm blind or it's not there.

Assuming that the script and the toolbox live in the same directory, there is a simple solution to the problem, however:

arcpy.ImportToolbox(os.path.join(os.path.split(sys.argv[0])[0], 'toolbox.tbx'))

This gets the path of the currently executing script (sys.argv[0]), gets the directory of that (os.path.split()[0]) and then appends the filename of the toolbox to that with os.path.join(). That is then passed to ImportToolbox() and everything is good. Until someone renames the .tbx file, that is...

Tuesday, August 31, 2010

Converting dbf to csv with Python

I wanted something in the ArcGIS toolbox that I could use to convert a dbf to a csv. I didn't see anything already there, and the scripts I found in the ESRI forums seemed needlessly complicated, so I wrote this little one and added it as a new script tool with two inputs of type File. The first one is an Input and is the dbf filename. The second one is an Output and is the csv filename.

This script uses the dbfpy module from

import csv
from dbfpy import dbf
import sys

dbf_fn = sys.argv[1]
csv_fn = sys.argv[2]

in_db = dbf.Dbf(dbf_fn)
out_csv = csv.writer(open(csv_fn, 'wb'))

names = []
for field in in_db.header.fields:

for rec in in_db:


Thursday, August 12, 2010

Smoother Pyramid Layers

So yesterday I was working with a couple of 1-band raster images. The original images looked fine but my processed ones looked pixelated when zoomed out.

The problem was that the pyramid layers were built using nearest neighbor (the default). I didn't see where to change that in ArcMap or IMAGINE right away, so I solved the problem with gdaladdo.

gdaladdo --config HFA_USE_RRD YES -r average all_utm.img 4 8 16 32 64 128 256 512

I later found the correct settings in ArcMap 10, but still have no idea where they are in IMAGINE 2010.

Pyramid options are in the Environment Settings in ArcMap 10

And here's what my final output looks like:

Friday, August 6, 2010

Creating an installer for a COM component for ArcGIS 10

I just wrote my first installer for an ArcGIS 10 COM component. I realize that ESRI has created this new thing called an add-in, but since I want my component to work with both 9.3 and 10, I figured it would be easiest to keep most things the same. So I built my base project, fired up ArcGIS 10, and my toolbar was right where I expected it to be. Then I built a setup program using the 9.3 instructions on the ESRI website. I built the setup program, ran it, started ArcGIS, and my toolbar was nowhere to be found. Hmmm....

Turns out that with version 10, ESRI decided not to use the regular Windows COM registration services and instead uses its own registration utility. This utility can be used to register COM components, or they suggest using the Add From File option on the Customize dialog in ArcMap. I didn't like that solution because that's asking a bit too much for some of our users (don't ask what they're doing using ArcMap!). I want them to just double-click a file icon.

So here's how I created my installer for ArcGIS 10:
  1. Open the solution that you want to create an installer for.
  2. Add an Installer Class to the project that you want to create the installer for.
  3. Add the following code to the new Installer class. Note that you may want to enclose things in try/catch blocks. I have it show an error message if it takes more than 5 seconds to unregister the component.
public override void Install(System.Collections.IDictionary stateSaver)
    string esriRegAsmFilename = Path.Combine(
    Process esriRegAsm = new Process();
    esriRegAsm.StartInfo.FileName = esriRegAsmFilename;
    esriRegAsm.StartInfo.Arguments =
        string.Format("\"{0}\" /p:Desktop", base.GetType().Assembly.Location);

public override void Uninstall(System.Collections.IDictionary savedState)
    string esriRegAsmFilename = Path.Combine(
    Process esriRegAsm = new Process();
    esriRegAsm.StartInfo.FileName = esriRegAsmFilename;
    esriRegAsm.StartInfo.Arguments =
        string.Format("\"{0}\" /p:Desktop /u", base.GetType().Assembly.Location);
    if (!esriRegAsm.WaitForExit(5000))
        MessageBox.Show("ESRI unregistration failed");
  1. Make sure the Specific Version property for the referenced assemblies in the project is set to False and rebuild the project.
  2. Add a Setup Project to your solution.
  3. Right-click the new Setup project in Solution Explorer and select Add | Project Output... 
  4. In the dialog box, select the project you want to deploy from the pulldown menu and then select Primary output and click OK.
  5. This will generate a list of detected dependencies. Right-click and exclude all of the ESRI assembles and stdole.dll.
  6. Open the Properties pane for the .tlb file in the dependency list, and change the Register property to vsdrfDoNotRegister.
  7. Right-click the Setup project in the Solution Explorer and select View | Custom Actions.
  8. In the Custom Actions window, right-click the Install folder and select Add Custom Action...
  9. In the dialog box, select File System on Target Machine from the pulldown menu and then double-click Application Folder.
  10. Double-click Primary output from <AssemblyName>.
  11. Repeat this for the Uninstall folder in the Custom Actions window.
  12. Build the setup project.

Thursday, August 5, 2010

Finding your custom ArcGIS extension with C#

An extension object is handy and logical place to store data when creating a custom component for ArcGIS. However, you need to be able access the extension object from your other objects such as commands. Here's how to do it.

Put something like this in the command's OnCreate() method (not in the constructor, especially if you're using a just-in-time extension, because the extension may not be loaded yet when the command is constructed):

UID uid = new UIDClass();
uid.Value = "MyNamespace.MyJITExtension";
m_extension = (MyJITExtension)m_application.FindExtensionByCLSID(uid);

The above method worked fine for me with ArcGIS 9.3 and Visual Studio 2008. I came across the following error when using ArcGIS 10 and Visual Studio 2010:

Error 4: Interop type 'ESRI.ArcGIS.esriSystem.UIDClass' cannot be embedded. Use the applicable interface instead.

To solve this problem, change the Embed Interop Types property for each referenced ESRI assembly to False. Or don't target .NET Framework 4.0.

Wednesday, June 16, 2010

Pinning folders to the taskbar in Windows 7

The fact that Windows Explorer initially opens up the Libraries in Windows 7 really annoys, but there are lots places on the web that tell you how to fix it. For the record, take a look at the shortcut properties and change the Target to the following value to make it open at the Computer level:
%SystemRoot%\explorer.exe /e,::{20d04fe0-3aea-1069-a2d8-08002b30309d}

The thing that nobody seems able to figure out, however, is how to pin a shortcut to a folder to the Taskbar. I've seen all kinds of methods to do it, but they're all way more complicated than they need to be.  There are also 3rd party apps that will do it, but here's a simple way to do it without them:
  1. Create a new shortcut on your desktop (or anywhere, for that matter). To do this, right-click and select New | Shortcut.
  2. In the dialog that opens, paste the following value into the Location box, but replace z:\data with whatever folder you want to open: %SystemRoot%\explorer.exe z:\data
  3. Click Next, name your shortcut, and click Finish.
  4. If you want to change the icon, you can right-click on your new shortcut, choose Properties, and then click the Change Icon button.
  5. To pin your shortcut to the Taskbar, right-click on it and choose Pin to Taskbar.
  6. Once you've pinned it to the Taskbar, you can go ahead and delete the original shortcut if you want.