Five Computers

Simple web server

Ever find you need to implement a web server or provide some web pages on a local network? Maybe you don't have an Apache server up and running or don't want to go to the trouble of configuring it. I've found this with Natural Docs documentation, where the tools will generate a bunch of .html files that you can read locally if you are on the machine where they are generated. Pygments also will generate html formatted files that you might want to serve. However, you might want to make them accessible to a small team. Here is a quick trick that can make this task very simple. Python ships with a basic HTTP server module. You can get it up and running, serving files from a given directory down with the following command:

python -m SimpleHTTPServer

and that's all there is to it! The default port will be 8000, but you can provide a different port on the command line (just add the port number after the SimpleHTTPServer - any number above reserved range of 0-1024 will work. The server will be accessible to any machines that can see that port on the machine it is running on. The server won't handle a high load and isn't particular secure, so I wouldn't make it available to the general public. But if you need to serve some pages quickly and simply, to a small number of users within a private network, this can be a really fast way to get to that point. If and when the load becomes an issue, or security concerns are important, then a heavyweight server like Apache is a much better choice.

Once the server is up and running you access it via a web browser at http://machine_name:8000 File paths are all relative to the directory the server is running in. If there is an index.html in that directory, it will be loaded by default when you access the server at that URL.

There are comments.

Simplified VPI iterators using PyHVL generators

I've been using PyHVL for a variety of verification tasks in the past few years. PyHVL is an open source Python integration for Verilog and SystemVerilog simulators. To give a quick taste of what it can do for you, consider the following SystemVerilog VPI C code.

void display_nets(mod)
    vpiHandle mod;
    {
       vpiHandle net;
       vpiHandle itr;
       vpi_printf("Nets declared in module %s\n",
       vpi_get_str(vpiFullName, mod));

       itr = vpi_iterate(vpiNet, mod);

       while (net = vpi_scan(itr))
          {
          vpi_printf("\t%s", vpi_get_str(vpiName, net));
          if (vpi_get(vpiVector, net))
          {
             vpi_printf(" of size %d\n", vpi_get(vpiSize, net));
          }
          else vpi_printf("\n");
       }
    }

Here is the equivalent VPI code, this time written in Python, using PyHVL.

def display_nets(module):

    print 'Nets declared in module', get_str(vpiFullName, module) 

    for net in vpi_iterator(module, vpiNet):
        print '\t%s %s' % ( get_str(vpiName, net), 
                get(vpiVector, net) and 'of size %d' % get(vpiSize, net) or '')

The magic happens in the implementation of the vpi_iterator() method, which uses a Python yield instruction to turn the method into a generator. Generators are much like functions, except they maintain a frozen stack frame, at the point where the yield occurs. All existing variables within the method maintain their state and execution picks up where it left off, just after the yield. The example also uses lazy evaluation of the result of get(vpiVector, net) to either call get(vpiSize, net) or not print the 'of size' additional string.

def vpi_iterator(handle, type=vpiNet):
    iterator = iterate(type, handle)
    if iterator:
        handle = scan(iterator)
        while handle:
            yield handle
            handle = scan(iterator)

This lets you write loops inside out as one of my colleagues aptly put it. The outcome is you can simplify the management of loops and indices and focus on the point of the loop. You write less code, you introduce fewer bugs. The code is easier to read and maintain as a result. This is just a very small example of some of the power of using a modern scripting language like Python, as an adjunct to a SystemVerilog simulator.

If you spend much time writing VPI code, you should take a look at PyHVL. It could make your life much simpler, or get in touch and I can help you with it.

There are comments.

csv the easy way

I've recently started using FogCreek's FogBugz project hosting for a small personal project. It has a built-in Mercurial source repository, with some enhancements, called Kiln, and good task/ project tracking features. Quite similar to those you find in Trac. Rather than putting together the various servers to track my own project, I figured I might as well use what they give away for free, for small project teams. I had a list of tasks that I wanted to import into the FogBugz task list, from a CSV file. In fact, this was a CSV file I'd exported from a different FogBugz site, but it could well have been any list of comma seperated values holding task information. The trick was how to get that information from my computer onto the FogBugz site. They didn't have any obvious CSV import option, but they do publish an XML API. Luckily enough, there is also a set of python bindings around the API. Amazingly enough, I was able to open and read the CSV file, parse it, login in to my FogBugz account and upload the new tasks to the server with just this little snippet of code.

from fogbugz import FogBugz
import csv
fb = FogBugz('http://my_site.fogbugz.com/') # URL is to your FogBugz install
fb.logon('my_login@my.site.com', 'password')

cases=csv.reader(open('Filter.csv','r'))
for case in cases:
    print case
    fb.new(sCategory=case[0], sTitle=case[3], sPriority=case[7]) 
fb.logoff()

There are comments.

creating a codeswarm movie

Download video (3Mb)

A codeswarm is a visualization of the activity within a source code repository. The image and linked video above shows the lifetime of one of Verilab's source repositories. You can see code being created, the check-ins as they happen and an indication of which users are doing the work at any given time. It is an example of an 'organic information visualization' and is created using the Processing toolkit. The original visualization tools were developed by Michael Ogawa and the source code is available on Google code.

In this particular case I created the animation on OS X 10.5, using a combination of codeswarm, ffmpeg and LAME. If you are interested in doing something like this yourself:

First you'll need to make sure you have a recent version of the Java Development Kit installed (JDK 1.5 or later). You'll also need a recent version of Ant installed. (I have version 1.7.0, which ships with OS X as default). Download the code_swarm source and install it. Then execute 'ant run'. If all is well, you should get a dialog box prompting you for the source repository, user name and password.

At this point, I put in the svn+ssh URL for the Verilab repository that I wanted to visualize. Everything fell over, with a Java error (NoClassDefFoundError within com/trilead/ssh2). From this I realised I needed to install the SSH libraries for Java, from Trilead. I downloaded those, unpacked them and added the jar file to my CLASSPATH. Along the way I found out the default OS X CLASSPATH definition is in /System/Library/Java/JavaConfig.plist which may be useful as a starting point.

With that fixed, I again ran 'ant run' and put in the relevant information. A bit of time passes as the checkin information is extracted from the repository, then the visualisation runs. You'll find that repository information that was extracted is saved, under the ./data directory (look for the latest realtime_sample.*.xml file) . This is useful for the next stages, as you don't have to fetch the information again. If you want to create a video of the visualisation, there are a few more hoops to jump through.

You will need to configure codeswarm to save the frames for each stage of the visualisation. You do this by editing the ./data/sample.config file. First off, copy it to a new version for your particular project. Then edit these values:

• InputFile= [Point it at the new realtime_sample<number>.xml file in the data directory, that contains the checkin information for your project]
• TakeSnapshots=true

That's all you really need to change. You can also change the other values, to alter the visualisation. The ColorAssignX= statements use regexp values to differentiate different types of checkin and colour code them accordingly. Play around with the other values, with TakeSnapshots set to false and re-run the visualisation until you get something you are satisfied with. Then run one more time with TakeSnapshots=true to save off the frame images. You can run with the new configuration by running 'ant run data/your_project.config'

After running with TakeSnapshots enabled, you'll have a set of images in the ./frames directory, (controlled by the SnapshotLocation option in the config file). The final step is to assemble those into a movie. The easiest way I found to do this is to use the command-line utility, ffmpeg. There are a variety of ways to install ffmpeg, but the simplest way seems to be to install ffmpegX and then extract the binary from the application bundle. You can also get it using Fink or MacPorts. If you want to use an audio track with your visualisation, you will also probably require LAME. With ffmpeg working, it is simple to point it towards the image files from codeswarm and produce the final movie. The finishing touch was adding some music from an mp3 file, then limiting the duration via the -t switch, to end when the video frames ran out, rather than playing all of the music.

ffmpeg -i frames/code_swarm-%05d.png -i 6_sym.mp3 -qmax 15 -t 100 -f image2 -r 24 <output_filename>.mpg

You can run 'ffmpeg' without any switches to get help on the options. If all goes well, you should end up with an MPEG format video in the file <output_filename>.mpg.

There are comments.

iPhone development

Interested in learning about iPhone development? Want to study at Stanford? Don't want to pay the tuition fees? On iTunesU (the lecture streaming part of iTunes) you can follow along with class cs193p from Stanford, on iPhone Application Development. In addition to the good quality video of the lectures, all of the class slides, handouts and assignments are available, for free. If you have an Intel Mac, you can also download the development tools, iPhone SDK and a simulator, again all free. If you do want to actually develop and test applications on an iPhone or iPod Touch, you'll need to pay the $99 developer fee to get the encryption keys that let you run applications on a phone and allows you to submit apps for the app store. At least for the basics, the simulator is useful as a target platform for testing, although there are differences between it and the final platform. (features such as multi-touch and the accelerometer are hard to test for example, unless you want to start shaking your computer).

Lifehacker recently had an article on all of the educational resources that are becoming available on the web, for free. iTunesU is a good example of the sort of teaching resources that are out there, if you look. The quality is variable, but there are some excellent resources if you are prepared to dig.

There are comments.

what did you say?

I've been poking around at the Twitter API, in part just out of curiosity about what features are exposed. I have an interest in writing some visualisation widgets based upon it. The iPhone development course is also using a Twitter client as something of a 'hello world' app, too. Today, Tim O'Reilly pointed to a wordle visualisation of all the things that he's tweeted and gave a link to some code that could be used to download everything you'd tweeted. I had a look at it and decided to write something similar, using the Twitter API directly, rather than scraping the Twitter site.

The Twitter API I've been using is the excellent, minimalist python twitter tools by Mike Verdone. The main advantage over other python Twitter APIs is that ptt doesn't redefine any of the API calls. It does exactly what it says in the published Twitter API. As a result, it is incredibly easy to use. The 100 or so lines it is implemented in are also a very instructive read, to see how it is put together. I think it is a great example of how the attributes in Python can be used.

The code I wrote is available for download. It respects the rate limiting imposed by Twitter and will output all of the tweets for a particular user, to a file called <username>.tweet in the file it is run from. You can change which users are fetched in the main() ftn. The resulting text file can be opened up and then copy/ pasted over into the wordle creator.

There are comments.

edward tufte and presenting data

I was lucky enough to attend a seminar from Edward Tufte, a couple of weeks ago, on the Presentation of Data and Information. Edward Tufte is probably best known for the book 'The Quantitative Display of Visual Information' and was an engaging and entertaining presenter. He has a very different style from the normal Powerpoint-driven presentation approach. In fact, much of his work is railing against the uses and abuses of Powerpoint and similar slide techniques.

The main take-away I got from the whole day was that if you have to communicate complicated data sets or information, that you really need to consider how people will use and interact with the data first. Too often, we go straight to presentation software and start trying to work out how to express the information in slides, rather than taking the time to consider if there are other, better ways to impart the information. Tufte was very keen on the concept of a 'super-graphic' which is a data rich, high resolution physical handout that lets participants see and consider a lot of data at once. A map is a great example of a super-graphic, or the weather page in a typical newspaper. A key part of this is that paper is much higher resolution than a typical computer screen (72dpi to 600dpi means you can show a whole lot more data in the same space). This is why multiple display screens are really useful for serious work. It also means that printing out and sharing data is a great way to get information infront of people in a meeting, rather than drip feeding it from slides)

I compare this idea to another guide I saw in the same week on creating powerpoint presentations that admonishes that there should never be more than 8 numbers on any slide or graphic. Tufte's response to this was repeatedly 'when did we become so stupid, just because we walked into a business meeting?' People handle large, complex data displays every day in the real world. People read and study sports scores in a newspaper, or financial reports without any trouble at all.

Let the data drive the presentation format, rather than the presentation software drive how the data is displayed.

There are comments.

maker's schedule, manager's schedule

I've always had a dread of mid-afternoon status meetings. Now maybe I understand it a bit better, because of Paul Graham's excellent essay on the difference between being on a maker's schedule and being on a manager's schedule. Seems to share a lot of ideas with Csikszentmihalyi and his ideas of creativity and flow states.

There are comments.

metrics

Listened to Brian Bailey talking on the missing metrics for verification coverage at the Certess booth on Wednesday. This was a really interesting talk discussing the standard code coverage and functional coverage metrics and pointing out that we don't really currently have a good metric to track when we are doing useful verification or not. Functional coverage is used for this, but care needs to be taken to ensure we are covering the right things.

There's a danger that functional coverage, which was originally designed to track the quality of random stimulus, is used to only track that stimulus. In this way, we might not consider if the behaviour is actually checked or not. The same ideas are represented in David Robinson's verification planning presentation at DAC. Requirements and functional coverage without checkers is a waste of time and effort. It is possible to get 100% coverage, with lots of cover points and no enabled or implemented checkers in a design. Careful testbench design, or tools like Certess, can be used to address these issues but only once you realise that functional coverage isn't a magic bullet.

I also saw a demo of the Certess product, Certitude. It randomly inserts errors into the device under test (DUT) to give a way to evaluate the quality of the testbench surrounding the DUT. The basic idea is that if the design is mutated in a particular way, the testbench is good if it can detect the error, or has a hole if this failure mode cannot be seen in the checkers. This artificial bug insertion is something I do on an ad-hoc way when developing a testbench and Certitude automates and expands on this commonly used approach.

You can read a paper from Brian on this topic.

There are comments.

a command line for the gui

I've been experimenting with Mac OS X now for a few months. Trying to work out if it is a reasonable platform for Verilab to use internally for our various computing needs. One thing that I've come to love and now struggle to live without is a small add-on called Quicksilver. Superficially it isn't very interesting. A fast application launcher - where's the fun in that? But after you delve a bit deeper, it becomes something else. It is really a command line interface for OS X, on top of the GUI.

I hit a key and up it pops. Type a few characters and I can launch an application. I don't have to search through drop down menus or find icons on a desktop - much faster. I want to get a music player to jump to the next track - hot key and type 'next'. Stop the music, type 'stop'. The magic is that I don't have to switch away from what I'm currently doing, go to a different window or application and do anything. Want to email someone? Hot key, type a bit of their name and select their email address - again, without switching away from what you are working in. I can even open and edit text files without starting up a text editor - extend a todo list, add a calendar entry to my google calendar, all from the command line within the GUI. All without switching away from the current context. I can run unix terminal commands, search in documents, select groups of files and email them to someone, do quick calculations. It is amazingly more efficient than using a mouse and hunting for applications and buttons.

I've always been a bit of a command-line junkie, wanting to know the keyboard shortcuts for things in a GUI environment, liking having a command prompt, but Quicksilver is different. It's a visual blend of command-line and GUI. Taking the best bits of both and putting them right under my fingertips. Highly recommended. Shame there isn't anything nearly as good for Windows XP or linux that I've found. Look here for more ideas about what Quicksilver can do.

There are comments.