Five Computers

A simple turtle graphics Domain Specific Language (DSL) parser

I've been working through some of the exercises in 'The Pragmatic Programmer' while using it to mentor a colleague.  In particular, I spent a bit of time working on the Domain Specific Language section and considering creating little DSLs to provide flexible control. I've painfully implemented DSLs in SystemVerilog/ UVM sequences in the past and thought about how I typically do this: build a parser, create tokens, then dispatch various execution functions to implement the commands.  Often, the pain of building all these pieces in a language like C or SV is enough of a barrier that I wouldn't even start.  

The example in the book is a simple Logo/ Turtle control language.  Pen up, Pen down, draw, turn, etc. It tries to simplify things by using single letter command codes and only one optional argument.

P 2

D

W 2

N 1

E 1 

S 1

U

I know Python has a Logo/ Turtle engine built in so I decided to try to write a command parser/ dispatcher that would work with that, and let me write commands in a text file and have them execute Python Turtle commands.

Here's an example script (you can see the results of this at the end of this post):

color "green"
up
goto 0 -50 # comments are allowed
down
circle 50
up
color "red"
write "Done!"
sleep 2
exit

I thought initially about having to parse out all the commands and arguments, then writing a large switch/ case statement (or the Python equivalent with a dictionary).  After thinking about it a little longer I realised I didn't have to do that at all.  I could use the Introspection in Python to look up available methods and if they exist, call them. In fact, as not finding the method will just cause an exception, I can just try executing any command and if it fails, catch it and move on.

So after parsing the input text stream (throw away comments, break up tokens using whitespace) then I just try to execute the command in the global namespace.  I've pulled all the turtle functions into that namespace, so any turtle function is a valid command in my little parser.  The globals()[command[0]] in the code below looks up the function in the global namespace and then calls it, using the other parts of the command (command[1:]), after they've each been processed through the Python eval function to convert them from strings to whatever format they happen to be (numbers or strings mainly).  The final trick in this is using the * operator to take take a list and pass it as one argument after another to the function:

globals()[command[0]](*(map(eval, command[1:])))

And that's all that's needed to implement a full Logo language parser and execution engine. The command syntax is fault tolerant and reports errors, with line numbers.  New commands can be added easily, by defining new functions.  They'll be automatically supported as they are added to the namespace.

Being able to pull something like this together quickly means that writing little Domain Specific Languages is possible and quite a low bar.  Doing something similar in C is often more daunting and even worse in a language like SystemVerilog, with such a poor string and file handling library.  There's a real value in being able to program at such a high level, that can greatly enhance what's possible or likely to be attempted in a verification environment. You could do this in SystemVerilog, but how often would you even think to attempt it, without rejecting it as too much work?

# pull all the turtle commands into the global namespace, so they are valid commands
from turtle import *

# Use these 3 lines to make 'sleep' and 'exit' useable commands
from time import sleep
from sys import exit, argv

# Given a handle to a series of strings of commands, do them
def parse_and_draw(commands):
    for (line_number, line) in enumerate(commands):
        line = line.split('#')[0]  # throw away comments

        if line:  # if there is anything left after getting rid of comments

            command = line.strip().split(' ')  # parse using spaces to delimit tokens  
                                               # a big limitation of this, we can't have strings with spaces
                                               # e.g.,  "hello world" won't work as it'll get split up
                                               # into ['"hello', 'world"'] neither of which bits are valid when eval'ed

            if command[0]:  # if we have any command left 
                            # (e.g., an indented comment would dissappear)

                try:
                    # The meat of the dispatcher is the next line
                    # using a try/ except means we can always try to run any command
                    # and assume that it is valid and catch if it isn't.
                    # globals() returns a dictionary of every function defined in the global namespace
                    # including all the turtle commands because of the from turtle import *
                    # command[0] is used as a key to look up the function name
                    # we then pass all of the other tokens (command[1:]) through eval (using map)
                    # and pass them as arguments to the function we looked up

                    globals()[command[0]](*(map(eval, command[1:])))

                    # a key error occurs if we don't find command[0] in the global namespace
                except KeyError:
                    print 'Unknown command', command[0], 'on line ', line_number

                    # Some other error occurred (e.g., the called function raised an Exception)
                    # report it here and continue on (if we didn't catch it, the program would end)
                except Exception as e:
                    print 'Invalid command', command[0], 'on line ', line_number,  e


if __name__ == '__main__':
    # open the first file on the command line, get commands from that and run them
    commands = open(argv[1]).xreadlines()

    # note that the parse and draw routine works on a list of commands, it doesn't know about files
    # or anything else.  This seperation is is useful, as we can get commands from anywhere
    parse_and_draw(commands)

There are comments.

A UVM layer for PyHVL

Over the past few years I've been building verification environments in Python, hooked up to SystemVerilog and Verilog simulators. The glue is PyHVL. But PyHVL is just the glue. It does open up the possibility of dynamic languages and scripting to talk to your test bench, but it isn't enough to implement a complete verification environment, quickly.

To that end, I've started building analogues for most of the UVM classes, in Python/ PyHVL. Monitors, drivers, checkers, interfaces, scoreboards, all the standard sorts of pieces that you'd expect from a modern verification environment. With no compiling between iterations, and a command-line REPL to drop to for debug and interactive exploration. All backed with the batteries-included philosophy of Python. The sweet spot for this is probably somewhere within a fully fleshed out SystemVerilog testbench, leveraging the best features of SV (RTL/design, randomization with constraints, coverage, assertions) with the best features of Python for the higher level pieces of the testbench. At the same time, there is the potential to write the entire testbench in Python and the whole design in Verilog and use a free simulator such as Icarus or CVer. Python brings a variety of useful features for testbench development; dynamic language, rapid development, rich set of libraries, fast iteration and a high level programming abstraction.

I've been able to write models of processors that actually execute instructions and check the RTL, where the initial development time for a working CPU model was about 4 hours. There is effectively no recompile time cost - just re-run and the Python verification environment is re-interpreted as the simulation starts up.

The main advantage of using Python for a testbench is writing less code. Less code means fewer bugs. The higher level language features of Python let you write more efficient code. As a quick example, here is the entire class definition for a PyHVL/ UVM interface, ready to hook up to RTL and monitor/drive signals in a DUT.

from pluvm.uvm_package import *

HOST_BUS_SIGNALS=('reset', 'clock', 'cycle_count', 'host_rdata', 'host_ack',
                  'host_err', 'host_address', 'host_valid', 'host_wdata', 
                  'host_strobe', 'host_rwb')

class host_interface(uvm_interface):

        def __init__(self, name, hierarchy):
                uvm_interface.__init__(self, name, hierarchy)

                for signal in HOST_BUS_SIGNALS:
                        self.add_signal(signal)

These classes can of course leverage the standard pyunit test frameworks and have unit tests in every class.

There are comments.

sshfs for OS X Lion

Installed the latest version of fuse and sshfs, using Homebrew. I had been using FUSE for a while with previous versions of OS X, but haven't had much luck under Lion. Tried again today, using Fuse4x, rather than libFUSE and everything worked very transparently:

brew install fuse4x sshfs

and that was it!
No, of course that wasn't true. First I had to re-update my Xcode command-line tools. This then removed autoreconf which Apple no longer provide as part of their XCode tools bundle [in neither the command line, or GUI release]. So, via Google, I find you have to install a different set of gcc tools, not shipped by Apple. Then, back to the brew install command line above, and things are a lot happier.
and that was it!
Well, except you really do need to read the installation message which says

==> Caveats

   Make sure to follow the directions given by brew info fuse4x-kextbefore trying to use a FUSE-based filesystem.

brew info fuse4x-kext

sudo cp -rfX /usr/local/Cellar/fuse4x-kext/0.8.14/Library/Extensions/fuse4x.kext /System/Library/Extensions 
sudo chmod +s /System/Library/Extensions/fuse4x.kext/Support/load_fuse4x

sshfs <remote_host>:<remote_path> <local_path>

umount <local_path> 

There are comments.

Simple web server

Ever find you need to implement a web server or provide some web pages on a local network? Maybe you don't have an Apache server up and running or don't want to go to the trouble of configuring it. I've found this with Natural Docs documentation, where the tools will generate a bunch of .html files that you can read locally if you are on the machine where they are generated. Pygments also will generate html formatted files that you might want to serve. However, you might want to make them accessible to a small team. Here is a quick trick that can make this task very simple. Python ships with a basic HTTP server module. You can get it up and running, serving files from a given directory down with the following command:

python -m SimpleHTTPServer

and that's all there is to it! The default port will be 8000, but you can provide a different port on the command line (just add the port number after the SimpleHTTPServer - any number above reserved range of 0-1024 will work. The server will be accessible to any machines that can see that port on the machine it is running on. The server won't handle a high load and isn't particular secure, so I wouldn't make it available to the general public. But if you need to serve some pages quickly and simply, to a small number of users within a private network, this can be a really fast way to get to that point. If and when the load becomes an issue, or security concerns are important, then a heavyweight server like Apache is a much better choice.

Once the server is up and running you access it via a web browser at http://machine_name:8000 File paths are all relative to the directory the server is running in. If there is an index.html in that directory, it will be loaded by default when you access the server at that URL.

There are comments.

SystemVerilog support in Pygments

I've used Pygments in the past to do syntax highlighting of Python and C code to drop into documentation. I've wanted SystemVerilog support for a while, but the tool didn't support the language. I eventually got frustrated enough by this to go and add it myself. The motivation was so that I could set up an internal LodgeIt! pastebin to allow colleagues to discuss snippets of code, without having to paste it into emails, or Yammer discussions, both of which always scramble formatting and make discussions harder than they should be. The pastebin that I decided to use, used Pygments as the backend, so I decided to add SystemVerilog support to that project.

The main task was adding the more than 220 keywords in SystemVerilog into the format required for Pygments. I ended up cutting and pasting the keywords from the specification into a text file, then using a quick script to generate the appropriate structures for Pygments. I found that generating it automatically was about the only way to avoid typos and saved quite a bit of time and typing.

If you'd like to try it out, installation is straightforward:

hg clone https://bitbucket.org/birkenfeld/pygments-main pygments
cd pygments
sudo python setup.py install

After that, you just run the 'pygmentize' command to see formatted source code.

pygmentize hello_world.sv

This sample below comes from the UVM source code 'hello_world.sv' example.

module hello_world;

  import uvm_pkg::*;
  `include "uvm_macros.svh"

  `include "packet.sv"
  `include "producer.sv"
  `include "consumer.sv"
  `include "top.sv"

  top mytop;

  initial begin
    $timeformat(-9,0," ns",5);
    uvm_default_table_printer.knobs.name_width=20;
    uvm_default_table_printer.knobs.type_width=50;
    uvm_default_table_printer.knobs.size_width=10;
    uvm_default_table_printer.knobs.value_width=14;
    set_config_int("top.producer1","num_packets",2);
    set_config_int("top.producer2","num_packets",4);
    set_config_int("*","recording_detail",UVM_LOW);
    //uvm_default_printer = uvm_default_tree_printer;
    uvm_default_printer.knobs.reference=0;
    mytop = new("top"); 
    uvm_default_table_printer.knobs.type_width=20;
    run_test();
  end
endmodule

A further challenge is working out how to embed this sort of code snippet into a TypePad blog post. In general, Pygments generates html code with CSS styles and classes. This doesn't play well with TypePad which seems to consume or remove the CSS styles. You can however get Pygments to generate inline styles, which can then be pasted into TypePad. The syntax for this is:

pygmentize -f html -Ofull,noclasses hello_world.sv

And then the output html can be pasted straight into an HTML editor.

There are comments.

Hobgoblins

Looking over the VPI iterator sample code I posted last night from PyHVL, I noticed something. I've defined the API as follows:

def vpi_iterator(handle, type=vpiNet):

itr = vpi_iterate(vpiNet, mod);

def vpi_iterator(type, handle):

There are comments.

Improving sshfs connections

I'm finding sshfs to be excellent for remotely accessing drives over ssh. There's a couple of tweaks to the ssh config file that can make things even better. If you haven't used an ssh config file, it can be used to store general and connection specific information, to save you typing it every time. The file lives in ~/.ssh/config

Host <hostname>

features

Host <otherhostname>

features

e.g.,

Host short

Hostname ssh.long_url_name.com
Username myusername

ssh short

ServerAliveInterval 15 

There are comments.

Pygmentize files and paste them into Microsoft Word in OS X

Writing some documents using Microsoft Word and I wanted to be able to drop in syntax highlighted versions of the source code. Pygmentize will generate RTF format, which is great for pasting in to Word, but you have to get it there. The best way I've found is using the OS X terminal (I like using iTerm2). I then use a feature of the command line open command, which makes it read from stdin and open the output in TextEdit. This text can then be cut and pasted directly in to Word, preserving all the formatting.

Run the command:

pygmentize -f rtf <file to highlight> | open -f

Then in the TextEdit window, press

a,

c to select all and copy it. Switch to Word and hit

v to paste.

There are comments.

Simplified VPI iterators using PyHVL generators

I've been using PyHVL for a variety of verification tasks in the past few years. PyHVL is an open source Python integration for Verilog and SystemVerilog simulators. To give a quick taste of what it can do for you, consider the following SystemVerilog VPI C code.

void display_nets(mod)
    vpiHandle mod;
    {
       vpiHandle net;
       vpiHandle itr;
       vpi_printf("Nets declared in module %s\n",
       vpi_get_str(vpiFullName, mod));

       itr = vpi_iterate(vpiNet, mod);

       while (net = vpi_scan(itr))
          {
          vpi_printf("\t%s", vpi_get_str(vpiName, net));
          if (vpi_get(vpiVector, net))
          {
             vpi_printf(" of size %d\n", vpi_get(vpiSize, net));
          }
          else vpi_printf("\n");
       }
    }

Here is the equivalent VPI code, this time written in Python, using PyHVL.

def display_nets(module):

    print 'Nets declared in module', get_str(vpiFullName, module) 

    for net in vpi_iterator(module, vpiNet):
        print '\t%s %s' % ( get_str(vpiName, net), 
                get(vpiVector, net) and 'of size %d' % get(vpiSize, net) or '')

The magic happens in the implementation of the vpi_iterator() method, which uses a Python yield instruction to turn the method into a generator. Generators are much like functions, except they maintain a frozen stack frame, at the point where the yield occurs. All existing variables within the method maintain their state and execution picks up where it left off, just after the yield. The example also uses lazy evaluation of the result of get(vpiVector, net) to either call get(vpiSize, net) or not print the 'of size' additional string.

def vpi_iterator(handle, type=vpiNet):
    iterator = iterate(type, handle)
    if iterator:
        handle = scan(iterator)
        while handle:
            yield handle
            handle = scan(iterator)

This lets you write loops inside out as one of my colleagues aptly put it. The outcome is you can simplify the management of loops and indices and focus on the point of the loop. You write less code, you introduce fewer bugs. The code is easier to read and maintain as a result. This is just a very small example of some of the power of using a modern scripting language like Python, as an adjunct to a SystemVerilog simulator.

If you spend much time writing VPI code, you should take a look at PyHVL. It could make your life much simpler, or get in touch and I can help you with it.

There are comments.

csv the easy way

I've recently started using FogCreek's FogBugz project hosting for a small personal project. It has a built-in Mercurial source repository, with some enhancements, called Kiln, and good task/ project tracking features. Quite similar to those you find in Trac. Rather than putting together the various servers to track my own project, I figured I might as well use what they give away for free, for small project teams. I had a list of tasks that I wanted to import into the FogBugz task list, from a CSV file. In fact, this was a CSV file I'd exported from a different FogBugz site, but it could well have been any list of comma seperated values holding task information. The trick was how to get that information from my computer onto the FogBugz site. They didn't have any obvious CSV import option, but they do publish an XML API. Luckily enough, there is also a set of python bindings around the API. Amazingly enough, I was able to open and read the CSV file, parse it, login in to my FogBugz account and upload the new tasks to the server with just this little snippet of code.

from fogbugz import FogBugz
import csv
fb = FogBugz('http://my_site.fogbugz.com/') # URL is to your FogBugz install
fb.logon('my_login@my.site.com', 'password')

cases=csv.reader(open('Filter.csv','r'))
for case in cases:
    print case
    fb.new(sCategory=case[0], sTitle=case[3], sPriority=case[7]) 
fb.logoff()

There are comments.