Running a Javascript function from the address bar

Posted on September 6, 2011 by inductiveload

It’s often convenient to be able to run a chunk of Javascript code from the address bar of your browser. If you do that, you can do anything Javascript stored on the server could do. If you put this code in a bookmark in place of an URL, you get the “bookmarklet“.

The thing to know is the whatever you put after javascript: in the address bar will be executed as Javascript. For example, the code below will simply pull up an alert saying “Hello world”.

javascript: alert('Hello world');

Returning a value

If the code returns a string value, the browser uses that as a new page to load:

javascript:x='Hello world'
javascript:x='<html><body>
<div style=\'text-align:center;\'>3</div>
</body></html>'

If you return an undefined type, the page is not reloaded with the returned variable. This is useful when you want to modify the page content and stay on the page. The easy way to do this is an anonymous function. The other benefit to this is that the scope of variables is limited to the anonymous function, so you won’t stomp on variable used by the original page author.

javascript:function(){var x=3}();

Remember that any assignment to the DOM produces results, so if you are not careful, you could load one of them!

Example: 4od Flash player resizer

A problem I often have with Flash players is that you can’t fullscreen them on one screen of a dual monitor setup and move focus to the other screen without the player minimising again.

A solution is to resize the player element. The Channel 4 4oD service has a button to give a larger player, but it’s barely half of a large monitor. Looking at the page source, we can trace back to the code that is called to produce this effect, add our own scaling parameter and calling it from a bookmarklet.

javascript:var ratio=2;
var flashPlayer=document.getElementById("catchUpPlayer");
 
if(flashPlayer!=null&&flashPlayer.resizePlayer){
    flashPlayer.width*=ratio;
    flashPlayer.height*=ratio;
    flashPlayer.resizePlayer(flashPlayer.width,flashPlayer.height-37);
}

I have added this to a Firefox keyword “4odr”, which means I can type just “4odr 2″ in the address bar. All you have to do for this is to replace the 2 in ratio above with %s

See more bookmarklets

Posted in Uncategorized | Leave a comment

Easy thread pools in Python with threadpool

Posted on August 31, 2011 by inductiveload

Some computation tasks fall into the category of “embarrassingly parallel”. There are no weird inter-dependencies on the execution: you can do it in any order and with as many separate processes as you like. Examples of this kind of processing might be batch processing image files, batch downloading files, or appending “on wheels” to the end of Wikipedia articles.

Python has several threading libraries, including threading and multiprocessing. These are fantastically flexible, but sometimes you just want it to be easy. A common pattern is the thread-pool, where you have a specified number of threads that may run at the same time (say, 4). If you run too few threads, you have idle capacity (CPU, network bandwidth or IO bandwidth), and if you run too many, the resource contention harms throughput. So, you have your fixed pool size and now you want to spin off each thread with the same task, but giving different parameters to each.

You could do this with the threading module, a while loop and a counter, but it would require book-keeping. The threadpool module does all of this for you. It is available on PyPi, so you can use pip (or easy_install if you are living in the past) to install it.

Fire and forget threads

If you don’t need a response from the threads (perhaps they are going to do file conversions) you can do that really easily:

  1. Create the thread pool
  2. Create the list of arguments to pass to the workers
  3. Create a list of requests using makeRequests()
  4. Hand these into the thread pool
  5. Wait for them to execute, or move on and forget them!
import threadpool
import time #to simulate a long process
 
def long_process(argument): #simulate a long process with args
    print( 'Starting process with argument %d' % argument)
    time.sleep(argument) #do lots of processing using our argument 
    print( 'Ended process with argument %d' % argument)
 
def process_tasks():
    pool = threadpool.ThreadPool(4)
 
    argument_list = [1, 6, 2, 4, 5, 6, 7, 8]
 
    #let threadpool format your requests into a list
    requests = threadpool.makeRequests(long_process, argument_list)
 
    #insert the requests into the threadpool
    for req in requests:
        pool.putRequest(req) 
 
    #wait for them to finish (or you could go and do something else)
    pool.wait()
 
if __name__ == '__main__': 
    process_tasks()

Threads with callbacks

Coming soon

Posted in Python | Leave a comment

Multi-line LaTeX expressions in matplotlib

Posted on May 28, 2011 by inductiveload

I have been playing with matplotlib recently, and I was immediately drawn to the ability to use LaTeX as the text renderer. However, I quickly found that multi-line equations are not handled well, and choking on newlines is difficult to overcome, since we also need to escape the LaTeX commands like \alpha.

How to break LaTeX’s lines

The only thing you need to do to enable LaTeX equations is the following:

matplotlib.rc('text', usetex=True)

How great is that? Anything you write for text in your graphics will now be rendered with LaTeX, so you can write something like $\mu=3$ and have it rendered lovingly by LaTeX! However, the picture gets less bright on closer inspection.

Display style equations

An in-line equation is denoted by either $eqn$ (the TeX shorthand) or by \(eqn\). On the other hand, display-style equations that you will need to specify for many multi-line equations are denoted by $$eqn$$ or \[eqn\].

You will know if you should be using the display-style equation, because you will get an error along the lines of 

! Package amsmath Error: \begin{split} won't work here.

All you do in this case is switch to the display-style form. But the “$$” style doesn’t work in matplotlib. If you write this, you will get errors such as:

! You can't use '\end' in internal vertical mode.
! Missing $ inserted.
! Display math should end with $$.

In this case, all you do is switch to the \[eqn\] style, and the problem will vanish.

The code to make multi-line equations

The next step is to find some code that formats a multi-line equation how I want it: with the equals signs aligned. I chose to use the “split” environment provided by the “amsmath” package.

\[
\begin{split}
    \mu+ \frac{2\pi}{3} & = \frac{a}{b} \\
    & = 4
\end{split}
\]

You can check that this works at Troy Henderson’s excellent Latex Previewer. Unfortunately for me I got bitten by the following error when I tired to use it 

! LaTeX Error: Environment split undefined.

All that is missing here is the “amsmath” package, which contains our “split” environment. You need to instruct matplotlib to add to the preamble for you:

matplotlib.rcParams['text.latex.preamble']=[r"\usepackage{amsmath}"]

Escaping LaTeX commands

LaTeX uses backslashes to indicate commands like “\begin{split}” and “\alpha”, and the backslash is an escape character. We do not want to have to escape every backslash in our LaTeX, as this will be a huge pain. So we also need to use r'Raw strings' in Python to prevent interpreting of the backslashes.

Failure to do this will produce errors like the following, caused by weird products of escaped characters reaching LaTeX:

RuntimeError: LaTeX was not able to process the following string:

Entering more that one line of LaTeX

I can get a bit obsessive about keeping equations tidy, and I hate cramming it all one one line. Even though LaTeX doesn’t care about white-space, I certainly do! I must have been using Python too much…

If you have been following along in Python as we go here, you’ll notice that you cannot enter the above LaTeX as a simple string, even in triple-quotes. The reason for this is beyond me, as LaTeX shouldn’t care about whitespace handed to it. But it does. If you are passing a triple-quoted string, you will get complaints about extra braces:

! Argument of \split has an extra }.

So, how are we to combine the desire to break our Python-based LaTeX up into tidy lines, while still escaping the backslashes? A raw string cannot be continued across a line break by a backslash-linebreak, as those characters are just considered part of the string. However, we can take advantage of the implicit concatenation of strings in Python:

r'\['
r'\begin{split}'
r'\mu+ \frac{2\pi}{3} & = \frac{a}{b} \\'
r'& = 4'
r'\end{split}'
r'\]'

Here, the line breaks we use to neaten the code are not part of the string, so they will not affect the LaTeX parser, and every string is raw, so we don’t need to escape the backslashes. It is a little cumbersome, but it is much better than converting every “\” to “\\” in your LaTeX code.

Putting it all together

Let’s now look at a timy program to use all of these points together:

import matplotlib
import matplotlib.pyplot
 
matplotlib.rc('text', usetex=True) #use latex for text
 
#add amsmath to the preamble
matplotlib.rcParams['text.latex.preamble']=[r"\usepackage{amsmath}"] 
 
#insert a multiline latex expression
matplotlib.pyplot.text(0.2,0.2, 
 
    r'\[' #every line is a separate raw string...
    r'\begin{split}' #...but they are all concatenated by the interpreter :-)
    r'    \mu+ \frac{2\pi}{3} & = \frac{a}{b} \\'
    r'    & = 4'
    r'\end{split}'
    r'\]',
 
    size=72) # make it big so we can see it
 
matplotlib.pyplot.show()

Amazing how much messing around can go into a 13 line program, when 6 lines are the same string! Below is a screen-shot of the output:

Multi-line LaTeX equation in matplotlib

Multi-line LaTeX equation in matplotlib

As always, if you have a better way to do it, please get in touch!

Posted in LaTeX, Python, matplotlib | Leave a comment

Basic use of GNU Stow/xStow.

Posted on February 27, 2011 by inductiveload

I install a lot of programs form source on my machine, and I have long been worried at a low level about how this will clutter the /usr/local/... directories on my system with many files. This is a problem that package managers are supposed to solve, but if you build from source, you don’t get a nice package. Once I have built and installed a program, what will enable me to remove all the files relating to it again? Sure, maybe I can use make uninstall, but only if the makefile has the uninstallation rules. That’s not much of a consolation.

What is Stow

I recently learned of the existence of GNU Stow (and it’s cousin, xStow), which are tools to keep your system clean, even when building from source. They are more flexible and straightforward in my opinion than CheckInstall, which creates a package to install using the normal package manager, but doesn’t work all the time.

You install your program to a separate folder hierarchy, which is used only by that program. Stow then creates symlinks from the “path” directory (the place you normally install to, often /usr/local) to the files in the program’s special directory. The symlinks relating to a certain program can then be easily identified by looking at which directory they point to, and stow can then remove them automatically at will. Thus, while your /usr/local/... folder will have the same number of files (actually symlinks to files), organised in the same way they used to be, every one of them will be bound to a program via the symlink target.

The diagram below shows the structure of a stow hierarchy based around /usr/local (showing only bin, lib and share). The files belonging two applications, “app1″ and “app2″ are identified by which application directory they point to, indicated by the colour of the arrow.

Stow workflow

Consider the “usual” installation procedure:

./configure
make
make install

When you build your program, you use the prefix options for ./configure and make install commands, which will install the program to a separate directory, keeping all the files associated with that program in their own folder hierarchy. Not all programs need the prefix in both steps, but it won’t hurt to provide it.

./configure --prefix=/usr/local/stow/appname
make
make install prefix=/usr/local/stow/appname

Now, change to the directory specified, and you will see that you have the familiar folders such as bin, lib, include, share etc. Which you have depends on the program. Of course, they are not in the /usr/local directory, so the program can’t be run, as the current directory isn’t on your PATH. The program is installed, but is not findable.

To create the symlinks from /usr/local/bin and friends, go to the /usr/local/stow directory, and run the following command:

stow appname

That is it, your program is now installed into /usr/local, but the files are safely filed away in their own folder. To remove the program again, just go to the stow directory again and enter:

stow -D appname

This will scan for the symlinks targeting the appname directory, and delete them. Notice that access to the original source and makefiles is not needed. It will not delete the directory itself, so you can “uninstall” temporarily without removing the files, so no rebuild is required.

On directories and environment variables

You do not need to use /usr/local for your installed programs. Stow will work just as happily with a directory in your home directory such as ~/opt. In that case the Stow directory will be ~/opt/stow, and you’d install to ~/opt/stow/appname. Of course, whichever directory you choose needs to be on your PATH environment variable for the programs to work.

A useful tip is to create a STOW environment variable, so that you can run the following commands:

./configure --prefix=$STOW/appname
make
make install prefix=$STOW/appname
cd $STOW
stow appname

Uninstallation is as simple as:

cd $STOW
stow -D appname

I find this helps visualise what is going on. You can add this command with the following line in your .bash_profile script:

export STOW=/usr/local/stow

Installing Stow using Stow

If you have a system without a package manager, and you build from source, then you can use the binary you just build to stow Stow, so to speak. You perform the same build steps as before, but in the last command, reach down into the Stow directory and use the binary directly. It hasn’t been linked into a directory on the PATH, so you can’t just call stow.

./configure --prefix=$STOW/stow
make
make install prefix=$STOW/stow
cd $STOW
./stow/bin/stow stow

About Stow and xStow

GNU Stow is a Perl script; xStow is a C++ program, which means you can install it using only a C++ compiler, and you don’t need a Perl installation to begin stowing things (like Perl). Thus xStow may be easier to bootstrap than Stow on a bare system.

xStow also supports various other operations, some of which are not compatible with stow. Thus if you use that operation with xStow, you cannot then change back to Stow. if you don’t do those operations, the two are equivalent.

Posted in Computers | Leave a comment

GTK locks up on modification of graphics context

Posted on February 14, 2011 by inductiveload

Another problem I had when writing a GTK+ program was that modifying the graphics context of a Drawable (in this case, DrawingArea) caused the program to lock up and send the CPU to maximum in an infinite loop. This problem has been encountered in 2008 in the PyGTK mailing list, but was never answered.

The problem

When you change a Drawable’s graphics context in the callback from the expose-event signal, the program locks up and CPU usage becomes very high. This change in my case was a call to modify_fg() or modify_bg(), which emits another expose-event, and causes an infinite loop.

Basically, the problematic code was this:

def set_pen_colour(self, colour):
    """Set the drawing area pen colour"""
    gc = self.drawingArea.get_style().bg_gc[gtk.STATE_NORMAL]
    gc.modify_fg(colour)  # color of rectangle
 
def on_expose_event(self, drawingArea, event):
    set_pen_colour(self.fgColour)
 
    #do redrawing stuff
 
def setup_widgets(self)
 
    #setup drawing area, window, etc etc
    #.....
 
    #connect the expose-event signal to the redrawing code
    self.drawingArea.connect('expose-event', self.on_expose_event)

The solution

Make a new graphics context at the start of the program and use that instead of the one attached to the Drawable. You derive the GC from the window, so you need to wait until you called show() on the window before you can get your GC.

def set_pen_colour(self, colour):
    """Set the drawing area pen colour"""
    self.drawingGC.modify_fg(colour)  # color of rectangle
 
def on_expose_event(self, drawingArea, event):
    set_pen_colour(self.fgColour)
 
    #do redrawing stuff, using self.drawingGC
 
def setup_widgets(self)
 
    #setup drawing area, window, etc etc
    #.....
 
    #connect the expose-event signal to the redrawing code
    self.drawingArea.connect('expose-event', self.on_expose_event)
 
    self.window.show()
 
    #get a new GC to use for the drawing
    self.drawingGC = self.drawingArea.window.new_gc()

That’s really all there is to it. It’s also easier to handle your GC without needing to derive it from with the DrawingArea all the time.

Posted in GTK | Leave a comment

Scrollable images in GTK

Posted on February 11, 2011 by inductiveload

As my first real foray into GUI programming (not counting Delphi GUIs at university), I have been messing with GTK recently. Again, I repeatedly ran into many things which may be obvious to those in the know, but not so clear to those of us temporarily without Internet, clutching only a copy of the PyGTK class reference and the Python docs.

This post (and probably most future GTK-related posts) will deal with PyGTK (since it is so simple to demonstrate things), but it is equally applicable to vanilla GTK, you’ll just need to fiddle with the function names and so on.

The first problem I had was to present a large (3000+ pixels on a side) TIFF image in a DrawingArea widget. Clearly, I needed scrollbars, so the DrawingArea widget was inside a ScrolledWindow. I then had an on_redraw() function which handled the redrawing of the DrawingArea, which is called whenever the window needs redrawing, or when I wanted to paint on top of the image.

The wrong way

The redrawing code contained this line to load the image from a Pixbuf saved as a variable back into the DrawingArea to “reset” it:

self.drawingArea.window.draw_pixbuf(gc, self.pixbuf, 0,0,0,0)

You can see the problem this will cause—this code will load a lot more of the image than we need, so much that the program will be significantly slower. At a window size of 300×300, this will load about 100 times more image data than we need. We need to load only that which we need to load.

The right way

The class reference for a gtk.gdk.Drawable says “The draw_pixbuf() method renders a rectangular portion of a gtk.gdk.Pixbuf…” which immediately tells use that this function will do more than paste the whole things into the Pixbuf.

The last four parameters in the draw_pixbuf() function are, respectively, src_x, src_y, dest_x, dest_y, width, height, and these are the parameters we will use to tell GTK which parts of the image we want to copy in.

First, we need to know the horizontal and vertical offset of the top-left corner of the visible part of the image. This will tell us where to start pasting our image from. We can do this by interrogating the values held in ScrolledWindow’s VAdjustment and HAdjustment using get_vadjustment().value and get_hadjustment().value. We need to convert to an int, because we will shortly use this as a pixel index.

vertOffset = int( self.scrolledWindow.get_vadjustment().value )
horzOffset = int( self.scrolledWindow.get_hadjustment().value )

The second bit of information we need is the size of the visible window of the ScrolledWindow. We do this by looking at the page_size attribute of the two adjustments:

visibleHeight = int( self.scrolledWindow.get_vadjustment().page_size )
visibleWidth = int( self.scrolledWindow.get_hadjustment().page_size )

We now have all the information needed to paste in just enough of the image to cover the whole window, but not a pixel more.

self.drawingArea.window.draw_pixbuf(gc, self.pixbuf,
                 horzOffset, vertOffset,
                 horzOffset, vertOffset,
                 width=int(visibleWidth), height=int(visibleHeight))

And that is it! If you only needed to redraw a smaller area (for example, blitting), you can tweak these exact same parameters to do that for you.

Example

This can make the difference between an unusable program and one with no perceivable lag. This sample program demonstrates this. You need to edit the source to point to some large image on your system, then run it with

python scrollable-image.py

You will need Python and PyGTK to run this code.

Screenshot of the PyGTK scrollable-image example.

Posted in GTK | Tagged , , , , | 2 Comments

Install IronAHK on Linux

Posted on December 17, 2010 by inductiveload

The one thing I have sorely missed since migrating to Linux has been AutoHotKey (AHK), a fantastic desktop automation and scripting program which lets you autocorrect spelling in any program, script hotkeys for anything and general greatness. However, it has always been a Windows-only deal, which is a great shame. There was an attempt to use Python called AutoKey, which, as well as being a terrible name for searching on as AutoHotKey was pulled into the result, was never as effective or reliable as AHK.

At the time of my migration, there was a side project called IronAHK—a C# implementation that used .NET or Mono for cross-platform functionality. I just got round to installing it and since documentation is a bit thin on the ground (it is alpha-release software), it took me a while to find out how to go about it.

Nightly builds

The nightly builds are precompiled IronAHK builds. This will give you access to the most recent features, but it will expose you to bugs and general edge-bleeding.

Preparation

First off, you’ll need Mono to run the program. You can get this by (if you use apt-get):

sudo apt-get install mono monodevelop monodoc

There are other things you might want, depending on how much you want to get involved in the project. More on mono packages here.

You will also need fromdos to convert a Windows text file to a Unix-style one:

sudo apt-get install tofrodos

Getting and installing IronAHK

Now we are ready to get IronAHK and install it. cd to the directory where you want to download and unzip the source (I like ~/src). Then do the following:

wget http://www.ironahk.net/download/nightly/zip
unzip zip
rm zip
cd IronAHK
chmod +x setup.sh
fromdos setup.sh
sudo ./setup.sh install

IronAHK should now be installed, and we can use the example .ahk script, Example.ahk to test it. This script consists of a single command invoking a message box reading “Hello World!”. We are going to use ironahk to compile it to an executable and then run the executable:

ironahk --out example.exe Example.ahk
./example.exe

You should see a message box appear with the text “Hello World!” and an “OK” button. Now the real sport begins—you need to write you scripts to do what you want. I have a large set for inserting special characters, expanding common text snippets, correcting spelling and starting programs with hotkeys. More on this later.

Thanks to Frankie n-l-i in this thread for those commands.

Compiling from source

If you want to compile from source, you need to download and unzip the source archive. You can get a list of releases from here, or you can get the most recent Git version if you want to do some hacking.

You will also need mono-xbuild to build the C# code.

wget http://download.github.com/polyethene-IronAHK-v0.7-0-g0ecb425.zip
unzip polyethene-IronAHK-v0.7-0-g0ecb425.zip
sudo chmod +x -R polyethene-IronAHK-0ecb425
cd polyethene-IronAHK-0ecb425
make
sudo make install

And that should be that!

Posted in AHK, Tutorials | 5 Comments

Change the XScreenSaver icon

Posted on November 2, 2010 by inductiveload

XScreensaver is an excellent screensaver/screen locker program with an admirable concern for security. This concern has led to the locking dialogue to be ugly, since no high-level toolkits are used to prevent a unknown bug from crashing the screen locker and providing access to the “locked” computer.

Current XScreenSaver logo

A lot of this ugliness comes from the icon, which looks like the EPS vector file was drawn by a child. There is a technical limitation that means the XPM file that is generated can have at most 90 colours (due to the fact that this is the most that can be represented by a single printable ASCII character), but even so, I think we can do a lot better. If only the icon can be made to look presentable, I think the prompt screen can be made almost tolerable!

The author of XSS has left a request in the utils/logo.c asking to not change the logo since it is “the identity” of XSS. However, it is FLOSS, running on my computer and, moreover, it is my eyes that have to deal with the old logo, so I am doing it anyway. If the author is looking, the logos here are public domain and you are welcome to them if you want them.

Since the icons are hard-coded as XPM files into the source code, we’re going to have to compile from source. However, in XScreenSaver’s case, it’s not too hard. First, here are my new icons, in the 180px and 50px sizes that seem to be required. These icons are released into the public domain, so do what you like with them. They’re not great but they’re a damn sight better than the old ones:

These PNG files are generated directly by Inkscape’s “Export Bitmap” function.

Generating the XPM files

XScreenSaver needs the XPM version of these images, and we are going to use ImageMagick to get them. XSS has only a cut-down XPM file parser that only has a subset of the XPM specification, so we need to tweak some things.

In an ideal world, we would run something like the following command to convert all the PNGs to XPMs and at the same time reduce the colour palette to 90 colours.

mogrify  -colors 90 -format xpm *.png

If you try it however, you’ll see that it produces nasty black backgrounds in the interaction with the transparent PNGs. We don’t want to lose the transparency, so we convert via the GIF format. It’s called ImageMagick for a reason: it doesn’t always make sense!

mogrify -format gif *.png
mogrify  -colors 90 -format xpm *.gif

If you simply do this, however you may find XSS will choke on the XPM files in two ways.

Firstly, XPM file name-mangling will cause the compiler to be unable to find the variable with the XPM data. The second line of an XPM file contains the declaration of a variable like this:

static char *logo_50_xpm[] = {

ImageMagick mangles the name of the XPM file to produce a C-compatible variable name inside the file by turning all non-alphabetic characters to “_”. This causes a problem because XSS is expecting to see “logo_50_xpm” and all it gets is “logo___”. You can fix this by either modifying the XSS source where it looks for the variables in utils/logo.c, the XPM files themselves (easiest) or you can be hardcore and recompile IM to do it like the GIMP and turn all non-alphanumericcharacter to “_” so we get the right name. See this IM forum post for how to do that. If you have a new version of IM you may or may not need to do this as there is a patch in the system as of today.

The second, more serious problem is that the XPM parser in XSS can’t handle X11 colour names like “gray70″ that might get generated by IM’s XPM export filter. This gives an error message of unparsable XPM color spec (at run-time). This is no fault of IM since these names are in the XPM specification: it is the problem of XSS’s weedy XPM input filter, which is so simple to keep it easy to security-audit. Instead of a complex hack of XSS, however, it is easier to modify IM to not look up colour names and just give you a hex number each time.

Looking in coders/xpm.c, in the WriteXPMImage function change the QueryMagickColorname function call around line 992:

(void) QueryMagickColorname(image,&pixel,XPMCompliance,name,

to

(void) QueryMagickColorname(image,&pixel,NoCompliance,name,

Recompile IM and run the commands I gave above to convert the image, and the XPM images should be digestible by XSS. Remember to copy the two XPM files to “utils/images/” in the XSS source and then recompile. After this you should be able to restart XSS and see your shiny new logo instead of the old one!

One further thing you can do is to copy the larger XPM file to /usr/share/pixmaps to give a higher resolution there.

Image notes

For those who just want the files and don’t want to mess with the image conversion:
All image files in tar.gz

The flame art is public domain clip-art from OpenClipArt.org by valessiobrito.

Posted in Icons, ImageMagick, Tutorials, XScreenSaver | 1 Comment

Multi-colored lines in GNU Octave/MATLAB

Posted on September 17, 2010 by inductiveload
Line plot of the Rabinovich-Fabrikat equations, with varying colour along the trajectory

Figure 1: Line plot of the Rabinovich-Fabrikant equations, with colour varying by distance along the trajectory.

Problem: we want to plot a line in 2D or 3D, varying the colour along it according to some function. In the simplest case, this could be changing the colour from the beginning to the end, or it could be another function. For example, the image below shows the evolution of the Rabinovich-Fabrikant chaotic system in 3D, with the line colour varying with time, as shown in Figure 1.

How not to do it

For this, let us assume that we have three vectors of length n: x, y, z, representing the line coordinates at each step.

You can’t use plot3() for this because the line in plot3() is all one piece, and any line colour option will affect the entire line. You could do the following, but it is slow, and breaks the line into n pieces:

hold on;
cmap = jet(n)
for i = 2:n,
    % plot each segment separately
    plot3([x(i-1), x(i)],[y(i-1), y(i)], [z(i-1), z(i)],'color',cmap(i,:))
end

How to do it

To do it, we will instead use surface() in a cunning way. Let’s make it a function to simplify the calling:

function handle = coloured_line_3d(x, y, z, c)
 
handle = surface(
    [x(:), x(:)],
    [y(:), y(:)],
    [z(:), z(:)],
    [c(:), c(:)], % the colour vector
    'EdgeColor', 'flat',  % colour the edges with flat shading according to the colour vector
    'FaceColor', 'none'  % don't colour the face (which is zero area)
);
 
endfunction

Now to produce the plot, we call it like so:

[x,y,z] = compute_my_data() % get your 3D data from some function
c = 1:length(x)' % increasing vector to cycle though the colourmap
 
colormap (jet(npts)); % set the plot colourmap
figure = coloured_line_3d(x,y,z,c); % plot the coloured line

Line plot of the Rabinovich-Fabrikat equations, with varying colour along the trajectory

Figure 2: 3D Line plot with colour varying by point index along the trajectory.

Of course, we can can change the colouring function to represent a different metric, for example distance from the origin. We can also use whatever colourmap we please:

colormap(cool(npts));
c = (x.^2 + y.^2 + z.^2).^0.5;
Rabinovich-Fabrikant trajectory, coloured by distance from origin

Figure 3: Rabinovich-Fabrikant trajectory, coloured by distance from origin.

2D plots

This is all very well, and surface() lends itself well to 3D plots, being a 3D function. But what if we want a 2D plot? You can simply use the same function as before:

[x,y] = compute_my_data() % get your 3D data from some function
z = zeros(length(x),1); % set the z-values to be all zero
c = 1:length(x)' % increasing vector to cycle though the colourmap
 
colormap (jet(npts)); % set the plot colourmap
figure = coloured_line_3d(x,y,z,c); % plot the coloured line

Rabinovich-Fabrikant equations trajectory projected onto the xy-plane.

Note: this is a hack because the output is a 2D curve floating in 3D space. However, it does lend itself neatly to creating “projections” of the curve:

zero = zeros(length(x),1);
 
hold on
coloured_line_3d(x, y, z, c); %plot the 3D curve
coloured_line_3d(x, y, zero.-1, c);
coloured_line_3d(x, zero.-1, z, c);
coloured_line_3d(zero, y, z, c);
 
view(231,45);

Rabinovich-Fabrikant equations with projections.

Posted in MATLAB, Uncategorized | 2 Comments

Simple XML parsing with Python and LXML

Posted on September 6, 2010 by inductiveload

One of Python’s primary uses is data munging, and it has some very powerful libraries available for this. XML is a common format for data storage, and LXML is a Python library to do all sorts of magic with it. Basic XML parsing with Python and LXML appears to have few simple examples on the Net. The official documentation is certainly complete, but dauntingly rigorously so.

Here’s a quick example of how to extract information from an XML file in Python. Imagine we are parsing an IM message log consisting of records with information in both the “tags” and “body”:

<Message Date="2010-06-09" Time="13:50:48">
    <From>
        <User FriendlyName="Alice"/>
    </From>
    <To>
        <User FriendlyName="Bob"/>
    </To>
    <Text>Hi!</Text>
</Message>

We want to extract, for every record, the date, time, sender, recipient and message text. We don’t care about anything else, even though the XML file may have additional data. We just want the “Messsage” records.

Retrieving the data

Firstly, we use findall() to generate our list of every “Message” record in the XML document. We use a double-slash to tell LXML that we don’t care whether the records are sub-records inside other records such as “Session” or what-have-you—we just want all the “Message” records in the document.

messages = xmlData.findall("//Message")

Then we iterate through the list, parsing each individual record. We have data in the “tag” of the “Message” record (such as “Date” and “Time”). A record’s attributes in LXML is a dictionary called record.attrib, so we can extract “tag” data by indexing this attribute dictionary appropriately:

msgDate = msg.attrib['Date']
msgTime = msg.attrib['Time']

Next we want to get the “From” and “To” data. This is more complex, since it is stored in the tag of a record two levels down: Message→From→User. We can use find() to drill down from the “Message” record, locate the first suitable “User” record for us, and then get our data out of the attribute dictionary as before:

msgFrom = msg.find("From/User").attrib['FriendlyName']
msgTo   = msg.find("To/User").attrib['FriendlyName']

One word of warning here: you can’t use msg.find("//User") or similar absolute paths here: you can only do that with the entire document. It must be relative to the msg object.

Finally, we want to get the message text. This is different to the previous cases because the relevant data is stored in-between the <text> and </text> tags. We use LXML’s findtext() function to do it:

msgText = msg.findtext("Text")

And there we have it: we have all our data, and can now print to console or write to file as we please!

print  '%s, %s: %s to %s: %s\n'%(msgDate, msgTime, msgFrom, msgTo, msgText)

Example code

from lxml import etree
 
inFile = "imLog.xml" 
xmlData = etree.parse(inFile) #etree.parse() opens and parses the data
 
# find all "Message" records in the document, regardless of level
messages = xmlData.findall("//Message") 
 
for msg in messages:
    #first, get the data in the "tags" of the record
    msgDate = msg.attrib['Date']
    msgTime = msg.attrib['Time']
 
    # then, find the from and to users, and get the "FriendlyName" attribute
    # notice how we drill down though the "From"/"To" level to the "User" record
    msgFrom = msg.find("From/User").attrib['FriendlyName']
    msgTo   = msg.find("To/User").attrib['FriendlyName']
 
    # then find the "Text" record, and get the content between the tags
    msgText = msg.findtext("Text")
    print  '%s, %s: %s to %s: %s\n'%(msgDate, msgTime, msgFrom, msgTo, msgText)

Namespace

Some XML documents have a namespace defined, and you need to take this into account when parsing them, otherwise find() and friends won’t find the records you want. Let’s consider SVG files, which are stored as an XML document, and which have a namespace of http://www.w3.org/2000/svg.

Say we are looking for all the groups inside the SVG; these are records like <g>stuff inside the group</g>. Then we can include the namespace in the string passed to the findall() method:

groups = svgXml.findall("//{http://www.w3.org/2000/svg}g" % SVG)

This is a little unwieldy, so let’s break out the namespace into a variable:

SVG_NAMESPACE = "http://www.w3.org/2000/svg"
SVG = "{%s}" % SVG_NAMESPACE
 
groups = svgXml.findall("//%sg" % SVG) # find all the groups

After that, we can access the group attributes and contents as normal.

Posted in Python | Tagged , , | Leave a comment