Python arborealis

02/01/2013

I’ve been implementing a k-ary circularly linked tree (that is, a tree where each node could have a previous, a next, an up, or a down node) in Python.

The page Trees – How to Think Like a Computer Scientist is a nice introduction to some of the background concepts.

To __init__ each node of my TreeNode class you just need to provide the node’s contents and its links

class TreeNode:
    
    def __init__(self,
                 body=None,
                 Prev=None,
                 Next=None,
                 Up=None,
                 Down=None):
        self.body = body
        self.Prev = Prev
        self.Next = Next
        self.Up = Up
        self.Down = Down

I thought hard about getting __repr__ and __str__ right for this class (see Built-in Functions – repr, for example), but __repr__ is a bit tricky if you want to avoid too much recursion. I wrote a small getRepr function for doing this on any class. Members that are instances of the parent are recursed to a depth of one.

def getRepr(self,
            depth=0):
    keys = list(self.__dict__.keys())
    keys.sort()

    fields = []

    for key in keys:

        if (isinstance(self.__dict__[key], self.__class__)):

            if (depth < 1):
                value = getRepr(self.__dict__[key], depth=depth+1)
            else:
                value = ''.join(['<',
                                 self.__class__.__name__,
                                 '(...)>'])

        else:
            value = str(self.__dict__[key])

        fields.append(''.join([key,
                               '=',
                               value]))

    return ''.join([self.__class__.__name__,
                    '(',
                    ', '.join(fields),
                    ')'])

Hence my TreeNode.__repr__ is then really simple, and my class’s __str__ just returns the node’s body

    def __repr__(self):
        return getRepr(self)

    def __str__(self):
        return str(self.body)

Traversing a node goes all the way down to the tip of each branch and then backtracks to the point that a sibling exists. By default my traverse method processes each node in print mode.

    def isLeaf(self):
        return (self.Down is None)

    def traverse(self,
                 process_node=None):
        depth = 0
        node = self

        while (node is not None):

            if (process_node is None):
                import sys
                sys.stdout.write(' '*depth*2 + str(node) + '\n')
            else:
                process_node(node)

            if (node.isLeaf()):
            
                while (node is not None and
                       node != self and
                       node.Next is None):
                    node = node.Up
                    depth = depth - 1

                if (node is None or
                    node == self):
                    break

                node = node.Next
            else:
                node = node.Down
                depth = depth + 1

When it comes to actually populating a tree, my specific application is unusual in that I have a tree already, which is output by an external C program. So I currently have no functions for creating (or deleting) trees – I just read in my external tree from a file and set the links for each node accordingly. As an example (albeit somewhat clunky) of setting up a small tree directly, here’s the tree for the pseudocode 7 = 1 + 2 * 3; end (with the correct operator precedence!)

root = TreeNode()
asgn = TreeNode('=')
root.Down = asgn; asgn.Up = root
lhs = TreeNode(7)
asgn.Down = lhs; lhs.Up = asgn
plus = TreeNode('+')
lhs.Next = plus; plus.Prev = lhs; plus.Up = asgn
one = TreeNode(1)
plus.Down = one; one.Up = plus
times = TreeNode('*')
one.Next = times; times.Prev = one; times.Up = plus
two = TreeNode(2)
times.Down = two; two.Up = times
three = TreeNode(3)
two.Next = three; three.Prev = two; three.Up = times
end = TreeNode('end')
asgn.Next = end; end.Prev = asgn
print(repr(root))
root.traverse()

giving

TreeNode(Down=TreeNode(Down=<TreeNode(...)>, Next=<TreeNode(...)>, Prev=None, Up=<TreeNode(...)>, body==),
  Next=None, Prev=None, Up=None, body=None)
None
  =
    7
    +
      1
      *
        2
        3
  end

In at the Dep End

30/10/2012

The NAG Fortran Compiler has been updated to Release 5.3.1, which includes a few fixes to the module-dependency analyzer. Fortran modules require compilation before they can be Used so generating the correct dependencies for a (GNU) makefile is pretty vital (and tricky).

Here’s an example project having a non-trivial tree of module dependencies

$ cat main.f90
Program main
  Use module6
  Use module5
  Call msub6
  Call msub5
End Program
$ cat module6.f90
Module module6
Contains
  Subroutine msub6
    Use module7
    Call msub7
  End Subroutine
End Module
$ cat module5.f90
Module module5
Contains
  Subroutine msub5
    Use module4
    Call msub4
  End Subroutine
End Module
...
$ cat module9.f90
Module module9
Contains
  Subroutine msub9
    Print *, "OK9"
  End Subroutine
End Module
$ cat module0.f90
Module module0
Contains
  Subroutine msub0
    Print *, "OK0"
  End Subroutine
End Module

and here’s a dumb Python script to make those files

#!/usr/bin/env python

nfiles = 10

for i in range(nfiles):
    file_fo = open('module' + str(i) + '.f90',
                   'w')
    file_fo.writelines(['Module module' + str(i) + '\n',
                        'Contains\n',
                        '  Subroutine msub' + str(i) + '\n'])

    if (i in [0, nfiles - 1]):
        file_fo.write('    Print *, "OK' + str(i) + '"\n')
    else:

        if (i > nfiles / 2):
            m_no = i + 1
        else:
            m_no = i - 1

        file_fo.writelines(['    Use module' + str(m_no) + '\n',
                            '    Call msub' + str(m_no) + '\n'])

    file_fo.writelines(['  End Subroutine\n',
                        'End Module\n'])
    file_fo.close()

file_fo = open('main.f90',
               'w')
file_fo.writelines(['Program main\n',
                    '  Use module' + str(nfiles / 2 + 1) + '\n',
                    '  Use module' + str(nfiles / 2) + '\n',
                    '  Call msub' + str(nfiles / 2 + 1) + '\n',
                    '  Call msub' + str(nfiles / 2) + '\n',
                    'End Program\n'])
file_fo.close()

To create an accurate view of the project’s dependencies for a makefile you first need a dependency analyzer. With nagfor =depend this is quite easy:

$ nagfor =depend -otype=make *.f90
NAG Fortran Dependency Analyser Release 5.3.1(909)
NI_EQ==
NI_SC=\;
main.o:	main.f90
main.o:	module6.mod
main.o:	module5.mod
module0.o:	module0.f90
module0.mod:	module0.f90
module1.o:	module1.f90
module1.o:	module0.mod
module1.mod:	module1.f90
module2.o:	module2.f90
module2.o:	module1.mod
module2.mod:	module2.f90
module3.o:	module3.f90
module3.o:	module2.mod
module3.mod:	module3.f90
module4.o:	module4.f90
module4.o:	module3.mod
module4.mod:	module4.f90
module5.o:	module5.f90
module5.o:	module4.mod
module5.mod:	module5.f90
module6.o:	module6.f90
module6.o:	module7.mod
module6.mod:	module6.f90
module7.o:	module7.f90
module7.o:	module8.mod
module7.mod:	module7.f90
module8.o:	module8.f90
module8.o:	module9.mod
module8.mod:	module8.f90
module9.o:	module9.f90
module9.mod:	module9.f90

Note that no .mod files are required to pre-exist.

For the project above a simple makefile might look like

$ cat Makefile
all: main.r

main.r: main.exe
        ./$< > $@ 2>&1
        cat $@

SOURCES := $(sort $(wildcard module*.f90))
OBJECTS := $(SOURCES:.f90=.o)

main.exe: main.o $(OBJECTS)
        nagfor $^ -o $@

%.o %.mod: %.f90
        nagfor -c $<

clean:
        rm -f *.r *.exe *.o *.mod

but of course that doesn’t take the module dependencies into account yet, so that trying to make results in something like

$ make
...
Fatal Error: main.f90, line 2: Cannot find module MODULE6
...

There are several great discussions around of advanced auto-dependency generation including one (primarily for C source, although some of the ideas are transferrable to Fortran) by Paul D. Smith that inspired the approach below.

Essentially we use GNU make‘s include statement to build in a ‘pre-pass’ that generates a dependency file for all Fortran source in the project:

DEPS := $(SOURCES:.f90=.P) main.P

%.P: %.f90
        nagfor =depend -otype=make $< -o $@

include Depends

Depends: $(DEPS)
        cat $^ > $@

Thus the file Depends is automatically built first for a clean make and it’s updated and re-included into the makefile if any of the dependent Fortran source changes. It will work with -jN parallel make.

Then we see

$ make
...
 OK9
 OK0

The scheme is also reasonably portable, so that other compilers can be used for building the executable – as long as they output .mod files at all. Of course, you may need to postprocess the output from nagfor =depend for compilers that uppercase the names of modules when they create .mod files. Plus it goes without saying that if you’re using make you’ll probably want to follow the rule of one module per file.

How to set up Fedora for work

14/07/2012

I got a new desktop machine at work. Our helpful sysadmin installed Fedora 17 64-bit for me (as a NIS client and all that jazz). This post is a note-to-self about the additional configuration I had to do to finish getting it ready.

I gave myself sudo privileges: as the local admin user who was added during the install,

sudo visudo
# Added me ALL=(ALL) ALL to the who-what-which section

(Reminder: <ESC>:wq saves and quits in vi! I always forget that syntax…)

Alternatively I guess I could have added myself to the wheel group…

I often need to build 32-bit code, and from this 64-bit environment with gcc -m32 I saw

/usr/include/gnu/stubs.h:7:27: fatal error: gnu/stubs-32.h: No such file or directory

To resolve that I needed to install glibc-devel.i686.

I ran into other missing 32-bit components too, namely

/usr/bin/ld: skipping incompatible /usr/lib/gcc/x86_64-redhat-linux/4.7.0/libgcc_s.so when searching for -lgcc_s

which, it turns out, is a somewhat-unhelpful way of saying that I don’t have the 32-bit libgcc package installed. yum provides was helpful here:

sudo yum provides "libgcc_s*"
...
sudo yum install libgcc.i686

Also, when doing a link using -static -lm -lutil, I got other terse messages

#/usr/bin/ld: cannot find -lm
#/usr/bin/ld: cannot find -lutil
#/usr/bin/ld: cannot find -lc

because I didn’t have glibc-static installed.

Other extra packages and setup I needed were:

  • colordiff: I like to have colorised svn diffs;
  • Jenkins. The documented installation instructions are good

    sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
    sudo rpm --import http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key
    sudo yum install jenkins

    That gave me a sandbox installation to mess with;
  • sudo gnome-control-center to set VPN routing (using the IPv4 tab) to access the new machine when I’m outside the work firewall. The resulting configuration is written to /etc/sysconfig/network-scripts/route-Wired_connection_1;
  • edited /etc/exports to export my home drive on the machine to the work network (/mydir *.workdomain.co.uk(rw,insecure,async)) and to the VPN IP. Installed nfs-utils and enabled (at boot) and started the mount daemon to complete the export

    sudo yum install nfs-utils
    sudo systemctl enable nfs-mountd.service
    sudo systemctl start nfs-mountd.service
  • additional Gnome configuration:

    sudo yum install gnome-tweak-tool gnome-font-viewer gnome-shell-extension-alternative-status-menu gnome-shell-extension-remove-accessibility-icon gnome-shell-extension-remove-bluetooth-icon gnome-shell-extension-dock gnome-shell-extension-alternate-tab
  • enabled RPM Fusion free and nonfree repositories using RPM Fusion Configuration and enabled MP3 support in Rhythmbox by installing gstreamer-plugins-ugly;
  • installed dependencies for Spark IM Client

    sudo yum install libX11.i686 libXext.i686 libXi.i686 libXp.i686 libXtst.i686 alsa-lib.i686 unixODBC.i686 compat-libstdc++-33.i686

    (although the Spark 2.6.3 RPM has a broken dependency on libodbc and libodbcinst);
  • installed python3, rdesktop, libusb-devel, lcov, octave, mpfr-devel, libmpc-devel.

Fedora-ing on a Rainy Afternoon

01/05/2012

The weather last Sunday afternoon was purgatorially dreary. I decided I’d try installing the Fedora 17 beta on my feeble and unmiraculous Dell Optiplex 260 at home in advance of upgrading other more important machines when the full release is available, so I downloaded the full i386 install DVD .iso. (Actually, being accustomed to having a poor broadband connection I had set this to download the night before.)

I don’t have any blank DVDs to hand at home, but I do have a nice 16GB Sandisk Cruiser Blade USB stick that I wanted to try installing from. (That’s bound to be kinder on the environment too, right?!?)

There is a good page on the Fedora wiki covering how to create and use Live USB that seemed relevant to this exercise. However, I made the initial mistake of only reading half the page and blindly followed the instructions to make a Live USB image using the LiveUSB Creator.

After RTFW (Reading The Fedora Wiki) a little more, I saw that these weren’t the instruction droids I was looking for; I wanted how to make a bootable USB drive to install Fedora instead of using a physical DVD, further down the page. I ran this advice without a hitch.

Nearly done? Well, I’d forgotten that I couldn’t boot the OptiPlex from USB (and began dimly remembering this rigmarole from Fedora-16 time). It was still dank outside, so I had a look around the web and found some great instructions from Dell which made me realise that if I’d bothered to update my BIOS since 2005 I would have support for booting from USB. Oops.

With the BIOS upgraded and the USB drive inserted I was given the option in the boot menu to boot from USB, and it worked, and the installation (well, upgrade really), seemed to go fine, and here I am writing up these notes using Fedora 17 Beta. Ta-da!

Fedora 17 Wallpaper

I’m pretty certain I used to be able to take screenshots differently with Fedora 16 though, and there are some odd messages when I try to shutdown, … Let’s report some bugs!

Installing Oracle Solaris Studio 12.3

03/04/2012

I only just noticed that Oracle Solaris Studio 12.3 was released last December. I’ve installed the past few releases on Solaris pretty easily (each Installation Guide is helpful), but I thought it would be good to keep a more permanent record of the process here.

First: what’s my machine and OS?

$ cat /etc/release

shows Solaris 10 on SPARC for me.

As root I downloaded the SVR4 *-pkg.tar.bz2 for the correct system from Oracle Solaris Studio Downloads and, naturally, bunzip2ed and untared it. I did this in /tmp/.

Then I ran the included install_patches.sh script. This gave a warning

For patch 147436-01, required patch 118833-36 does not exist.

The machine I’m installing on is probably about 7 years old, and I don’t think many system patches have been applied to it in the past. According to We Sun Solve!, patch 118833-36 is a massive system patch from 2007, while patch 147436-01 is a smaller fix to the linker from 2011. Hopefully we can live without it! Indeed, there seems to be some self-contradictory information from Oracle in the Installing the Required Oracle Solaris OS Patches section of the Studio 12.3 Installation Guide, which says that

…patch 147436-01…is required only on systems running Oracle Solaris 8/11

even though earlier on the page it doesn’t seem to be talking about this particular patch. Hmm. Otherwise, the patches installed fine.

Anyway, we’re ready to run the Studio installer now. For me, /tmp/ is too small, so I see

The /tmp temporary directory does not have enough free disk space for the installer.

plus Java is installed in a non-standard location, so I also see

Java installation was not found on this computer

All this means I have to invoke the installation script as

$ ./solarisstudio.sh --non-interactive --verbose --tempdir /export/home/OSS12.3 --javahome /opt/jre1.6.0_23

which runs successfully.

Dear disk – don’t die

30/03/2012

My trusty crusty home desktop (a Dell OptiPlex GX260) is nearly ten years old, and it’s been a bit flaky on recent occasions when booting up Fedora 16. One time this week I let it have about six attempts before I gave up; it would get stuck – after quite a while – and say something like

udevadm settle - timeout of 120 seconds reached, the event queue contains:
/sys/devices/pci0001:00/0001:00:02.0/0001:01:0b.1/usb3/3-2 (623)
/sys/devices/pci0001:00/0001:00:02.0/0001:01:0b.1/usb3/3-2/3-2:1.0 (624)
/sys/devices/pci0001:00/0001:00:02.0/0001:01:0b.1/usb3/3-2/3-2:1.0/0003:05AC:1000.0001 (625)
/sys/devices/pci0001:00/0001:00:02.0/0001:01:0b.1/usb3/3-2/3-2:1.0/0003:05AC:1000.0001/hidraw/hidraw0 (626)

(Unfortunately I don’t have the exact details to hand. I’m writing this from memory, but the OP’s question at udev reaching timeout is pretty much what was happening to me.)

The Disk Utility in the Accessories area

Accessories, Disk Utility

told me in the SMART Data section for the drive that there are bad sectors

SMART Data, bad sectors

For any repair work to be able to happen the drive has to be not mounted, so I booted from an old Fedora 15 Beta LiveCD I had lying around. I ran

$ sudo lvdisplay

to find my root partition (since it’s on a logical volume), and then ran

$ e2fsck -c my_root_partition

After several days of usage following this, booting seems a lot more stable. Hopefully there are another few years left in the old girl.

libc ya later, alligator

28/03/2012

Yesterday, like a n00by n00b, I deleted my Fedora’s /lib64/libc.so.6 symbolic link. It was intentional; not an act of deliberate vandalism, but neither in retrospect do I think it would have actually helped me achieve what I was trying to. The details of what I was attempting aren’t important. What shocked me is
how destructive this simple mistake was
: remember – I’ve only deleted the symlink, not the actual target library.

Try it yourself!

$ sudo rm /lib64/libc.so.6
$ ls
ls: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory

It’s all gone wrong. You can get some functionality back by setting LD_PRELOAD, e.g.

$ setenv LD_PRELOAD /lib64/libc-2.13.90.so
$ ls
(Stuff)

Yay. Ish: we really want to restore that little symlink though.

$ su
su: error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory

I guess that LD_PRELOAD trick isn’t supposed to work here with stuff like su, otherwise we could load in any old junk library to bypass security, etc.

Is there really no way this can be fixed while the system is still live?

The superuser question How to restore /lib/libc.so.6? helped with the above, but I was hoping to avoid having to continue with the advice there which is to restore using a LiveCD. The problem has such a tantalizingly simple cause I was rooting for a simple solution.

In the end I dug out an old installation CD, but not first without seeing if the system would reboot. Ha!

Kernel panic - not syncing: Attempt to kill init!

and so on and so forth. But in the end sorting the problem out with the installation CD was simple. The rescue option mounted my installation under /mnt/sysimage/, so I just

$ cd /mnt/sysimage/lib64
$ ln -s libc-2.13.90.so libc.so.6

and with a reboot again, everything is back to normal.

I said ‘Hello World!’

28/03/2012

To be precise:

import antigravity
print("Hello World!")


Follow

Get every new post delivered to your Inbox.