Emacs Prelude with Cygwin

Some notes on installing Emacs prelude with Cygwin. To quote the prelude website at http://batsov.com/prelude/

“Emacs Prelude has the goal to ease the initial Emacs setup process and to provide you with a much more powerful and productive experience than the one you get out of the box. “

Although I’ve been using Emacs for nearly fourteen years now I decided to ditch the masses of cruft that had built up over the years and launch into a new setup. Prelude was a nice starting point for me, giving me a nice working environment into which I was able to merge in some of the cruft that I realised I couldn’t live without. Many thanks Bozhidar.

I’m using Cygwin 1.17.7-1 for the record. I am going to assume you can follow the cygwin install instructions and the prelude install instructions. If you do get stuck feel free to ask. I’m no Cygwin expert (or emacs expect) so don’t expect too much ;). This post just lists some stuff you might run into under windows.

First you’re going to need to install Cygwin (obviously) and the following extra packages:

  • emacs-w32
  • curl
  • git
The entire prelude install process runs through a downloaded shell script. curl will give you en error “(77) error setting certificate verify locations”. You can switch this off by doing:
   echo insecure >> ~/.curlrc
Obviously that’s not the right way of doing it. Feel free to let me know the proper way. Similarly you need to tell git to be wild and free about what it’s connecting to. Once again this is NOT the right way to do this but for the record
   git config --system http.sslverify false

The next issue that might trip you up is if your home directory has spaces in it.  My home directory is defined as

   /home/Ian J Cottee/

and that will make the installer upset. In this case I felt the best thing was to fix the problem for good as no doubt other utils outside of prelude might be thrown by this.

   juice: / $ cd /home
   juice: /home $ ls
   Ian J Cottee/
   juice: /home $ ln -s Ian\ J\ Cottee/ ian

I then modified /etc/passwd so the home directory was set to /home/ian

If you now start emacs you should see prelude setting itself up. It’s possible you may need to remove your existing .emacs.d (in my case it got created during one of the failed processes). If you’ve been using emacs before under Cygwin you’ll want to back it up.

The final issue is you may see in the console messages such as:

   0 [main] emacs-w32 9732 child_info_fork::abort:  

and plenty more gumph and in emacs messages such as:

   File error: Doing vfork, resource temporarily unavailable

To fix this follow the instructions at http://cygwin.wikia.com/wiki/Rebaseall

And there you go. You should be in business. Note I’ve only just got it running here. I’ll try and keep this post updated with all the pitfalls I will no doubt run into.

 

 

 

 

 

 

The joy of legacy Zope/ZODB systems

I have a large number of legacy systems that – when all other avenues fail – become my responsibility to sort out. Some of those are very old Zope systems written by others and which never fail to reduce me to tears. This morning I came across some particularly good design decisions which I thought I’d share. Yes, that ‘good’ is sarcasm.

First of all let’s remember that Zope, by default, uses the ZODB. In my past lives when I used to use Zope I used to use it as a frontend to Postgres (which sounds nuts now but at the time we didn’t have lots of fancy MVC frameworks to spend our hours with). The legacy systems I have currently been gifted with however use the ZODB to save their data. You could argue that for content management systems (which is what we’re talking about here) that the ZODB is not a bad fit. I’m not going to argue one way or the other, although my own personal point of view is that it’s shit) – however, for the purposes of today’s “issue” the data was real data – i.e. information that would sit quite comfortably in a relational database as god intended.

We’re talking about parent data with child records. This particular issue was that the customer had made a mistake and I needed to remove a bunch of child records from one particular parent record and this had to be done by the sixth of March or the world would end or something similar. Fair enough, so I poke around the code and the database.

Poking around a ZODB database isn’t like poking around a relational database. You can’t look at a database schema and say “these records in this table have these fields”. All you can say is that this record in this big bucket of crap has these fields. And the record next to it could be the same type of record with different fields. Or a completely different type of record. So you poke around and try to work out what is held where. If you fancy a laugh you take a look at the code that writes the records and try and make some sense of it but frankly you’re better off pulling the nerve endings away from your fingers one by one, with a rusty pair of pliers.

So eventually I find the child records and I find a method which is called something like ‘deleteChildRecord’. It turns out this doesn’t just delete child records – it deletes parent records as well because in this brave ZODB world they’re all sitting in the bucket of slop together. Which is OK in the scheme of things because by this time its only 05:30 in the morning and we’re beyond caring. By the way – there is a doc string on deleteChildRecord but it doesn’t seem to make much sense … at all. Then I realise the same doc string is used to document virtually every method in the file. Somebody copied and pasted the same doc string twenty times and never thought it useful to change it to something else. But that’s OK. We’re used to that.

So I write some code that works out which child records to delete and run it and get the client to check. They respond saying that when they look at the parent record they can still see ‘stubs’ of the deleted child records. The codes of the deleted records are still showing against the parent but no details of the child records. This doesn’t surprise me that much because in my ZODB world it’s very hard to get rid of data. Like the brown stains around the porcelain after a particularly heavy dump, there’s always some lingering remnant that remains in the crusty crevices of the database even after repeated flushings. Often it’s because the indexing system that Zope and it’s content management framework (CMF) uses hasn’t removed it’s indexes of metadata even when the real record has been removed. Because you should never query the database directly (unless you fancy wallowing through 50GB of binary data record by record) code will just query the indexes (known as the catalog) which contains the main info you need and will then pull out the records for each catalog item it finds.

Am I boring you? By the way, it just occurred to me that the best way to ensure that all traces of a particular record are removed from Zope are to tell it that’s it’s vitally important that it should be kept. I can guarantee you’ll never see it again. Anyway – back to the story. Oh god. So I think … the catalog has not updated itself. I will rebuild the catalog. This I do. This is a simple thing to do – you just click on a button labelled ‘Rebuild catalog’.

I now have no data.

You see, whatever genius designed this part of the system decided to store the data in the place where you’re supposed to just store the indexes and metadata of a record. Just as you thought you were winning and the baddie had been despatched to the lower reaches of hell you see a shape in the window and it turns out you’re still thirty minutes from the final credits and you’re not halfway through the body count yet.

Fortunately, like the seasoned campaigner I am, I am not doing this on the live system – having transferred the 50GB bag off poo to a staging site before commencing this exercise. Hell, I don’t log onto a bloody legacy Zope site of ours without taking a backup, dumping it to tape and moving it 100 miles offsite.

Right start again and lets start looking at the code.

I can see code that creates parent records. I can see code that creates child records (all got the same doc strings). I can actually see quite a lot of code that creates child records. And I can see code that deletes child records (with the same doc string). Unfortunately that code doesn’t tell a parent record that the child record has been removed so if you do actually use that routine you’ll find the system fails spectacularly the next time you try and view anything. OK – so … the parent record stores the list of related children as a tuple of id’s. So we write some code that takes the tuple (which is immutable) to a list, and then modifies the list every time we remove a child record, converts it back to a tuple at the end and then writes it back to the parent record when we finish.

But of course that still doesn’t work because things such as ‘child count’ are not calculated automatically – they’re stored as properties as well on the parent record. So we manually count the records and update the child count property but then find that even though we have updated that – we have two other counts which are held as two other fucking properties, one for the two possible type of child records we can have and the parent record is wetting itself in the only way it knows how by vomiting python tracebacks over the screen.

So in short when you delete a child record you have to manually a) tell the parent record the child record has gone b) tell the parent record to decrement a specific count for the type of child record you have deleted and c) tell the parent record to manually decrement a specific count for the total of child records you now have.

This I do – give the results to the client who says it all looks fine apart from one record which they don’t recognise the code for and has a completely different child type assigned to it from anything else and would I know why that was?

No, I don’t. I really don’t. I don’t understand this data, I don’t understand the structure (partly because I don’t think there is one). Frankly I hardly know where the floor is at this point and even such concepts as light and dark have gone hazy.

Python – comparing floats and decimals

No matter how old I get, I keep being bitten by the joys of having some data as floats and some as decimals.

ipdb> value
Decimal('1.473')
ipdb> from_value
1.473
ipdb> value < from_value
True

because …

ipdb> from decimal import *
ipdb> Decimal(from_value)
Decimal('1.4730000000000000870414851306122727692127227783203125')

So work out what accuracy you need and do something like

from_value = Decimal(from_value).quantize(Decimal('0.0001'))

Samsung Galaxy Note – Best phone I have ever owned

My home screen on the Note
My home screen on the Note
 

Some time ago, a meeting was held in Samsung regarding a new product. The design team already knew what they wanted to do. A phone, similar to their successful Galaxy II S. A big phone sporting a 5.3″ screen. In that meeting (or probably meetings) there was probably a lot of worry. Others had done similar and they had all failed. The Dell Streak didn’t set the world on fire (although it didn’t totally flop either) but somebody in that meeting wanted to make a very similar device.

Big phone. Big hardware. And stick a stylus on it as well.

I don’t know who in that meeting made the decision. Maybe it was just one or maybe it was many. But they decided to make it. And now it has been released. The Galaxy Note.

It’s an absolutely awesome phone.

So order one and wait for the delivery man to come. Open the box and look at it. You were thinking/hoping the box has padding. It didn’t. The phone is that high and wide. Calm. Calm. You may have also noticed it’s very thin and light. Now turn it on.

I’ll give you a little time.

Recovered? Yes – that was nice wasn’t it? That screen is a “HD Super AMOLED 1280 x 800, 285 PPI”, Which roughly translated means (I think) “I AM FUCKING AWESOME”. Personally I don’t know what those specs mean. All I know is the phone makes me giggle when I turn it on. Or think about it. I had a very strange experience the other day when I noticed my MacBook Air’s display seemed slightly blurred. After fiddling around I realised it was because I’d been browsing the web on the Note all the time and the razor sharp display was just making a normal laptop screen seem inferior. On a similar note (sic), when I browse the web on this device I don’t think to myself “I’m on a phone”, I just browse the web, it’s a really good experience.

I’ve had the phone for nearly two weeks now and none of my initial thoughts have changed. Its big, but not heavy and sits perfectly weighted in my hand. The screen is bright and crystal clear. The size means that typing on the touchscreen keyboard is a joy, not a chore. Let’s go through the initial fears I had.

Firstly – the size. Would I need to carry it around in a special case? Where can I put it? I bought mine from Clove Technology (http://www.clove.co.uk/samsung-galaxy-note) and it came with the official Samsung cover which replaces the back of the phone and gives you a seamless front cover which works extremely well. I stick it in the front or back pocket of my jeans without thinking about it. I’ve not come across a pocket on my attire it doesn’t fit into. It is big though – here’s a picture of it compared to my previous phone.

23-Nov-2011 03:28, Canon Canon PowerShot A480, 3.0, 6.6mm, 0.017 sec, ISO 160
 
. That size also leads to another fear, will people laugh at you when you pull it out and use it? Well I’ve been using it in public and nobody seems to have noticed. The most attention I got was in an Indian restaurant when I was by myself and browsing the web. Four different waiters also came up, asked me about the Note and complimented it. It does work very well as a phone. Dialling numbers is easy (it’s hard to miss the onscreen buttons when they’re that big), call quality seems good and the speaker is nice and loud if you want to lay it on the desk for hand free use.

Secondly – battery life. That vivid screen along with a 1.4GHz Dual Core Processor would surely mean keeping an eye on the battery life. In fact, it’s the most battery friendly smartphone I have ever owned. It comes with a 2500 mAh battery. As a test I fully charged it and unplugged it at 8am and used the phone normally during the day. Nothing too heavy. I was syncing mail/calendar/google reader every hour in the background. I had wireless and mobile internet on. During the course of the day I took a few phone calls, played some games (maybe 30 mins tops), did some web browsing whilst waiting in the car and at home in bed (maybe another 30 mins tops). 24 hours later the phone was still showing just over 50% full. Last weekend I used it on a four and a half hour work, constantly streaming data to Runkeeper and accessing the GPS. After five hours battery life was down to about 60% – massively better than my previous Wildfire S (a tiny phone that I’d upgraded the battery for, making it (so the label says) the same rating as the Note).

Thirdly – the Samsung Android interface and software. Both my previous Android phones were HTC’s – and I knew that everybody said it’s interface was the best. I have to say, I was still not sure about Android as a platform. The HTC Desire was, without a doubt, the worst phone I have ever owned in my life (totally unreliable, poor battery life, an onscreen keypad which just didn’t do anything for me) and coming from the iPhone 3G, everything just felt a little shoddy. The Wildfire S I replaced it with was a cheap and cheerful phone but the Android world seemed a little more mature then and Android 2.3 a suitable step forward. The Note also comes with 2.3 (an upgrade for 2.4 will come out next year) and I’ve not missed anything from the HTC world in terms of widgets etc. The beginning of this post has a screenshot of my current home screen heavily customised from the stock version. Has everything I want and the resolution means you can fit a lot into it.

The Android world continues to improve in terms of applications. Yes, quality is worse than the Apple store but the speed and resolution of this phone means you have access to some beautiful high res games (To the right is a screenshot of Jelly Defense – also available for iOS, click to see it in its high definition glory).

So any minus points? Not really. It comes with a Stylus so you can draw on the screen (it fits neatly into a slot on the phone) and the only time I’ve ever used it for real was to take the screenshots for this review. If you’re an arty type google for it – it’s not just a piece of plastic – you can ‘do stuff’ with it, but it wasn’t a selling point for me – I could live without it. The size also means that whipping out the phone to take quick snaps as a camera is a little more work than optimum. But I’m really looking for something to criticise now.

In short, this is a big phone and it’s a beautiful phone. Everybody else’s phone now looks small to me. I was waiting for the iPhone 4S but wanted a bigger screen – well I certainly got that and then some. Outstanding display, great battery life and enough oomph to run the latest Android apps (ok, ok – games then), makes this the best phone I have ever owned.

Staticfiles in Django 1.3 not working?

Wasted a lot of time on this today when I shouldn’t have done.

If you find that your staticfiles in Django are just not working for you, when the docs say they should be, try this. The docs say here that:

“This view is automatically enabled and will serve your static files at STATIC_URL when you use the built-in runserver management command.”

Well, it doesn’t work for me. As per the example after I had to amend my urls.py specifically – i.e.:

from django.contrib.staticfiles.urls import staticfiles_urlpatterns

# ... the rest of your URLconf goes here ...

urlpatterns += staticfiles_urlpatterns()

If anyone knows whether this is a bug or just me please tell me.

Django/Postgresql – inserting new record doesn’t return id.

You have an old database and you find the following happens.

>>> ap = ValidAirPressureCode(code='9', description='9', barcode='9', comments='
9'
)
>>> ap.save()
>>> ap.id
>>> ap = ValidAirPressureCode.objects.get(code='9')
>>> ap.id
11

In short Django doesn’t return the id of a newly inserted record. If you insert records in admin and choose to continue editing, you’ll get an error saying it can’t find a record with a null id. The problem (in my case) is that Django uses a postgres function called “pg_get_serial_sequence” to identify where the last inserted number given to a table’s record is stored but if your tables were not created using the serial type, this will return null. In my case it was because the database was nine years old and serial types did not exist then.

A proposed solution can be seen on this ticket but at the time of writing it is not implemented.

To fix it in my case I took a look at what my sequences were called. For example:

id | integer| \
    not null default \
      nextval(('t_valid_airpressure_pk_seq'::text)::regclass)

. I then wrote the following piece of middleware (just inserted a call to it in my settings.py middleware section).

class FixLastInsertId(object):

    def __init__(self):
        """ """
        def my_last_insert_id(self, cursor, table_name, pk_name):
            """ This code fails on the alc database, so we fall back to a method of
            getting the last id that does work"""

            sql = "SELECT CURRVAL(pg_get_serial_sequence('%s','%s'))" % (self.quote_name(table_name), pk_name)
            cursor.execute(sql)
            result = cursor.fetchone()[0]
            if not result:
                sql = "select currval('%s_pk_seq'::regclass)" % table_name
                cursor.execute(sql)
                result = cursor.fetchone()[0]
            return result

        from django.db import connection
        connection.ops.__class__.last_insert_id = my_last_insert_id

        # Tell Django to forget about this middleware now, we have
        # had our evil way.
        from django.core.exceptions import MiddlewareNotUsed
        raise MiddlewareNotUsed

Basically I monkey patch the last_insert_id method. I call the normal code and if that fails try with my own way. All my sequences have the same naming convention (tablename_pk_seq) so it works for me.

Thanks to Ross for pointing me in the right direction, although he didn’t approve of my monkey patching solution 😉

Converting Legacy Databases to Django 1.3 – Day 0.5

I have done this quite a few times in the past, but not recently. I was given three days to convert an existing postgres database (front ended with Zope) to Django 1.3. I thought it would be useful to document what I did here for my own future reference and to record any gotchas for posterity. The database I am converting is nine years old, so plenty of cruft through the years, although the basic structure is sound.

Note that the website of this application is not particularly complicated. Most of the core work of this application is done via backend processes written with Twisted. The front end is used for displaying the system status and allowing for data to be modified, with a few additional complications. The system will work without any web front end (although this is hardly ideal) without stopping production, so the risk is minimised.

This is the first day – although I only started from the beginning of the afternoon, so it’s the first half a day.
Continue reading

Postgres 9.04 on OS X 10.7 Lion

I installed PostgreSQL on my new MacBook Air today via Homebrew. Having installed it I was mystified why I could not connect without specifically specifying the host was localhost. e.g.

yes:postgres ian$ psql --list
psql: could not connect to server: Permission denied
	Is the server running locally and accepting
	connections on Unix domain socket \
            "/var/pgsql_socket/.s.PGSQL.5432"?

Instead I would have to specify “psql –host=localhost –list”.

Turns out that 10.7 has postgres installed by default (or at least bits of it), but not configured to run. The postgres commands that you type are picked up by the default install as they are first on the path (they are in /usr/bin/, whereas the Homebrew binaries appear in /usr/local/bin/). This posting has more info.

I have changed my previous solution (moving the postgres files out of /usr/bin), which was stupid (and will give similar issues with other Homebrew installs. The correct solution is to amend ~/.bash_profile (create if it doesn’t exist) and add the following line:

export PATH="/usr/local/bin:$PATH"

Start a new terminal and you should be good to go.