Use your favorite editor to edit encrypted files
salgado
In the Linaro Infrastructure team we have several shared credentials for external services used by one or more of the projects we maintain. Recently, Paul started a discussion about the best way to store those credentials securely while still making them accessible to everybody in the team. We agreed that one reasonable way to do so would be to store them in a gpg-encrypted file (with keys from each of us), stored in a private branch in Launchpad, so I cooked a small script which will decrypt a file, open it in your favorite editor and encrypt it again once you're done. I'm sharing it here as I figure it might be useful to somebody else.

In its current version you have to specify the name of the Launchpad team for which the file will be encrypted, but it'd be trivial to change it to either use just your own key or a set of keys you pass to it. Also, since it gets the list of people for which the file will be encrypted from Launchpad, it takes a few seconds to complete after you're done editing. Oh, and you'll be asked to confirm the keys belong to the people you want to share the file with, so do yourself a favor and double-check them before hitting 'y'.

Pulling data from Gerrit using JSON-RPC
salgado
At Linaro we want to get some metrics for the patches we submit upstream, so we've built a system based on Patchwork that parses email messages to extract patches and categorize them by project/author. This works fine for projects that use mailing lists to keep track of patches (e.g. the kernel, qemu, etc), but it doesn't work for one project to which Linaro has already contributed and expect to contribute a lot more: the Android Open Source Project, which uses Gerrit to track changes.

For that reason we decided to pull that data directly from AOSP's Gerrit instance into our system. Unfortunately, the only public API provided by Gerrit is this one over SSH, which doesn't give a piece of data that is very important to us: the date a change was proposed.

Luckily, James pointed me to this discussion where a neat trick is suggested: watch the requests your browser sends when rendering http://review.source.android.com to figure out how to use Gerrit's internal JSON-RPC API. Yes, it is not a public API (so we should not expect it to be stable), and having to watch your browser requests to learn how to use it is not the kind of documentation I'd like, but that was all we had (ok, I could check Gerrit's source, but since that's Java I'd rather watch the requests sent by the browser) so it had to do.

After experimenting a bit I was able to get a list of changes proposed by a given person as well as all the details of a given change. Here are the methods one can use for that:

Retrieving a list of changes


  endpoint: /gerrit/rpc/ChangeListService
    method: allQueryNext(query, pos, page_size)

  Return up to page_size items that match the given query whose changeSortKey is lower than pos.

  # This will get the list of changes authored by jserv@0xlab.org
  curl -i -X POST -H "Accept: application/json,application/jsonrequest" \
       -H "Content-Type: application/json; charset=UTF-8" \
       --data '{"jsonrpc":"2.0","method":"allQueryNext","params":["owner: jserv@0xlab.org","z",10],"id":1}'\
       https://review.source.android.com/gerrit/rpc/ChangeListService


Getting the details of a change


  endpoint: gerrit/rpc/ChangeDetailService
    method: changeDetail(id)

  Return the details of the change with the given ID.

  # This will get all details of change 16615
  curl -i -X POST -H "Accept: application/json,application/jsonrequest" \
       -H "Content-Type: application/json; charset=UTF-8" \
       --data '{"jsonrpc":"2.0","method":"changeDetail","params":[{"id":16615}],"id":1}'\
       https://review.source.android.com/gerrit/rpc/ChangeDetailService


Pretty simple, right? Just note that you need to specify your charset in the Content-Type header or else you'll get a server error from Gerrit, and the
JSON-RPC requires the 'id' param to correlate a response with its request, but you don't need to worry about that if you're doing things synchronously.

That was all I needed to know in order to write the code that keeps the changes authored by Linaro engineers in sync between our system and AOSP's Gerrit instance, and it should be enough for you to get started if you ever need to get data out of a Gerrit instance (assuming the API hasn't changed in the meantime ;).

How to shoot yourself in the foot using contextlib.contextmanager
salgado
I quite like the contextlib.contextmanager decorator as it makes it very easy to create a context manager to use with the 'with' statement. For some reason, I seem to mostly use that in tests when I want to replace parts of the system with test doubles. It looks something like:

@contextmanager
def do_run_mocked(mock):
    orig_run = mod.do_run
    mod.do_run = mock
    yield
    mod.do_run = orig_run


And then I use it in a test like this:

class TestRun(TestCase):
    def test_run(self):
        mock = MockDoRun()
        with do_run_mocked(mock):
            run(...)  # This will end up calling mod.do_run
        # Here I can assert that run() works by inspecting what is stored in
        # the mock.


However, there's one thing you need to remember when using contextlib.contextmanager (and specially when using it for this purpose): if an unhandled exception occurs in the block, it is reraised inside the generator at the point where the yield occurred.

Even though it is mentioned in the docs, neither myself nor the people who reviewed my code seemed to realize its consequences -- I hope they'd call my attention to that if they did realize.

And the most important consequence (in this use case) is that when something inside your 'with' block raises an exception, the mocking won't be undone and thus the remaining tests may be affected (and fail!). Of course, once you fix the 'with' block to not raise, the mocking is undone and the remaining tests won't be affected but you'd still be left with no clue as to why they failed earlier and why they started passing again, all of a sudden. I surely wouldn't trust such test suite.

By now you might be thinking that this can be easily avoided by wrapping the yield statement with a try/finally (and do the unmocking there). Indeed it can, but I'm sure I'll keep forgetting about it so from now on I'll try to use only test fixtures (as Robert proposes) for this purpose.

A Django view used in just one test
salgado
So today I had to change a test in Launch Control to make sure we were protected against CSRF attacks. The existing test was using the login form, which is being removed, so I had to use another form in the test. Given that Launch Control doesn't have any forms now, I had to write a new one, but since it was only going to be used in that test I wanted all the code (specially the urlpatterns) for it in the same file as the test.

After reading/struggling a bit, this is what I came up with. It doesn't look too bad to me, but I'm wondering if there are any simpler/cleaner ways of doing this?

# CSRFTestCase is just a custom test case which provides a TestClient with an
# UnprotectedClientHandler.
class CSRFConfigurationTestCase(CSRFTestCase):

    @property
    def urls(self):
        urlpatterns = patterns('', url(r'^test-form/', test_form))
        return type('urls', (), dict(urlpatterns=urlpatterns))

    def test_csrf_token_present_in_form(self):
        if django.VERSION[:2] == (1, 1):
            # This feature is not supported on django 1.1
            return
        form_path = reverse("dashboard_app.tests.other.csrf.test_form")
        response = self.client.get(self.form_path)
        self.assertContains(response, "csrfmiddlewaretoken")

    def test_cross_site_form_submission_fails(self):
        if django.VERSION[:2] == (1, 1):
            # This feature is not supported on django 1.1
            return
        form_path = reverse("dashboard_app.tests.other.csrf.test_form")
        response = self.client.post(self.form_path, {'text': 'text'})
        self.assertEquals(response.status_code, 403)

def test_form(request):
    t = Template(template)
    html = t.render(Context({'form': SingleTextFieldForm()}))
    return HttpResponse(html)

class SingleTextFieldForm(forms.Form):
    text = forms.CharField()

template = """
    <html>
     <body>
      <form action="." method="POST">
       <table>{{ form.as_table }}</table>
      </form>
     </body>
    </html>
    """

Vim plugin to automatically fold python imports and __all__
salgado
After a couple nights using eclipse to hack on tomdroid I started to wish that vim could automatically fold the global python imports like eclipse does.  As usual I just googled around hoping to find something that I could dump in my .vim/ to do that, but in this case I didn't find anything, so I had to do it myself: myfolding.vim

Some things to note:
  • It will only fold your global python imports and __all__
  • It will not fold anything over 200 lines. This is to make sure it doesn't add a significant delay when loading files, as most plugins that use fdm=expr do
  • It expects you to have at least one blank line between your global imports and your code, but why on earth would you not have that, right? ;)
  • Feedback and patches always welcome

bzr unshelve --preview on a nightly ppa near you
salgado
Since the shelf plugin was moved into bzr core I've been missing a way to see the actual changes that I have shelved without having to unshelve them. Having waited long enough for someone else to implement it, I decided it was about time to do it myself.

With some help from jam, it was merged into bzr-2.1.0b1 and is now available in the bzr-nightly-ppa.
Tags:

postgres database size
salgado
Today I spent a reasonable amount of time trying to get the size of each database we have in our postgres server, to see who's responsible for eating all our disk space. Although I found a link to a script to do this for all tables, the site was down and I had to use web.archive.org to retrieve it. So, for those of you who also need that, here is it (with some tweaks; the original version failed because one of our databases was too big to fit in an int4):

-- Don't forget to run psql -d  < dbsizes.sql

CREATE TABLE tablesizes (
   "name" name NOT NULL,
   "size" int8 NOT NULL
);

CREATE FUNCTION alltablesizes() RETURNS setof tablesizes AS '
 DECLARE
  actual tablesizes%ROWTYPE;
  row RECORD;
  size int8;
 BEGIN
 FOR row IN SELECT datname FROM pg_database LOOP
 SELECT INTO size database_size(row.datname);
 size:=size/1024;
 SELECT INTO actual a.datname,size FROM pg_database a WHERE a.datname=row.datname;
 RETURN NEXT actual;
 END LOOP;
 RETURN actual;
 END;
 'LANGUAGE 'plpgsql';

SELECT * FROM alltablesizes();

(no subject)
salgado
Finished the configuration of Ubuntu to be run on the diskless workstations. Now we can boot either Debian Woody or Ubuntu Warty (with all that bleeding edge stuff that comes with it). Ubuntu rox!

Back to work
salgado
I'm back to async. This time working on zope3, which seems to be a great platform, but quite complex.
Now that I'll have something useful to post here, my main problem will be time to post, since I'm finishing my studies and working.