I’m not sure this is the best approach. The volume of data the NSA may or may not have access to every day, if they are able to access email, phone records etc, would be astronomical. We (the technological and scientific community) often find it hard to extract meaning from  massive well structured datasets. How is the NSA expected to systematically derive relevant information from as fragmented, diverse and unstructured data as personal communications? If I worked for the NSA, I’d select a number of robust digital targets or flags to look for inside this information. One of those targets would almost certainly be encrypted emails. You can use fairly simple machine learning techniques (Bayesian filters come to mind) to pick key-encrypted text out from normal human readable text.

If a nefarious individual were planning something criminal and using email at all, I would expect them to encrypt or disguise the content in some way. Such an individual would have to assume all communications were compromised  - just because we, the public, have not thought this until now does not mean that any organization participating in illegal activities does not already make this assumption. It’s probably also fair to say that, until last week, very few people encrypted the contents of their emails.

Considering this, if I worked for the NSA, I would probably think, “Well – this is one, easy way to drastically reduces the set of data we have to examine” and to essentially treat encrypted emails as a flag. If programs like PRISM are as far reaching as they’re claimed to be, then email is one dimension of a network of data which can be built around a potential target. Encrypting your email may essentially attract the very attention its trying to avoid.

One  solution to this, is to use old-school codes and cyphers. Like Enid Blyton’s youthful heroes and heroines, or every spy kit given to every budding childhood spy, secret messages encoded within apparently benign or routine notes may be a far less conspicuous way to sneak secrets through the tubes. Of course, encrypted or coded messages are largely uninformative without other pointers. I suspect this is the precise reason why PRISM is reported to be so massively encroaching – simply put, using just email  or cellular metadata alone does not provide even close to the granularity or certainty needed to make informed analyses of potential terror threats. That being said, how the NSA would deal with such a vast, massively complex, semi-overlapping, semi-complementary data set is, theoretically at least, an incredibly interesting problem.

Say I wanted to do something with the p53 protein sequence. My general approach would be to;

• Go to UniProt (http://www.uniprot.org)
• Search for p53
• Open the relevant record (p53 in Humans)
• Open the sequence in .fasta (http://www.uniprot.org/uniprot/P04637.fasta)

Now I have a problem – I don’t want all these line breaks in my sequence – I just want it to be a single string. What I used to do was manually delete the line breaks, but this is a drag. It’s not hard, takes maybe 20-30 seconds, but feels very lo-fi.

To get around this I wrote klean (one of my biotools scripts). A command line utility which takes a string as an argument, and copies the string without any spaces, tabs, new lines etc to your middle button clipboard for easy manipulation (the CTRL-V clipboard is left untouched).

For example
klean "MEEPQSDPSVEPPLSQETFSDLWKLLPENNVLSPLPSQAMDDLMLSPDDIEQWFTEDPGP
DEAPRMPEAAPPVAPAPAAPTPAAPAPAPSWPLSSSVPSQKTYQGSYGFRLGFLHSGTAK
SVTCTYSPALNKMFCQLAKTCPVQLWVDSTPPPGTRVRAMAIYKQSQHMTEVVRRCPHHE
RCSDSDGLAPPQHLIRVEGNLRVEYLDDRNTFRHSVVVPYEPPEVGSDCTTIHYNYMCNS
SCMGGMNRRPILTIITLEDSSGNLLGRNSFEVRVCACPGRDRRTEEENLRKKGEPHHELP
PGSTKRALPNNTSSSPQPKKKPLDGEYFTLQIRGRERFEMFRELNEALELKDAQAGKEPG
GSRAHSSHLKSKKGQSTSRHKKLMFKTEGPDSD"

Will copy this whole badboy to my clipboard as a single string, so when I press the middle mouse button it pastes it wherever I want.

As with any executable script, to make it system wide either put it in your [/usr/bin] directory, or any other directory in your PATH (see [echo $PATH] in bash/zsh for these locations). Simple, totally unnecessary  but has significantly improved the last few days!

From a purely selfish point of view, I can say with absolute certainty I’ll never have the privileged of interacting with a  guy who, by all accounts, operated on an intellectual and functional level far, far above normal people. Somewhat more importantly though, is the  reminder that depression is not something people can control. It’s not something you can rationalize, or defeat using a logical approach. If it could have been “solved”, as one might solve a technical problem, I have no doubt that Mr. Swartz would have solved it.

• Understandable (and readable)
• Maintainable
• Testable
• Extendable

Which are four things I think software written by non-computer scientists often lacks, to various degrees. Clearly this is not always the case, and I am not for a second saying that non-computer scientists are unable to write good code, but if you’re thrust into an environment where your only goal is to solve a problem, these best practices often slip into the background compared to issues like, “what is an environmental variable?“, “why doesn’t this compile?” or “how the hell do I get out of VIM?“[1].

More generally, if no one teaches you best practice, how are you meant to learn? Often people will develop a sense over time, but for that first year or so you may not even know you’re missing crucial ideas or approaches. “Version control – sure, I save incremental file names like ‘mycode_1.pl’, ‘mycode_2.pl‘”.

A recent article aims to address these issues by summarizing scientific computing best practices. It’s short, well written and to the point, but encompasses a huge number of ideas crucial for software development. HIGHLY recommended reading for anyone involved in writing even the most mundane scripts. Read it for your collaborators. Read it for future students and postdocs. But most of all, read it for future you, because there is no greater pain than not understanding something you know you once understood, because you wrote it.

[1] – Full disclosure, I still get trapped in VIM from time to time.

I’m currently using it to build what will end up being a 100 000 row table, using about 2000 batch queries, and so far it’s going well [ UPDATE - Table built successfully over a weekend]. I’m really pleased with how it handles bad accession values in batch mode - responsiveness is fast and we find error causing entries very quickly. Also, being able to install globally with a single command is awesome. Thank you pip!

I want to use the ORM create a new table in the database which has a foreign key in one of the existing tables – not an unreasonable request. The database is already setup, and a desc; when logged in through the MySQL monitor of the hopeful parent table gives the following output;

So, looking at this output we can define the following class for the ORM;

Using this class I can connect to the database and query through using the ORM without a hassle. For example, below I connect to the database, and then I query it for the first item in the table egtable;

However, if we try and create a table using this same syntax/style, we hit a problem. Look at this class – we should be able to create a table from this, right?

We only have three columns, and we define the foreign key in exactly the same way as above (i.e. an Integer of width 10). Given we can read the database using this ORM, surely that means it’s a valid way of describing the data? Well no – this fails. It fails with a 150 error, which is a foreign key error.

Why? Because with MySQL the signs of foreign keys have to match, but the SQL specification doesn’t have an unsigned integer. To get around this, you can import in the MySQL Integer dialect via

from sqlalchemy.dialects.mysql import INTEGER as Integer

I found a port-schematic on the Lenovo website, and glancing through found the “Kensington Lock” item on the key. What I failed to notice, however, was that despite being on the key for the laptop the corresponding number “8″ doesn’t actually appear on the laptop image. Instead, there are two “7″s.

Well played Lenovo, you win this round…

Picture from the meetup – 10 points if you can spot me!

In a nutshell, Julia is a dynamic language which uses an LLVM-based aggressive JIT compiler for incredibly fast and flexible scientific computing. It allows high and low level programming in a single language, with the ability to easily call C, Python and Fortran libraries. While it doesn’t yet have a threading model, it allows parallel computing over a distributed system, meaning you can transparently send and receive information from distributed computers, making it ideal for big data, the buzzword of the moment. The initial, pre-release version was pushed out in February this year, and since then there has been a huge amount of work updating and building out the necessary support. Its development has been based on the following aims;

“We want a language that’s open source, with a liberal license. We want the speed of C with the dynamism of Ruby. We want a language that’s homoiconic, with true macros like Lisp, but with obvious, familiar mathematical notation like Matlab. We want something as usable for general programming as Python, as easy for statistics as R, as natural for string processing as Perl, as powerful for linear algebra as Matlab, as good at gluing programs together as the shell. Something that is dirt simple to learn, yet keeps the most serious hackers happy.”

Those are pretty lofty goals, but after seeing what Julia can do, I’d say Stefan and co are well on their way to meeting them. There were a few things during the talks which I felt really stood out.

• Optional typing. This means you can define a variable as a specific type (in which case you impose the various [runtime] typing restrictions you expect) but you don’t have to. This option is awesome. It means you can write typesafe code where appropriate, and hack together something very quickly (as you might in Python or Perl) in a matter of minutes. The best thing, though, is that using or not using typing has no impact on performance, meaning its purely a design choice, not an optimization one. It also allows for a totally natural way to make calls to C functions, which is typically less syntactically clear in a totally untyped language. What typing does allow, however, is multiple dispatch, which allows for massive generalization of functions and code. With type checking occurring at runtime only, there’s no compile time type checking, which allows for a dependent type system which would otherwise be impossible (because a runtime type is only actually generated at runtime as it can use real values, not just types). This adds a lot of flexibility into how you develop your systems.

• Speed. A lot of Julia’s speed comes from a type inference algorithm which allows the specialization of code. This means that rather than generating generic methods which then type check, type-specific code is compiled, making the language incredibly fast. I guess it’s kind of obvious, but both the benchmarks Stefan showed and the feedback from the developers who presented was pretty impressive. Wes McKinney blogged about benchmarking some pretty trivial array operations more slowly than you’d expect given the benchmarks shown. Stefan’s response to this was good, and I’d hope that any of the hiccups or inconsistencies seen this early on are simply due to a mixture of immature libraries and people not doing things in the most efficient way. Fingers crossed compiler optimization can try and correct for the latter, and given the language’s age this is pretty understandable.

An honorable mention should probably go the JS V8 engine, which is pretty speedy, although not very well suited for scientific computing.

• Syntax and flexibility. Syntax is perhaps not the most crucial thing for seasoned programmers, but for new programmers having a syntax which basically looks like matlab + python is only a good thing – code looks like pseudocode at times (example below)

Another thing I liked was the codes flexibility, where even the most unassuming expressions can trigger massive operations. As Stefan put it,

“ (a+b) can do a single machine instruction or start up a cluster”

Despite all the awesome language features, one of the things I was most impressed with was the actual presentation. Trying to push a new language and make it appealing is not an easy task. Stefan’s presentation[1] was awesome – in my opinion it was pitched at precisely the right level, the mix of overview and code was great, and he dealt with questions exceptionally well. The presentations from Shane Conway[2] and John[3], only backed up what Stefan had said, and indeed further showcased the languages power and simplicity, but in a totally non-fanboyish way.

Julia’s future looks pretty exciting too. Stefan said that there will hopefully be a package manager and a module system out in the next month or so. I think when this happens, we’ll start to see some of the general scientific packages ported over (something I’d be interested in doing). Additionally, we can expected improved performance and stability ahead of the 1.0 release.

For me, this is very much one to watch, and perhaps get involved in…

For those who missed it, Stefan and Jeff Bezanson (another of the core team) gave a talk and an interview at the lang.NEXT conference this April.

[1] http://files.meetup.com/1406240/julia.pdf
[2] http://files.meetup.com/1406240/julia_lm.pdf
[3] https://github.com/johnmyleswhite/julia_nyhackr

Despite this, Google search autocomplete seems to think I’m a Skyrim fiend.

I checked out my add preferences to see if that was the issue – nothing about video games there at all, which makes sense as I’m not really a gamer any more. To explore a little further I signed out of Google, and opened IE9. No difference.

Next I VPNed to the UK (I live in NYC) and tried searching (again through IE9 while logged out of the big G). This was interesting – I didn’t get exactly the same results, and in fact my “What is the best…” prompt failed to generate any Skyrim related queries. Despite this, other question openers generated those same, Skyrim-centeric questions

Am I the only person experiencing these results? Does Google know what I want before I do? Is the algorithm broken?

For the last 14 million years, the world has been evolving with (largely) the same selective pressures, and, most importantly, with the potential for biological exchange between different habitats. Obviously, this exchange is often dependent on geography – hence why Australia has some strange animals not seen elsewhere, but even in Australia you have things like wind, rain, sun, seasons etc. The “general”  macro-environmental selective pressures are the same. In the oceans you don’t have this geographical separation – a fish can swim from the Pacific to the Atlantic, and can swim from the sea bed to the surface. Whether or not the modern day versions of the species we see actually make this kind of migration is another (somewhat irrelevant) question, but the fact it’s possible has a lot of implications for how those water-dwelling creatures have evolved. Lakes are a little different, but typically big lakes are connected to other lakes and the oceans by rivers. They are still part of this “surface” habitat, which has sun, changing water levels, the impact of land animals and birds, seasons, wind, etc.

Lake Vostoc has neither the same selective pressures, nor biological exchange. There has probably (I’m always uneasy with absolutes) been no biological exchange for 14 million years with the outside world. On top of that, it is an environment so foreign compared with anywhere else on the planet. There’s no light, so the food chain must be based on chemosynthetic organisms (as opposed to photosynthetic ones – plants and algae). The water is cold, estimated at -3 degrees Centigrade, but kept liquid by the lakes high pressure. There are no seasons, no lake “surface” as the lake’s interface is either rock or ice, although there are small tides. Finally, the level of oxygen and nitrogen  in the lake is 50 times higher than in normal freshwater lakes.

Needless to say, it’s quite a unique environment.

You might well assume there’ll be nothing down there, and any life found would be highly inactive to try and conserve the minimal energy it has access to. Those as widely optimistic as myself would disagree. When scientists explored similar, but much smaller caves that had been sealed off from the rest of the world in Romania, they found over thirty new species, where the largest were related to the size of their environment (i.e. they could still move around freely). These species were highly active, and to some extent resembled modern insects but had many features that were totally novel. Granted, these caves were open air (as opposed to water), but water provides a more stable environment for growth and evolution. Even if there is no “large” life (I’m hoping for mermaids, but that’s probably too much to ask for) the micro-organisms that exist could be truly extraordinary.

In a lake the size of Lake Ontario, but twice as deep in places, the possibility of new species, and even new ecosystems within the lake is truly incredible. More excitingly, for the biochemist in me, is the potential for totally new routes of evolution – new systems for energy generation, new skeletal formation to deal with the pressure, new circulatory systems to take advantage of the high oxygen concentration. Who knows – while carbon is readily available on the surface through photosynthetic fixation, in water this is very much not the case – perhaps an ecosystem where sulfur, nitrogen or even oxygen are it’s principle “biological” component exists?

The main risk is that in the process of entering the lake to “have a look around” (at 4km below the surface, this is a bit of an engineering feat) we contaminate the lake. The various parties involved (Russian, American and British teams are all drilling) have been careful to try and avoid this, but in what may be an ecosystem radically different from our own, it seems to me that assessing what would be a contaminant is an impossible task – surely anything represents a contaminant.

Whatever the case, the next stage is to recover water, which is expected towards the end of 2012. Still more waiting, but hopefully, something truly fascinating will be at the end of the wait.