Microsoft Test Manager

Today I was conducting a test on Effektor using Microsoft Test Manager. It's a simple tool that makes it easier to record findings in explorative tests. I also got to do a bit of T-SQL and learned that you can't SELECT on an EXEC command and have to jump through all kinds of hoops if you are unlucky enough. If you are lucky you can INSERT INTO some tmp table and then select from there, but I had an error because of an excisting rollback in the procedure.

Yesterday was the first day in the coursera course "Introduction to mathematical Philosophy" which was really interesting. I learned a bit about Zeno's paradox and Cantor's theorem, although I think I will have to watch that last one again. The course also reminded me of a few of the interesting things from Discrete Math, really recommendable. Course is here: https://class.coursera.org/mathphil-001/class/index

Last thing was something I relearned but will have to try and remember. When working in windows always use "alt+space" a lot, it saves the long trip to the upper right corner.

MS Test

Today Michael Nygaard showed me how to do a unit test for Effektor. Finally the amount of interfaces begin to make some sense :)

70-461 Exam

Today I passed the 70-461 Microsoft exam about SQL Server 2012. I realised that I know nothing about varbinary and filestream, so I spend some time reading up on it here:

http://msdn.microsoft.com/en-us/library/gg471497.aspx

It's apperanetly the way to go for storing files in sql server.

A proper running average in TSQL

Sometimes I am amazed at the lenghts I have gone to in TSQL to construct things that fit in a few lines, once you know how. Today I was reading about window functions. They are excellent for the kind of grouping where you want to keep each row, but also include some totals or other things that relate to the group that that row belongs to. Things like a sale and the sale total for that category of items.

Previously I would have made a subquery that found the category totals and the joined the two tables, but using a window function you can do it like:

SELECT category, sale, sum(sale) OVER (PARTITION by category) as categorySale

It also makes running averages much much easier than some of the hack I remember doing before:

SELECT custid, freight, AVG(freight) OVER (PARTITION by custid ORDER BY OrderDate, orderid
ROWS BETWEEN 2 PRECEDING AND CURRENT ROW) as movingaverage
FROM Sales.Orders

Triggers

Preparing for the 70-461 exam I read about triggers. They are store procedures that are automatically invoked upon some actions on a table. A kind of event handler for SQL-tables. They can be used to enforce rules that are more complex than what would fit in a CHECK constraint. They can also be used for any other kind of action that applies every time the data is edited. The max. depth for circular references is 32 calls and returning results from a trigger is deprecated. 

Analysis center, .NET and LSA project

Today I was testing a new analysis center for Effektor and we ran into some issues with different DLL version numbers. I learned about the GAC which is a piece of .NET CLI that maintains various DLL's using strongly typed names. You can access the GAC through the normal filesystem in %windir%\Assembly but you will only see a shadow, since normal FAT32 does not support files with the same name. The GAC is able to used the strongly typed names to find the correct version of a DLL for a program, thus avoiding DLL HELL.

As promised I added the LSA project to my open bitbucket repos.

Streaming to a MediaElement in C# Windows 8 Apps

Today I learned that things can be too easy sometimes :) I was trying to set the source of a media element to an .mp3 stream from DR. I tried all sorts of tricks to convert a URI to a StorageFile to a stream. After some hours of this I stumbled upon the solution:

mymediaelement.Source = new Uri("Address");

KISS

Stemmer

Today I did a danish Stemmer in C#. I will put it on my bitbucket asap. It is an implementation of the algorithm foud here: http://snowball.tartarus.org/algorithms/danish/stemmer.html. I plan to continue the project into a full LSA machine in time. I also saw a strange bug in telerik grids where calling get_selecteditems() returned a null the first time being called after loading a page, but first getting the mastertable, and then calling get_selecteditems() on that, got the correct items. A very annoying bug.

More buttons and strings in Effektor

Today I fixed some wrong strings and corrected a small mistake in some button-activation javascript in effektor. It really didn't teach me anything, perhaps that it's better to proof-read and slow down than having to go back and correct text later. I also printed some text on Bloom Filters which is a datastructure I am looking into for my thesis. I had my first meeting with Rasmus Pagh yesterday and he suggested this topic, so I will try to give it a shot, it sounded rather interesting.

Fermats last sentence

Today I had time for some leisure reading. I recently got Fermats Last Theorem by Simon Singh. It's a pleasure to read, I do feel that it's really lacking in sources especially in the stories about Pythogoras which is told in (uncredible?) detail. But some of them are interesting enough. I learned the Pythogoras sentenced Hippasus to drowing because he discoverede the irrational number sqrt(2) which was considered some kind of heresy. I think the stories retold by Simon Singh are taken from Pythagoras: His Life, Teaching and Influence by Christoph Riedweg and found some online debate that indicate that they are not really well supported. I learned the concepts of the theorem, but it's annoying that the book is not more critical of its sources.

Matematical analysis and linear algebra

Today I passed my exam in Mathimatical Analysis and Linear Algebra (again). The exam was on linear systems and I was told that I should have spend less time on basic topics and move to more advanced topics faster. The subject was very basic though and the examiner (Dan Witzner) and censor did not really push me for more advanced topics so I was a bit diasppointed in that feedback. I also got scolded for not answering all questions but the only question I didn't specifally adress was matrix multiplication which I had already coverede in depth.

I can't complain about the grade so I wont, but to try and learn something from it I will say that I learned that when going for a math exam it's enough to just mention the execistens of some property and only proof it when asked specifically for a proof. I think this also confused me since my math courses at Aarhus University  where focused on proofs.

CORES2013 and exam prep.

It has been very quiet here for a while since I have been preparing for my exams and attending CORES2013. Today I hope I can return to my daily 10 minutes. Today I learned how to properly construct the single value decomposition of a matrix. A=UEV:

U = { U1 U2}

U1 is an ortonormal basis for R(A)

U2 is an ortonormal basis for N(A^t)

E = {(E_1,0)^t,(0,0)^t}

Where E_1 contains single values aka. the squareroots of the non-zero egienvectors along the diagonal.

V={V_1^t, V_2^t}^t

V_1 is an ortonormal basis for C(A)

V_2 is an orthnormal basis for N(A)

Changing the encoding in windows cmd.exe

Today I saw something nice somewhere on the net. To change the codepage of windows cmd you can add a string value registry key to the path:
"HKEY_CURRENT_USER\Software\Microsoft\Command Processor"

Name it "AutoRun" and give it a value of "chcp <codepage>" like ex. chcp 850.

Eignvalues noch einmal

Today was the last day of mathematical analysis and Linear Algebra with Dan Witzner. Me and Florian did a lot of exercises with eignvalues and vectors. I guess I would take away that the procedure for finding eignenvalues for a 3x3 matrix involves solving a third-degree characteristic polynomial and I did not remember how to do that. My "memory path" for finding eigenvalues and their egenvectors:

1. Ax=λx -> (A-λI)x=0  
2. This means that det(A-λI) must be 0, which leads us to the characteristic polynomium.
3. Solve the polynomium to find λ
4. for each λ find the linear combination that fulfills (A-λI)x=0
5. All linear combinations of the vector(s) found in 4 are eigenvectors belonging to λ.

In general I need to study these topics again before my exam:
SVD, Gram-shmidt, the advanced parts of the calculus corriculum.

Last day of mining

Today I was working with Florian and Victor on the last coding day of our datamining project. I learned a few tricks for latex and git (As always when i work with Florian). It's nice to do "git rebase master origin" instead of just git pull. The rebase will lift all my commits, apply commits I am missing and then apply my new commits on top. I also learned of gitk, a nice tool to get a an overview of the branches.

Finally I discussed the meaning of showing a confusion matrix for 10 runs of KNN. I argued that it does not make sense as we are talking about the quality of a specific model, not the general ability of KNN. In real life I guess we would run KNN and test it, and the for the final implication we would train on all data and not leave any for testing. I should ask Julia Trogelius about that.

More MKC

Today I spend all day coding on the MKC solution. I went through a lot of painful debugging of my shifts-creator. In the end it turned out I had a special case in a nested loop that would sometimes not reset a counter. I learned that it's a good idea to check that counter resets are handled in one place instead of in each individual nested loops.

Certutil and powershell

Today I learned how to add and remove certificates with "certutil" in powershell. The help is pretty usefull but there where a few things I had to spend time guessing:

There are stores. It's kind of hard to find out what store you are in, but if you use "certuil -viewstore" in powershell the console will show the name of the store it's showing. In my case "CA", you can view another store by using certutil -viewstore "storename".

I also learned that powershell will write "delstore command completed sucessfully" even if provided with names that did not lead to a certificate. So it's important to check if "certutil -delstore" acutally deleted anything in the lines above. You can refer to the certificate by writing its full nickname or title as it appears in the store view that appears using "certutil -viewstore"

certutil -viewstore "storename"
certutil -addstore "storename" "cert. path"
certutil -delstore "storename" "cert. nickname"*

*find the nicenames and store names by calling -viewstore.

Index of smallest entry in array

In MKC planner I had to find the index of the array index with the smalles values. I learned how to use Enumerator Aggregator to do this. Even with some random breaking for ties. From memory it went something like:

Random r = new Random();

Enumerator.Range(min,max).OrderBy(i=>r.Next()).Aggregator((a,b)=>array[a]<array[b]? a : b)

I also learned how to install RockBox on an Ipod "Classic" (6th gen) which turns out to be very simple using Emcore.

Database In Depth

Today I was reading some of C.J. Date's book "Database in depth", I learned that mr. Date doesn't not have much love for SQL*. But I also learned a lot about the relational model. To be specific it's worth to remember that the relational model is based on a relation as a heading containing a set of attributes, and a body containing a set of touples conforming to the heading. Because the relations use sets there are no duplicates and no ordering in a relation, unlike SQL-Tables.

I also really liked the part about base and derived relations where Date stresses the fact that the relational model specifies nothing about physical storeage and the thinking that "views are virtual, tables are physical" is wrong. The only true requirement is that there is some mapping between the physical storeage and the base relations, which in turn can be combined to get the derived relations. Everything else is up to the implementation.

"Base relations are no more (and no less!) "physical" than views are, so far as the relational model is concerned" - C.J. Date

*In general I really enjoy the almost palpable but subtle dispise for SQL eminating from this book so far. It feels like SQL is a personal insult to mr. Date and the thought that some people will call SQL a "relational" language will bring him to the barricades at any time, to stand up for what's right. There is a vibe of Luke 23:34 to the whole thing at the same time. Great writer!

Minimum Spanning Tree

Today I was correcting handins about building MSTs and I re-read some chapters from Sedgewick. A new thing i "learned" or at least that came to my attention is that when we are talking about big-O runtime it really doesn't matter if we are talking O(E log E) or O(E log V) since E <=V^2 -> log(E)<= log(V^2)=2log(V) --> O(log(E)) = O(log(V)).

So jumping lots of hoops to get from E log E to E log V really doesn't do much but allow you to practice building Union-Find datastructures and priority queues.

Self validation

Today I learn about the Validation Application Block. It's nicely described here. But unfortunatly I just could not get it working for the longest time. It turns out Mikael Nygaard had a solution for me. Adding the [HasSelfValidation] tag in the metadata Class instead of the actual Class solved the problem.

Eigenvalues and SVD

Today I saw Dan Witzner perform more matrix magic. Nothing really new, but this stuff keeps falling right out of my head so I will try to remember this:

If Ax=yx then x is an eignvector af A and y is the corresponding eignvalue. 

If A is invertible then it's non-singular and det(A)=! 0 and vice versa.

If A is not invertible then it's singular and det(A) = 0 and vice versa.

Geocoding

Today I prepared a short presentation on GeoFlow. I was not very impressed by it, but it is still in preview so I hope it will improve. Most significant misses is that you cannot use it to enrich a data set. It uses Bing  Maps to geocode you dataset, but unfortunatly there seems to be no way to save this data and add it to the dataset for use with ex. Maps in SSRS. This still has to be done with SSIS packages or with heavy scripting or other hack-is ways. GeoFlow provides some nice basic plotting, but for doing anything away from a very limited template it's useless.

I learned of Requests, a nice way to work with http-requests in Python. For instance to grab a GPS coordinate from a request to Google maps. Something that you can do 2.500 times a day with no cost.

Linear algebra

Today I was at ITU and saw a lot of nice matrix magic by Dan Witzner. To take one thing I learned that N(A)⊥C(A^t). I have seen that statement before, but not really understood it. Today it became clear to me when presented like: Ax=0 so every vector x belonging to the null-space is perpendicular to any row in A since their dot product is 0.

MySQL and GIT

Today I was working on an ITU Datamining project where I am forced to use MySQL instead of MSSQL. I learned that the ISNULL() function on MySQL has a boolean return and will not work like the T-SQL ISNULL(). I also saw the use of "make" even though I am yet to understand the point of makefiles.

Florian showed me lots of tricks:

ctrl-z plus the command "bg" to put something in the background thread in the shell or just start emacs with the "&" flag.
alt-q in emacs to get some pretty formatting.
ispell in emac to spellcheck.
using ssh jovt@ssh.itu.dk to get to my itu network drive from my own computer.

Treating two departments as one for some purposes, but as two for other.

Today I was working on MKC Planner where I needed to treat two departments as one in some cases and as two in other. I the case where they where one I then need to split the work assigned to the "union" between the two actual departments underneath. I ended up solving it by using three methods:

WeakUnion:

Given department a,b,c,d,e we want (a + b),c,d and e to have equal work for some tasks and a,b,c,d and e to have equal work for others:

1. Ignore(dep a)
Distribute(some)
2. Reconsider(dep a)
3. Split(from b, to a)
Distribute(others)

Also: when you want GIT to ignore a folder you can use "git rm -r <foldername>". If the folder is already in older versions of the repository you can use "git rm -r --cached <foldername>".

Generating design file from .resx

Today I was working on the resource files for Effektor, I was trying to make that work easier using Visual Localize when suddenly my solution stopped building. I learned that the generation of the design file is controlled by settings stored in the project file that had been altered by visual Localilze. To change it back I had to change the settings in the properties of the .resx file. In this case I had to use "GlobalResourceProxyGenerator" rather than "ResXFileCodeGenerator".