On building websites: Comparing Scala/Play vs GO

I just pushed code to <https://github.com/talmai/gemawebsite> where I have the very same website coded to run on Scala 2.9.2  + Play 2.0.2, and on Go 1.3. Both code work, and generate the same output (checkout www.gemaestudio.com).

As of today, when given an option, I will choose Go over Scala anyday. The code speaks for itself. But out of curiosity, I took a look at memory and cpu requirements... (screenshots below are from the 'top' command).

VIRT stands for the virtual size of a process. In a nutshell we are talking about the sum of memory it is actually using (memory it has mapped into itself, shared libraries, and memory shared with other processes). VIRT represents how much memory the program is able to access at the present moment.

RES stands for the resident size, which is an accurate representation of how much actual physical memory a process is consuming. (This also corresponds directly to the %MEM column.) 

Scala uses 74MB of RAM (RES), but 297MB for VIRT. 

Go, as expected, uses only 5.4MB of RAM (UPDATE: to be honest, it's more like 3.8MB), but has mapped 828MB (VIRT). This I didn't expect.

Anyhow... it was fun while it lasted. The awesome part is knowing how much more RAM I got freed up on my box after this exercise. ;)

How to combine multiple worksheets into one in Excel? (using vba)

I have several excel workbooks (.xls files) and each workbook contains multiple worksheets. The number of worksheets (as well as their names) are not constant. Each worksheet is formatted in the same way (i.e: same header). Now I want to copy all the data on each worksheet and copy it into a single worksheet. 

Launch an Excel file that you want to combine other workbooks into. Click Developer > Visual Basic, a new Microsoft Visual Basic for applications window will be displayed, click Insert > Module, and input the following code into the Module. 

Oh, and before that, create a sheet called "Summary" where we will collect the info.

Sub SummarizeSheets()
    Dim ws As Worksheet
    Application.ScreenUpdating = False
    Sheets("Summary").Activate
    Path = "C:\path\to\your\files\"
    Filename = Dir(Path & "*.xls")
    Do While Filename <> ""
        Workbooks.Open Filename:=Path & Filename, ReadOnly:=True
            For Each Sheet In ActiveWorkbook.Sheets
                For Each ws In Worksheets
                    If ws.Name <> "Summary" Then
                        ws.Range("A01:AX201").Copy
                        ActiveSheet.Paste Range("A65536").End(xlUp).Offset(1, 0)
                    End If
                Next ws
            Next Sheet
        Workbooks(Filename).Close
        Filename = Dir()
    Loop
End Sub

Then click  button to run the code, and all of the worksheets (including the blank worksheets) within the workbooks have been merged into the master workbook.

References: http://www.extendoffice.com/documents/excel/456-combine-multiple-workbooks.html

bit-for-bit copy of DVD on osX (or: how to generate .iso on mac)

0) Insert DVD
1) Open up terminal
2) Find out where it was mounted (diskutil list)
3) unmount it (sudo umount /dev/disk1)
4) generate .iso (dd if=/dev/disk1 of=~/dvdImage.iso bs=2048 conv=sync,notrunc)

How to install django-celery in windows (python, django, celery, erlang, rabbitmq)

A project of mine goes through a realistic simulation that doesn't take too long (15 seconds), but that, unfortunately is about 15 seconds too long. So I had to find a workaround for it. Threads was the first thing that came to mind, but Python is worthless for this....

Anyhow, let's get straight to the point:

***************************************
Install erlang

Install RabbitMQ

See: http://www.rabbitmq.com/install-windows.html

pip install kombu

pip install celery

pip install django-celery

the above commands might cause errors for sub-installation of kombu and billiard. If this occurs:

install MinGQ

in the directory for billiard (ex: C:\Users\YOUR_USER_NAME\build\billiard)

run: setup.py install build --compiler=mingw32

See: http://blog.eddsn.com/2010/05/unable-to-find-vcvarsall-bat/

then rerun:

pip install kombu

pip install celery

pip install django-celery

then update settings.py according to following link and run SyncDB:

http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html

***************************************

Credits: I have Dan (http://www.linkedin.com/pub/dan-ziembienski/17/720/679) to thank for this.

Scala 2.9.2, Play! 2.0.2, and Amazon EC2

Summary: I describe how to setup a domain on amazon's EC2 (using my personal named server), and the tricks on getting scala + play (2.0.2) to run on a micro instance.

---

Introduction: So recently I got an iphone and while out and about, I wanted to look at the source code of a page on safari. It so happens that I can't do that. (Yes, I know, I took it for granted.) So while there are some apps out there, I refuse to buy an app that I'll use only once. As a matter of luck, it so happened that this weekend there was a "$.99 .com" sale, so I hastily grabbed "www.seesster.com", and considered building a site that "sees what your browser sees".

It so happens that I've been been reading about Scala as well as the Play framework, and Friday night started to code seesster up. I won't go into details of the Scala/Play code and what I'm doing (long story short: scala + phantomjs + pygments), but I will describe how I was able to set this up (running!) on my free amazon EC2 micro instance.

DNS and sub-domains: So when I login to my AWS management console, there are two things that need to be done:

1) I need to name my instance with the exact subdomain I am planning on redirecting the service to. In this case, I've named the instance www.seesster.com

2) I also need to jot down the Public DNS (which I'll use for the CNAME entry of my named config file). In this case, it is ec2-174-129-174-216.compute-1.amazonaws.com

I've highlighted the entries I'm talking about in red....

Next up, I need to update the config file. While I won't show you the whole file, as long as you know *anything* about setting up your own DNS server, this should suffice... In a nutshell, we need to setup a CNAME entry that points to the public DNS. Note the "." after the url string.

www    CNAME    ec2-174-129-174-216.compute-1.amazonaws.com.

At this point, when I tried "www.seesster.com" I would correctly end up on amazon's server. However, this was the easiest part of my weekend.

Play won't run: My coding environment has plenty of memory, so I never thought once on how this could impact me. That is, until I tried to run play on it... "play run", "play dist", "play start" and even "play compile" would end up with memory allocation problems. I was between a rock and a hard place.

Solving this involved:

1) Manually downloading and installing java 1.7 (jdk1.7.0_05).
2) Updating the symbolic links:

$ ll /usr/lib/jvm/jre
lrwxrwxrwx 1 root root 31 Jul  2 03:09 /usr/lib/jvm/jre -> /home/ec2-user/jdk1.7.0_05/jre/

$ ll /usr/bin/java
lrwxrwxrwx 1 root root 35 Jul  2 03:10 /usr/bin/java -> /home/ec2-user/jdk1.7.0_05/bin/java

3) on my dev box, running "play dist" and uploading that stand-alone file to amazon
4) updating the "start" script in the stand-alone folder to:

$ cat start
#!/usr/bin/env sh

exec java -Xms64M -Xmx128M -Dhttp.port=80 $* -cp "`dirname $0`/lib/*" play.core.server.NettyServer `dirname $0`

What this allows me to do is run my play application with limited memory usage on port 80. (Reminder: this is on Play 2.0.2).

Happy ending: Cool or not cool? As of today, www.seesster.com is running and working. I will be updating code during the week for sure. If you are reading this in the future (wow!), and the site doesn't work, I am probably aware of it. After all, a year from now the domain will expire. And unless it is making me money, I'll probably *not* renew it.

:)

mysqldump with wildcard

Today I had to dump an existing database, but only wanted a subset of tables. Currently, mysql does not have a built-in option to do this, but with awk I got this to work:

1) filter out the table you are looking for. connect to you mysql server and test you wildcard combination

> mysql -u user -p test_db
> password: **********
> show tables like 'iesna%'

2) now combine that with awk and mysqldump command

> mysqldump -u user -p test_db 'mysql -DN test_db -e "show tables like 'iesna%'" | awk '{printf $1" "}''

:)

Install MySQL for Python (MySQLdb) on Windows 7

I loved it how people praise python as being so simple, but lack to explain the fact that they either download pre-compiled packages and libraries, and/or use linux. Lackluster is what comes to mind when I refuse to download this (http://code.google.com/p/soemin/downloads/detail?name=MySQL-python-1.2.3.win32-py2.7.exe) and rather try to compile from scratch on windows 7 (using python 2.7, mysql community 5.5).

Given that I've put in more than 15 minutes to get this rolling, I thought I would blog about it. There are other options online, but nothing better than documenting your failures, right? :)

1) First step is installing mysql *with* the development tools. Also the Mysql Connector C (http://dev.mysql.com/downloads/connector/c/).
2) Next up is installation of the setuptools (http://pypi.python.org/pypi/setuptools). And while there is a windows executable, that didn't work for me. So download the ez_setup.py and it should download/update your lib.
3) Check if your site.cfg file has the correct registry key for your version of mysql server (registry_key = SOFTWARE\MySQL AB\MySQL Server 5.5).
4) Also double-check if your mingw path is updated in your environment variables (computer >> properties >> advanced system settings >> environment variables >> ADD to your path "C:\MinGW\bin")
5) You need to comment out the lines regarding to the static build in setup_windows.py, as well as instructing the compiler to use the dynamic lib version (add 'client = "libmysql"').

    # XXX static doesn't actually do anything on Windows
    #if enabled(options, 'embedded'): B
    #    client = "mysqld"
    #else:
    #    client = "mysqlclient"
    client = "libmysql"

6) If you are lucky, running python setup.py build –compile=mingw32 will work. I wasn't as lucky since I got a missing 'config-win.h'. (Check this out: http://stackoverflow.com/questions/1972259/mysql-python-install-problem-using-virtualenv-windows-pip).