Thursday, December 29, 2011

No error dialogs in eclipse headless test

Here's a small funny thing I learned a while ago while running GUI tests of Eclipse application and now stumbled upon it again.
When running Eclipse tests headlessly, certain dialogs don't show up as they would normally (e.g. when writing/recording the test). This is because of flags:

org.eclipse.jface.dialogsErrorDialog.AUTOMATED_MODE = false;

So if you for example have a test scenario that DOES expect an error dialog to show up, make sure you have above flags set correctly, or find out a different assertion than waiting for an error dialog.

Monday, July 25, 2011

Orion PHP Support demo

If you're following Eclipse planet, you couldn't have missed Karol Gusak's weekly status reports about PHP support that's being built for Orion.

If you still haven't tried PHP in Orion yourself, here's a little demo now on YouTube that nicely shows syntax highlighting, content-assist as well as overall PHP experience in Orion. Click below to check it now.

Thursday, July 21, 2011

Making extensible GUI with org.eclipse.ui.menus

It's actually first time I had to make a custom part of GUI extensible. In an Eclipse forms editor like following, we wanted custom actions to come from various plugins:

Since eclipse APIs provides this sort of extensibility with their org.eclipse.ui.menus extension point, I looked at how to reuse that mechanism. It turns simple. All you need to do is, while creating the editable area, add:

final Composite parent = ... // that's the area we want to make extensible

ContributionManager contributionManager = new ContributionManager() {
public void update(boolean force) {
for (IContributionItem item : items) {

IMenuService service = (IMenuService) getSite().getService(IMenuService.class);
contributionManager.add(new GroupMarker("testing"));
service.populateContributionManager(contributionManager, "toolbar:org.zend.editor1?after=testing");

Now to contribute something to that area, you need to extend org.eclipse.ui.menus extension point:

<extension point="org.eclipse.ui.menus">
<menuContribution allPopups="false"
… Your contributions here …

So now, if you for example prepared some commands in advance, you can easily link them without any extra coding. Since I wanted some nice looking link, I used control contribution. It let's you create entirely custom widget.

See the end result:

Lessons learned:
1. When specifying locationURI, it's worth choosing 'toolbar', over 'menu' or others because only 'toolbar' locations support custom control contributions.
2. Only supported URI parameters are 'after', 'before' and 'endof'. When using different parameters, I'm not sure how (if at all) it will be handled and where exactly your contribution will be landed.
3. Not quite sure why, but all the existing docs that I have come accross, only say how to use menus extension point in standard places like view menubar, window toolbar, etc. Nothing about re-using whole mechanism..

Thursday, July 14, 2011

Starting and immediately using OSGi bundle: possible?

Lately I had this little thing to do: an OSGi bundle (SuperTool) comes with some other OSGi bundle (GreatSdk) included.
SuperTool depends on logic in GreatSdk so the first thing it needs to do during activation is to install GreatSdk and immediately after that start loading classes from GreatSdk.

At first thought it seems logical to make SuperTool-GreatSdk dependency optional, to let SuperTool install and start safely without GreatSdk. Once installed, during startup SuperTool should install GreatSdk and start using it, for example:

BundleActivator.start(BundleContext context) {
greatSdk = context.installBundle("file:///path/to/great.sdk_1.0.0.jar");

GreatSdkStarter gss = new GreatSdkStarter(); // separate class in SuperTool that references some classes from GreatSdk
gss.start(); // creates some GreatSdk classes

Unfortunately this fails in gss.start() with ClassNotFoundException:

java.lang.ClassNotFoundException: greatSdk.Sdk
at org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(
at org.eclipse.osgi.internal.loader.BundleLoader.findClass(
at org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(
at java.lang.ClassLoader.loadClass(
at supertool.GreatSdkStarter.start(
at supertool.Activator.start(

What does work is reflectively creating GreatSdk class instance via obtained Bundle:

// again in superTool activator.start()
Class clazz = b.loadClass("greatSdk.Sdk");
Object ob = clazz.newInstance();

So problem seems to be in the SuperTool information about the world. It looks like SuperTool is not yet aware that greatSdk was installed in the meantime.

Another thing that works is refreshing SuperTool via OSGi console:

osgi> start superTool
// loads greatSdk, but fails with ClassNotFoundException for classes referenced in GreatSdk
java.lang.ClassNotFoundException: greatSdk.Sdk....
osgi> ss
1 ACTIVE greatSdk
2. RESOLVED superTool

osgi> refresh superTool // magic
osgi > start superTool

After the refresh, starting superTool again makes it properly see greatSdk. Unfortunately doing refresh from within superTool activator.start() doesn't work so sweet. PackageAdmin.refreshBundles(superTool) terminates successfully, but CNFE is still thrown.

What's other way to activate and immediately use optional dependencies in an OSGi bundle?

(Except 'don't do it' :-) )

Wednesday, July 13, 2011

Installing V8Cgi on Mac

V8CGI is a very thin Apache module encapsulating V8 - Google's JavaScript engine that's used e.g. in Google Chrome. Those concerned with NodeJS stability may consider V8CGI a better choice, thanks to stability provided by Apache base.

A short while ago, on DevCamp we've been playing with V8CGI, RingoJS and Node. Thanks to good performance, intuitive and high quality APIs, V8CGI turned out to be the black horse of the whole event, although installing it on Macs was little tricky.

Readers with Windows and Linux should follow to V8CGI home page, where they can get precompiled Windows binary and learn how to install binary on Ubuntu.

Mac users, let's open terminal and try to build the V8 apache module ourselves.

Getting V8CGI to work is four steps:
1. Getting the source code
2. Building it
3. Configuration
4. Test drive

1. Getting the source code
When checking out sources from SVN, we actually need two packages V8 and V8CGI:
a) svn checkout v8cgi
b) svn checkout v8

You should end up with two new directories: v8 and v8cgi. When downloading a ready package from V8CGI home page, V8 sources are already included.

2. Build V8 and V8CGI
We'll use scons - a software construction tool. A good way to get started with scons is:
scons -Q -h

It gets you a list of available build configuration options. Worth visiting when you hit a wall compilation errors.

Let's first build V8:

cd v8
scons library=shared arch=x64
sudo cp libv8.dylib /usr/lib/libv8.dylib

We need x64 arch, as that's default Mac's Apache architecture. As a result your newly built libv8.dylib should be now available in /usr/lib directory.

Now let's build V8CGI.

cd ../v8cgi
scons module=1 mysql=0 gd=0 apache_path=/usr/include/apache2/ apr_path=/usr/include/apr-1/

Note that I added some extra options (run scons -Q -h for list of all options):
mysql=0 - disabled mysql support, that is on by default. Not quite sure if all I need to enable this option is on my Mac, and I don't need mysql for now, so let's just disable it.
gd=0 - disabled image-processing library 'gd', that is on by default. Same as with mysql, let's try without gd for start.
apache_path and apr_path - that's the path to Apache header files. Necessary to build against our HTTP server.

For me this command ends up with following error:

Undefined symbols for architecture x86_64:
"_iconv_open", referenced from:
ByteStorage::transcode(char const*, char const*)in bytestorage.os
"_iconv", referenced from:
ByteStorage::transcode(char const*, char const*)in bytestorage.os
"_iconv_close", referenced from:
ByteStorage::transcode(char const*, char const*)in bytestorage.os
ld: symbol(s) not found for architecture x86_64
collect2: ld returned 1 exit status
scons: *** [mod_v8cgi.dylib] Error 1
scons: building terminated because of errors.

Above error message says that 'iconv' library is missing for our architecture x86_64. Iconv is used for converting different character set encodings, like UTF to ISO, ISO to ASCII, etc. It's not critical part of V8CGI, specially if you only want to evaluate that technology.

For evaluation purposes, we could just rip out the iconv calls from v8cgi source code but let's build iconv ourselves. Create another directory "iconv" next to v8 and v8cgi and execute following steps there:

curl -O
tar -zxvf libiconv-1.11.tar.gz
cd libiconv-1.11
make install

Note that latest iconv library is 1.13.1. We're using 1.11, because that one is compatible with iconv version included by default on Mac. Building with 1.13 you will end up with following error when trying to start apache next time:

httpd: Syntax error on line 118 of /private/etc/apache2/httpd.conf:
Cannot load /usr/local/apache2/modules/mod_v8cgi.dylib into server:
dlopen(/usr/local/apache2/modules/mod_v8cgi.dylib, 10):
Library not loaded: /usr/local/lib/libiconv.2.dylib\n
Referenced from: /usr/local/apache2/modules/mod_v8cgi.dylib\n
Reason: Incompatible library version: mod_v8cgi.dylib requires version 8.0.0 or later, but libiconv.2.dylib provides version 7.0.0

Let's go back to v8cgi directory and continue it's build. We need two extra params to link our new iconv module:

scons module=1 mysql=0 gd=0 \
apache_path=/usr/include/apache2/ \
apr_path=/usr/include/apr-1/ \
libpath=/usr/local/lib cpppath=/usr/local/include

The build proceeds few steps forward and fails again with:

g++ -o mod_v8cgi.dylib -bundle -bundle_loader /usr/sbin/httpd src/common.os src/system.os src/cache.os src/gc.os src/app.os src/path.os src/lib/binary/bytestorage.os src/mod_v8cgi.os -L/Users/jacek/Downloads/aa/v8 -L/usr/local/lib -lv8 -lv8 -lapr-1 -laprutil-1
Undefined symbols for architecture x86_64:
"_libiconv_open", referenced from:
ByteStorage::transcode(char const*, char const*)in bytestorage.os
"_libiconv", referenced from:
ByteStorage::transcode(char const*, char const*)in bytestorage.os
"_libiconv_close", referenced from:
ByteStorage::transcode(char const*, char const*)in bytestorage.os
ld: symbol(s) not found for architecture x86_64
collect2: ld returned 1 exit status
scons: *** [mod_v8cgi.dylib] Error 1
scons: building terminated because of errors.

This time, compilation of v8cgi library is almost done, but breaks on gluing everything together. Let's invoke the same command adding -liconv:

g++ -o mod_v8cgi.dylib -bundle -bundle_loader /usr/sbin/httpd src/common.os src/system.os \
src/cache.os src/gc.os src/app.os src/path.os src/lib/binary/bytestorage.os src/mod_v8cgi.os \
-L/Users/jacek/Downloads/aa/v8 -L/usr/local/lib -lv8 -lv8 -lapr-1 -laprutil-1 -liconv

And success! No errors and your mod_v8cgi.dylib is out there, ready to try.

Configuration is pretty well described in v8cgi/INSTALL text file so I'll just copy it here:

sudo cp v8cgi.conf.posix /etc/v8cgi.conf

- configure /etc/v8cgi.config
sudo vi /etc/v8cgi.conf

- configure the apache module:
sudo vi /etc/apache2/mods-available/v8cgi.load
* The only line in the file should be:
LoadModule v8cgi_module ~/src/v8cgi/
sudo vi /etc/apache2/mods-available/v8cgi.conf
* The only line in the file should be:
AddHandler v8cgi-script .sjs .ssjs .jst

Restart Apache and you're ready to go!

Tuesday, July 5, 2011

Search by multiple keys in CouchDB

While developing a simple Twitter clone on V8CGi + CouchDB stack, we faced an interesting problem.
We have a CouchDB database with users and messages. Each user has many followers and he'd like to see messages from all these
followers. So we need to query tweets database for all documents that have author belonging to the list.
In SQL it's something as simple as:

SELECT * FROM tweets WHERE author IN ("szymon", "wiktor", "marek") SORT BY timestamp;

In CouchDB it turned out to be an open issue

As far as I remember we ended up with running two separate queries. I'd be happy to hear how others solved this!

Thursday, April 28, 2011

Signing firefox extensions

Signing FF extensions is pretty straightforward. Lately I've been going through some strains related to them, so I thought I'd share lesson learned.

You have two sides that need to match with each other: online update URL and extension itself. Online update URL is a single file - update.rdf. Extension is an XPI file, with install.rdf file inside.

Update.rdf contains a list of available extensions. Each extension should have checksum information, e.g. in sha1 algorithm.
Once update.rdf is complete with all information and checksums, it must be signed with signer key.
Note that, every time you change any of extension, it's checksum will likely change and so you need to update and re-sign update.rdf as well.

On the other side, the install.rdf included in our XPI file needs to have our public key. Once you create install.rdf and put public key there, you can forget about it. XPI file does not have any checksum or signing.

This all sounds nice and well but sometimes it doesn't go as well. Let's suppose public key included in XPI file was wrong and the file is already in the public. Every update attempt will fail with error about malformed signature or updateKey. That's because during update, Firefox tries to match public key in XPI to signature in update.rdf.
The only solution is to manually install extension again, with correct public key. It's necessary to clear browser cache before performing manual install, because FF uses cache first if the extension is already there.

Detailed information about RDF file formats is here.

Friday, February 18, 2011

Building Windows installer with WiX

WiX is Windows Installer XML (WIX) Toolset. It's an open source project coming from Microsoft. Actually, it was the first project that Microsoft has officially published under Open Source license. Initially it appeared on SF.NET, but recently has moved to Microsoft's owned Half of the internet still points to SF though.

This tool let's you pack a bunch of files into a user-friendly MSI installer, including GUI, silent install, repair, windows registry editing and Start menu customizations.

WiX is a bunch of cool ideas. The first one goes with the each tools names. There's Heat, Candle and Light. They act as preprocessor, linker and compiler. So no matter whether you know everything about compilers, or you always left it to the IDE - tools seem friendly from the very first look.

As name says it, the spine of the toolset is XML. To turn on the heat, developer needs to specify files to install, user interface, user interactions, install steps, resources and custom actions. All this lands in one or more XML documents following provided XML Schemas.

Microsoft Visual Studio users are lucky, as there's GUI in VS that hides WiX to some extent. However I'm not sure how much flexible it is, i.e. how far can you go without having to manually edit XMLs. I was working on plain XML file in Eclipse IDE. Thanks to XSD Eclipse was able to provide content assist. This saved me from many typo-like bugs and helped find right tags faster, than by looking at the docs.

The installer to build had some requirements. It had to be good-looking, native user experience. Some 3rd-party artifacts needed to be installed in the system as well: a dll, another product msi, and an exe to invoke in install time.

User interface
Installers are typically wizards, built of couple of pages. WiX provides several default pages for standard operations. They're easily customizable with some space for branding. Unfortunately some predefined pages look bad. For example features selection dialog uses ugly drop-downs, rather than checkboxes. Maybe it's useful in some scenarios, but for 99% cases they're confusing.
Those not happy with default GUIs can implement their own by putting basic widgets (Button, Text, Label) together. All in XML. That's a bit of work for a beginner who previously built GUIs only in HTML or Java. There are also little caveats here and there, that can turn simple changes into a whole day of work. For example, there's no support for transparent background in checkboxes, or background color is different on various Windows versions. So good luck, if you're planning for many customizations.

Overall, user interface looks nice. It's simple, minimal, doesn't hurt an eye. But at the same time, it's not something impressive, not eye catching.

Installing 3rd party stuff
It's very simple to invoke any system command from WIX. If system command is not enough, then a DLL or VB script can be called as well. Unfortunately it get's more complicated starting with Windows Vista and Windows 7. They have UAC (User Account Control) which denies pretty much all activity that was not explicitly accepted by user - e.g. blocks write access in some areas, disallows some programs. This is not really WIX issue, but being closely coupled to underlying MSI technology, WIX exposes some difficulties that we usually don't want to see or understand. So, any custom action (e.g. custom command invocation), that requires UAC signoff (or "privileges elevation") must be ran deferred. So in fact, it will be ran separately after the installer. Unfortunately deferred command does not see any properties or variables set during install process. So for example, if you want to remove extra files on uninstall, you need to think about communication mechanism, that will tell remove command the location to be removed and then run remove command deferred. That's much more than "rm -rf $DIR". The advantage is that you decide what privileges are used for which step (it's not so obvious with other 'advanced' tools).

Installing files
When installing the files, MSI maintains extra bookkeeping for rollback support. Unfortunately this is quite time consuming. In our case, whole application was 500M which meant extra precious minutes of installation time (and first impression) only for ability to click "Cancel"/"Repair". Disabling this bookkeeping is supported only in more recent MSI versions. This makes MSI technology look very bad when compared to any Archive/ZIP-based installers. The good side is, that latest MSI (e.g. included in Win7) is pretty fast, and it's only Windows XP where things are slow. During uninstall, naturally, only the installed files are removed. This is usually advantage, but if application generates any files, like cache, or stores files in many locations, then we want that removed as well. There's no out-of-the-box "remove dir" functionality.

MSI supports updates. Update can replace particular files in install. This time, our tool was managing updates by itself and there was no easy way to automatically map those updates to certain files. So we didn't use this functionality.

That's pretty much my first top five thoughts about WIX. At the end, it's also worth mentioning the community. WIX has extremely lively forum. Only rare questions pass without an answer. Unfortunately not many people make an effort to dig a bit into archives, so many basic questions get asked all the time.

I've got some other installer tooling to blog about as well. So for next entry, I need to decide whether to get deeper into WIX or give an overview of it's alternative. Don't hesitate to comment, if you'd like to hear anything in particular about WIX.

Friday, January 14, 2011

Two Eclipse p2 repository operations to make you happy

On the integrations side of the Eclipse business I often happen to need two things:
1. add a random OSGi jar to repository
2. copy from one repository to another

Looking thru p2 manual it's not always entirely obvious how to do it, so here goes my take on the subject.
All my automation uses Ant, so I'll speak a bit of Ant now:

1. Add a random OSGi jar to repository

repository="file:/path/to/destination/repository" e.g. ${buildDirectory}/repo
source="/absolute/path/to/folder/with/jar" />

Few notes:
1. source attribute must be an absolute path.
2. source path must contain "features" and "plugins" directories. So if you have only one bundle, make sure to put it under plugins dir first.
3. make sure repository attribute is a URL, not plain path

2. Copy from one repository to another

<p2.mirror verbose="true">
<repository location="file:/path/to/destination/repository" />
<repository location="file:/path/to/source/repository" />
<iu id="pluginName" version="" />

1. Why not use p2.director? It copies only bundles for current architecture
2. Mirror task is still not perfect, but enough for most copying operations, it has some extra options, worth looking into.

Wednesday, January 12, 2011

Extending Eclipse Orion

Yesterday Eclipse E4 team announced Orion - an experimental Eclipse Web IDE. There's plenty of information about the project on Orion wiki.

Orion is a set of JavaScript and HTTP APIs that simplify common development part of rich applications, like file storage, preferences, authentication, dialogs, selection. Since it's coming from Eclipse, Orion is skewed towards IDE development, so from start it also includes text editor with some decent features, like JavaScript syntax highlighting and JSLint integration.
The project may go towards strengthening it's JS IDE feature set, as well as enhancing it's core side - extensibility, client APIs ...or do it all at once. In any case it's going to be fascinating journey!

So how to extend Orion? One can re-implement existing HTTP API in language other than Java; add more HTTP APIs, e.g. for task management, calendar, etc. Same with JavaScript API - it seems fairly easy to add more. Looking at this from classic Eclipse adopter point of view, I tried to add some GUI contribution that brings new functionality to existing Orion - let's say a PHP editor to complement the JS one. Naturally all without leaving Orion. I took following steps:
1. create new Orion bundle with new editor
2. install on Orion server
3. test and improve the contribution

Creating new bundle for Orion
Orion is technically a simple Equinox http server, so it's pretty straightforward. We need MANIFEST.MF with bundle description, plugin.xml that describes what web content should be published and the content itself.

New bundle structure, looking at it from Orion:

Pretty much ,standard Eclipse plugin manifest.

This one specifies that our bundle contains some static web content under /web directory and that it should be accessible in the container under /simple alias. Straightforward, thanks to Equinox.

Interestingly, at the moment Orion team decided to put all configuration in one bundle, so e.g. webeditor or webide JS code currently doesn't decide on it's own where will it appear, but instead a separate configuration bundle defines that.

Installing bundle into Orion
To install just created bundle, we need to figure out where is it stored and install it. By default Orion stores files in serverworkspace directory in it's home dir. On Mac this happens to be \\Contents\MacOS\serverworkspace

In order to install bundle, it's best to use equinox osgi console - available in Orion thanks to the fact that it's equinox application. Orion docs mention something about server console, but I'm not entirely sure if this is the same, or some other console.
So first start Orion with osgi console:

./eclipse -console

and then, install the bundle:

osgi> install file:/Users/jacek/Downloads/eclipse-orion/
Bundle id is 75

osgi> start 75

I used absolute path from Orion workspace, to easily make updates later. Surprisingly my project dir is named "A" instead of it's real name. Another interesting thing is that there's no all the well known .project and .settings files. Web developers are going to like it, and I'm wondering where is this information stored currently.

Test and improve the contribution
So now that our bits are installed, let's see them:
http://localhost:8080/sample/index.html and yes! It works!

Curious mind would be interested what was the source of the index.html, that generated such nice "Hello [object Object]" message. I included some Orion Client API to check if it's available and yes, it is! From now on, further documentation can be found at Orion Client API wiki.

The source is following:
After updating the bundle contents - e.g. to turn this sample to real new PHP editor - there's another command required to make Orion see our changes. From osgi console type:

osgi> update org.eclipse.sample

The only thing that I was not able to figure out is how to actually add a link or other UI contribution to code editor, files navigator, or other existing UIs. At the moment user has to manually enter the URL to see my contribution. I'd be happy to hear how to do this, if you happen to know.