Henrik "Mauritz" Johnson's Blog

Flow control for your script

Henrik “Mauritz” Johnson — Wed, 07 Oct 2009 19:16:00 GMT

Undocumented functionality in TDA, Toad for SqlServer, Toad for DB2 and Toad for MySQL to use advanced flow control when executing scripts.

Inside the application data directory

Henrik “Mauritz” Johnson — Thu, 23 Oct 2008 07:28:00 GMT

If you have ever looked in the Toad data directory you were probably a little bit bewildered about all the stuff in there. This article will try to shed some light on what all the files in there are used for by the Toad application.

Code completion revealed

Henrik “Mauritz” Johnson — Wed, 22 Oct 2008 00:44:00 GMT

From the first version of Toad we have always had some form of code completion (It was originally called Toad insight or something like that) but it has always been a fairly simple affair where you either got a list of columns in select statements or a long list of objects of many different types completely unaffected by the context in which you invoke the code completion.

Starting in 4.0 the code completion support was completely rewritten to be very context aware. Now Toad should know that if you are writing a select statement and are between SELECT and FROM you probably want a list of columns and if you write JOIN you probably want to join on a foreign key from tables already specified etc. There has been little bit of a backlash on this and I have to admit that this code did have some bugs and when that happened you usually didn't end up with any code completion at all which is worse than what we had in 3.x. Hopefully for the soon to be released 4.1 release most of these should have been fixed and you will really see the benefits of the new framework.

Handling code completion in this way is kind of complicated. The main problem is you can't do strict parsing because we are not dealing with complete statements. Instead we have to do our best at guessing what the you are aiming at writing.

So let me go through the changes in more detail what new features are introduced with this framework.

Smarter auto invocation

The most obvious change that people will immediately see is that as you start typing a select statement the moment you write "FROM" and then space you will be presented with a list of tables and views. Before 4.0 the only time we automatically invoked code completion was when you typed a dot. In 4.0 we have certain "hot tokens" that invoke code completion after you write a space as well. The full list of tokens isn't that important, but if you ever think we are missing something just let us know because it is very easy for us to change it.

Context sensitive object lists

We now look at the statement type to decide what kind of object list you probably are interested. For instance if you write "CALL" or "EXEC" you are probably interested in a list of procedures, when you are writing "UPDATE" you probably want a list of tables while when you are in the "FROM" clause of a select statement you probably want both tables, views and in case it's supported functions returning tables.

Context aware sub-select completion

Toad should now keep track of sub selects so that it knows which objects you can address when you are writing complex sub-selects as shown in the screenshot below.

Auto generated joins

When you are writing select statements we now automatically generate the full join statement based on foreign keys in the database. We can generate both the part following "JOIN", the part following "ON" or in case of a cross join the part in the "WHERE" clause.

Snippet code completion

In some cases the code completion generated still need you to fill in some extra data to be a valid statement. We now do this by inserting an auto generated code snippet. Specifically we use this when you are using code completion on code to let you fill in the parameters of the call we generate.

Complete statement generation

We can now generate the full statement for some statements after you have specified the database object to operate on. Specifically this includes update, insert and delete statements.

Good handling of owners

This only applies to SQL Server, but in 4.0 we really didn't have that good support for handling owners in code completion. As of 4.1 that we now handle owners in a really clever way. Basically it should now "just work". If SQL Server accepts the syntax we should handle it correctly.

Support for local variables in batches

On SQL Server we now keep track of local variables defined in a batch and includes these in the places where they make sense.

Options available for code completion

The option page for code completion has changed quite a bit between 3.x and 4.0.

First some things that haven't changed. The auto list members, quoting, casing, expand "*" and include system objects work exactly as before so I will not go into these (And they should be pretty self explanatory).

Item sort order

This setting now has one more item which is the default and that is the "Each category". This means that the code completion list is sorted per type of items. For instance in a select statement we start with the "*" and then columns followed by variables, snippets and a bunch of other stuff. Chances are that you are looking for the columns and in this case they will be at the top instead of being lost in the list of hundreds of function snippets also included.

Merge secondary items

When you invoke code completion in some cases we have items that are common to choose from and some items that are not so common. Normally the not so common items are not included in the code completion, but if you want to get to them you can still do so by choosing the last item in the list which says (More).

If this option is checked this means that we should generate the code completion list with all items right away instead of separating this into two different lists. This is the default.

Known issues

The one known issue in regards to code completion is that we currently do not honor the setting to separate statements with double newlines (On by default I think). For version control to function property with multiple statements in the editor you need to end you statements with ';' (Except for SQL Server which doesn't have this issue).

Connection and transaction handling explained

Henrik “Mauritz” Johnson — Mon, 20 Oct 2008 19:10:00 GMT

In an application like Toad connection and transaction handling will invariant become a complicate affair. During the years we have been working on this we have tried to make it as intuitive as possible so that you usually don't have to think about it. However, in some cases you can still run into problems so this post will try to explain how we handle connections and transactions so that you can avoid the pitfalls.

Toad version numbers explained

Henrik “Mauritz” Johnson — Mon, 20 Oct 2008 19:08:00 GMT

Version numbers between the different components of the different products can be pretty confusing ever for us who work on it. Before 3.0 of the Core (The part of Toad that is shared between the products) each product had it's own release schedule and with that each had it's own release version of this. After release 3.0 we now try to coordinate releases between the different products (They are usually staggered with a few weeks, but they are all based off of the same core version now).

When I talk about versions in this blog I will usually unless otherwise specified refer to the Core version. This should in most cases correspond to the version of the product except for TDA in which case you just reduce 2 major release version.

Also to confuse matters even more a product can choose to skip a release (As for instance the DB2 product is doing now with the 4.1 release). Usually this happens because we need more time on that specific platform to get the features in that we want to have in a specific release.

If you are interested in what Core version your Toad is currently running you can check the .Net assembly version of the ToadCore.dll assembly of your application.

Here is a list of how the different versions of Toad relate to each others and the corresponding Core versions.

Core	Toad for SqlServer	Toad for DB2	Toad for MySQL	Toad for Data Analysis
1.0	-	-	1.0	-
1.1	-	-	1.1	-
1.2	1.0	-	-	-
1.3	-	1.0	-	-
2.0	-	-	2.0	-
2.1	2.0	-	-	-
2.2	-	2.0	-	-
3.0	3.0	-	3.0	-
3.1	3.1	3.0	3.1	1.0
3.2	3.2	3.1	3.2	1.1
4.0	4.0	4.0	4.0	2.0
4.1 ¹	4.1	-	4.1	2.1
4.5 ¹	4.5	4.5	4.5	2.5

¹ Since these releases are not done yet this information is subject to change.

Reducing startup time

Henrik “Mauritz” Johnson — Mon, 20 Oct 2008 19:08:00 GMT

For the upcoming 4.1 release I have actually been able to spend quite a lot of time to reduce startup time in Toad and hopefully there should be a quite noticeable difference. One of the changes made is to make the initialization multithreaded so the improvement should be even more pronounced if you have more than one CPU core.

There are a couple of other things you can do yourself in both 4.1 and versions before that to reduce startup time if you are annoyed by it.

Keep auto connect's to a minimum.
Don't show the web page on startup (This one will shave several seconds off your startup time because we don't need to load the IE component).
Disable the startup croak. This really doesn't save you much so if you don't mind it just leave it as is.
Don't both show connection manager tool window and new connection dialog on startup. The list of connections is a complicated control that takes a while to create so there really is no need to create two of them. Choose one way of opening your connections.
Try not to have every open tool window showing. Tool windows that are showing are initialized on startup and makes the application startup slower. Tool windows are not created until they are first shown so if you can't see them we don't waste time on them. Auto hide is also your friend here as the tool window isn't actually created until you zoom it in when auto hidden.
I might also add a note that collecting usage statistics takes so little time I can hardly measure it when profiling. Please enable it, it is really useful for us to evaluate where to concentrate our efforts for future development.

These two tips are only for advanced users who don't mind some more drastic methods to reduce startup time.

If you don't customize your toolbars and menues you can turn off saving and restoring these by unchecking this checkbox.
If you are ok with wasting some memory you can create a shortcut to run toad /prestart=true and then put that in your startup program folder. This will make Toad start pretty much instantaneous but the feature is not that well tested and might have some issues (Especially in regards to automation and running multiple instances of Toad).

Hopefully these tips will help you get to work faster in the future. As you can see from the previous post on the history of Toad for SqlServer we have already gone from 12 seconds to 5 seconds even though we have added tons of functionality.

Toad nomenclature

Henrik “Mauritz” Johnson — Mon, 20 Oct 2008 19:06:00 GMT

We have a bunch of terms referring to different aspects of the Toad application internally and I thought I would share some with you because it's not that uncommon that it slips out in the Yahoo Boards when we talk about it so why not just go through them and explain them.

Command line reference

Henrik “Mauritz” Johnson — Mon, 20 Oct 2008 19:03:00 GMT

The documentation of what command line options are available on the Toad.exe is not necessarily that well covered in the documentation. True there is rarely any need to specify anything on it unless something goes wrong. I just thought I'd go through what is available.

First of all the format. Toad allows any parameters to be specified regardless of if it knows what they are or not because some plugins might want to read it later on unbeknown to the original startup code. All parameters are specified using this format.

toad /{parameter name}={value} {files to open on startup}

Also if you want to specify an option with space in it you can quote it like this.

toad "/{parameter name}={value}" "{A file to open}"

Here are the following parameters that we support in Toad as of version 4.1.

Parameter	Value	Description
datadir	Directory	Specify an alternative data directory for Toad to read it's settings from instead of the default location.
profile	"true"	Start Toad and generate profiling information. This is sometimes used by us if you Toad is behaving very slow to track down what is wrong. The information generated is included in the support bundle if it is available.
batch	"true"	Run toad in batch mode. This means that it will just open all available files without opening up the user interface and then exit. Used by automation.
prestart	"true"	Pre start Toad. This means that toad is run in the background to be available to "pop up" when you actually want it. This feature is never actually used but if you are ok with pre loading on windows startup you can add toad /prestart=true to your startup menu and have lightning fast startups when you actually need Toad. Will use memory even if you are not using Toad. This is how a lot of other apps fake quick startup times like for instance Adobe Reader.
installinit	"true"	Used by installation to initialize a cache of the available plugins of to the application. When run by this it needs to write to the install directory, but in case you have this which is normally the case you can rerun this as many times you want.
feedback	"true"	Enable usage reporting.
license	"edit"	Display the license entry form on startup.
transpose	"1"	Allow transposing when exporting grids. That means switching X and Y axises of the grid.
exportbenchmark	"1"	Generate a log file to help benchmarking the export engine. The file will be included in your next support bundle
excelverbose	"1"	Generate a lot of verbose info when exporting into Excel.
profilevc	"1"	Generate a profiling log to help benchmarking the version control code. The file will be included in your next support bundle
debuglog	Filename	Generate a log of some debugging events. The file will always be placed in the application data directory.
yukondebuglog	"1"	Generates debugging information for the SqlServer 2005 debugger. The file will be included in your next support bundle.
dbmsverbose	"1"	Generate very verbose DBMS_OUTPUT output.
oraclelastplan	"1"	Include the latest Oracle explain plan to your support bundle.
db2debuglogreports	"1"	Generate debugging information for the DB2 debugger. The file will be included in your next support bundle.
db2debuglog	"1"	Generate debugging information for the db2 command line executions. The file will be included in your next support bundle.

To end with an example running Toad like this will enable usage reporting and using a new application data directory and then open a file on startup.

toad /feedback=true "/datadir=c:\Users\MyUser\My Documents\ToadDir" "c:\Users\MyUser\My Documents\A Sql File.sql"

The history of Toad for SQL Server

Henrik “Mauritz” Johnson — Mon, 20 Oct 2008 18:55:00 GMT

I recently took some time to look through the different releases we have made of Toad over the years since we started and just thought I would share it with you guys. I chose the Toad for SQL Server release because that was the one which had made the most releases (For instance Toad for DB2 never made the 3.1 release). I have to say that looking at from where we have come Toad is becoming a lot more polished. Especially starting with 4.0 it seem that we have started putting more time into polishing stuff up than just trying to cram as much features as possible. It is also pretty remarkable how little Toad has changed from our first release. Even if you went straight from 1.0 to 4.0 you would probably feel pretty at home with the application even though there is a lot more functionality in it.

This only goes through some really superficial first impressions. There is tons of functionality that has been added all over the application which is not covered at all in this retrospect. Also this only covers the .Net based version of Toad (Everything except the original Toad for Oracle). Finally the Toad for Data Analysis obviously has a very different look even though it is based on the same code and that isn't covered covered either and it isn't that interesting either since it did not join "the family" until version 3.0 of Toad for SQL Server and as such hasn't seen that much change.

Toad for SQL Server v1.0

When you start the app up you get nothing but the startup page (Which is interestingly a different page in this version than all the other versions) and a new connection dialog. Also fetching around a 200000 record long query takes over 1 minute in this release (It’s also N*Log N so fetching 500K takes over 5 minutes). This is also the only release we made that had the history node in the connection manager.

Toad for SQL Server2.0

Very small visual changes for this release. We added the composite image manager for better icons in the object palette. Also switched out SandDock to UIDock for the tool windows. We also fixed the fetching speed when you fetch everything we are pretty fast in this release, you don’t get any updates until you fetched all the data though. When you fetch just the first 50 rows and then continue we are still really slow. If you look closely you can also see that we switched the tab control in the object palette, added filtering and the column list pane at the bottom. We also added the connection output to the editor. In this release we finally managed to get NGen to work properly so startup time went from 12 seconds in 1.0 to around 9 seconds in this release.

Toad for SQL Server 3.0

This is the first release that figured out that my installed SQL Server instance is called SQLEXPRESS automatically. It also the first with FastData in it so fetching is now pretty fast (Pretty much as fast as before but with updates as you are fetching. This is also the only release which has indication of not null (The blue column header).

Toad for SQL Server 3.1

The most obvious changes here is the new connection tool bar (Which replaced the old connection bar). We also have the new quick search functionality implemented here. We have also added spooling to FastData so we now fetch the 200000 row query in about 5 seconds. This is the first release which doesn’t limit results in the editor by default any more.

Toad for SQL Server 3.2

Here the most obvious change is the panes of the editor. Apart from that there are few visible changes from before. This was mainly a stability improvement release.

Toad for SQL Server 4.0

Now this one is different! First of all we trimmed the grids so that you see a lot more rows from the start. The tabs in the editor has been tweaked to something that most people will find more useful (And they also follow the skin of the main application). The browser is in tree view. You also no longer get the configuration wizard on start up. The toolbars are rearranged so you can see all the buttons (We also no longer show the connection toolbar by default). A lot of the icons have been cleaned up. This is also the first release where the two tool windows showing on the left show up that way by default (In all previous screenshots I opened and rearranged them manually). And also we add the auto hidden tool windows to the right of the screen. And finally we changed the name of the object palette to object explorer.

Toad for SQL Server 4.1 Beta

An interesting change here is that we fixed a bug all the way from 1.0 of the product to not apply a skin by default. We also fixed the “edit” indicator button on grids. You can also see that the connection output is now gone and replaced with the messages tab. More work on cleaning up icons. We also see that we don’t expand the database node by default anymore in difference from 4.0. Another change in the 4.1 release is that on my machine this starts in around 5 seconds. Every release up to this release has started in around 9 seconds except for the 1.0 release which took 12 seconds.

Starting this blog

Henrik “Mauritz” Johnson — Mon, 20 Oct 2008 18:44:00 GMT

Every once in a while I've been writing Toad stuff on my personal blog and I figured it is time to stop doing that and try to do this blog into something a little bit more useful. Entries in this blog will also be cross posted on the site Toad Wrangling

I will try to write tips and tricks about how to use Toad as I can think of them.

I'm one of the original developers of the .Net versions of Toad (All products except the original Oracle one) and currently work on what we internally call the "core" which means the piece of the application that is shared between the different platforms.

For feedback and direct communication with all the developers of the product the best venue is till the Yahoo Groups. The information here has not gone through the normal peer review process of our normal documentation and sometime it might be about features that have not been properly tested (And that is why we aren't documented because we aren't ready for people to start banging on it). So use the information you find here at your own risk. Also please excuse me for all my spelling and grammatical errors, English isn't my first language (Truth be told I'm not that great at spelling in Swedish either).

I have a bunch of stuff I want to write about already so originally there will probably be a bunch of postings as I start out (Assuming I stick to it). After a while it should settle down a bit to just a few posts a month as I've worked through my backlog.