The blog of dlaa.me

TextAnalysisTool.NET is a free program designed to excel at viewing, searching, and navigating large files quickly and efficiently.

I wrote the first version of TextAnalysisTool back in 2000 using C++ and Win32. In 2003, I rewrote it using the new .NET 1.0 Framework - and upgraded to .NET 1.1 later that year. There were a variety of improvements between then and 2006, the date of the last update to TextAnalysisTool.NET. Along the way, I've heard from a lot of people who use this tool to simplify their daily workflow! In the past year, I've started getting requests for 64-bit support from folks working with extremely large files that don't fit in the 4GB virtual address space limits of a normal 32-bit process on Windows. Although .NET 1.1 didn't support 64-bit processes, .NET 2.0 does, and I've decided it's finally time to take the plunge.:)

With this release, TextAnalysisTool.NET has been compiled using the .NET 2.0 toolset and the AnyCPU option which automatically matches a process to the architecture of its host operating system. On 32-bit OSes, TextAnalysisTool.NET will continue to run as a 32-bit process, but on 64-bit OSes, it will run as a 64-bit process and have access to a significantly larger address space. This makes it possible to work with larger log files without the risk of crashing into the 4GB memory limit (which can end up being as low as 1.7 GB in practice)!

Other than a few exceedingly minor string updates, I have made no changes to the way TextAnalysisTool.NET behaves - so the new version should feel just like the previous one. The framework update means .NET 1.1 is no longer a supported platform and .NET 2.0 is now natively supported. The included .config file allows the same executable to run under .NET 4.0 as-is (for example on Windows 8 without the optional ".NET Framework 3.5" feature installed).

If you've ever run out of memory using TextAnalysisTool.NET, please give this new version a try! And if not, go ahead and continue using the previous version without worrying that you're missing out on anything.:)

Click here to download the latest version of TextAnalysisTool.NET

Click here to visit the TextAnalysisTool.NET web page for more information

Many thanks to everyone for all the great feedback - I love getting messages from people around the world who are using TextAnalysisTool.NET to make their lives easier!

Aside: As a matter of technical interest, details on the one bug I found with 64-bit TextAnalysisTool.NET: the following code had worked fine for the last decade (message.WParam is an IntPtr via Form.WndProc for WM_MOUSEWHEEL):

int wheelDelta = ((int)message.WParam)>>16; // HIWORD(WPARAM)

However, when that assignment ran under .NET 2.0 on a 64-bit OS, it quickly threw OverflowException! I was surprised, but it turns out this is documented behavior. Because that line interoperates with the Windows API, I couldn't change the types involved - but I could tweak the code to avoid the exception by avoiding the problematic explicit conversion:

int wheelDelta = ((int)((long)message.WParam))>>16; // HIWORD(WPARAM)

Yep, the proverbial one-line fix saves the day!

I wrote TextAnalysisTool.NET a number of years ago to streamline the task of analyzing large log files by creating an interactive experience that combines searching, filtering, and tagging and allow the user to quickly identify interesting portions of a large log file. Although .NET 2.0 was out at the time, I targeted .NET 1.1 because it was more widely available, coming pre-installed on Windows Server 2003 (the "latest and greatest" OS then). In the years since, I've heard from folks around the world running TextAnalysisTool.NET on subsequent Windows operating systems and .NET Framework versions. Because Windows and the .NET Framework do a great job maintaining backwards compatibility, the same TextAnalysisTool.NET binary has continued to work as-is for the near-decade since its initial release.

But the story changes with Windows 8! Although Windows 8 has the new .NET 4.5 pre-installed (including .NET 4.0 upon which it's based), it does not include .NET 3.5 (and therefore .NET 2.0). While I can imagine some very sensible reasons for the Windows team to take this approach, it's inconvenient for existing applications because .NET 4 in this scenario does not automatically run applications targeting an earlier framework version. What is cool is that the public Windows 8 Developer Preview detects when an older .NET application is run and automatically prompts the user to install .NET 3.5:

That's pretty slick and is a really nice way to bridge the gap. However, it's still kind of annoying for users as it may not always be practical for them to perform a multi-megabyte download the first time they try to run an older .NET program. So it would be nice if there were an easy way for older .NET applications to opt into running under the .NET 4 framework that's already present on Windows 8...

And there is! :) One aspect of .NET's "side-by-side" support involves using .config files to specify which .NET versions an application is known to work with. (For more information, please refer to the MSDN article How to: Use an Application Configuration File to Target a .NET Framework Version.) Consequently, improving the Windows 8 experience for TextAnalisisTool.NET should be as easy as creating a suitable TextAnalysisTool.NET.exe.config file in the same folder as TextAnalysisTool.NET.exe.

Specifically, the following should do the trick:

<?xml version="1.0"?>
<configuration>
  <startup>
    <supportedRuntime version="v1.1.4322"/>
    <supportedRuntime version="v2.0.50727"/>
    <supportedRuntime version="v4.0"/>
  </startup>
</configuration>

And it does! :) With that TextAnalysisTool.NET.exe.config file in place, TextAnalysisTool.NET runs on a clean install of the Windows 8 Developer Preview as-is and without prompting the user to install .NET 3.5. I've updated the download ZIP to include this file so new users will automatically benefit; existing users should drop TextAnalysisTool.NET.exe.config in the right place, and they'll be set as well!

Aside: Although this trick will work in many cases, it isn't guaranteed to work. In particular, if there has been a breaking change in .NET 4, then attempting to run a pre-.NET 4 application in this manner might fail. Therefore, it's prudent to do some verification when trying a change like this!

[Click here to download a ZIP file containing TextAnalysisTool.NET, the relevant .config file, its documentation, and a ReadMe.]

TextAnalysisTool.NET has proven to be extremely popular with support engineers and it's always nice to hear from new users. I hope today's post extends the usefulness of TextAnalysisTool.NET by making the Windows 8 experience as seamless as people have come to expect!

I went public with TextAnalysisTool.NET a few years back - it's a handy tool for interactive log file analysis that's popular with people in many parts of the company. The introductory post has more background and detail, but this snippet from the README gives an overview:

The Problem: For those times when you have to analyze a large amount of textual data, picking out the relevant line(s) of interest can be quite difficult. Standard text editors usually provide a generic "find" function, but the limitations of that simple approach quickly become apparent (e.g., when it is necessary to compare two or more widely separated lines). Some more sophisticated editors do better by allowing you to "bookmark" lines of interest; this can be a big help, but is often not enough.

The Solution: TextAnalysisTool.NET - a program designed from the start to excel at viewing, searching, and navigating large files quickly and efficiently. TextAnalysisTool.NET provides a view of the file that you can easily manipulate (through the use of various filters) to display exactly the information you need - as you need it.

And here's an animated GIF showing TextAnalysisTool.NET working with some MSBuild output:

I don't talk about it in the introductory post, but one of the things TextAnalysisTool.NET supports is a flexible plug-in architecture for handling custom file formats. Here's what the included documentation says:

TextAnalysisTool.NET's support for plug-ins allows users to add in their own code that understands specialized file types. Every time a file is opened, each plug-in is given a chance to take responsibility for parsing that file. When a plug-in takes responsibility for parsing a file, it becomes that plug-in's job to produce a textual representation of the file for display in the usual line display. If no plug-in supports a particular file, then it gets opened using TextAnalysisTool.NET's default parser (which displays the file's contents directly). One example of what a plug-in could do is read a binary file format and produce meaningful textual output from it (e.g., if the file is compressed or encrypted). Another plug-in might add support for the .zip format and display a list of the files within the archive. A particularly ambitious plug-in might translate text files from one language to another. The possibilities are endless!

Over the years, I've been contacted by various people wanting to use the plug-in architecture to add support for specialized file formats. One of those people is Tomer Rotstein who recently released a WPPFormatter plug-in. (If the acronym "WPP" isn't familiar to you, you might start by reading the Windows software trace preprocessor entry on Wikipedia - it's the file format used by the event tracing infrastructure in Windows.) But Tomer has gone above and beyond what I ever had in mind - as you can see from the following screen shot of WPPFormatter's "open file" dialog:

I encourage people to read Tomer's post to get a full sense of what WPPFormatter does. There's a download link at the end of that post, so please try it out if it seems useful!

And for those of you who aren't already TextAnalisisTool.NET users, please:

[Click here to download the latest version of TextAnalysisTool.NET.]

Aside: Tomer's accomplishment is even more notable when you realize that TextAnalysisTool.NET was the first .NET application I ever wrote and that it targets .NET 1.1 - from a time before there were generics, extensibility frameworks, and all the other goodness we've come to take for granted! Truth be told, the plug-in model for TextAnalysisTool.NET is downright wacky in some ways, so I congratulate Tomer on succeeding in spite of my goofy design. :)

A number of years ago, the product team I was on spent a lot of team analyzing large log files. These log files contained thousands of lines of output tracing what the code was doing, what its current state was, and gobs of other diagnostic information. Typically, we were only interested in a handful of lines - but had no idea which ones at first. Often one would start by searching for a generic error message, get some information from that, search for some more specific information, obtain more context, and continue on in that manner until the problem was identified. It was usually the case that interesting lines were spread across the entire file and could only really be understood when viewed together - but gathering them all could be inconvenient. Different people had different tricks and tools to make different aspects of the search more efficient, but nothing really addressed the end-to-end scenario and I decided I'd try to come up with something better.

TextAnalysisTool was first released to coworkers in July of 2000 as a native C++ application written from scratch. It went through a few revisions over the next year and a half and served myself and others well during that time. Later, as the .NET Framework became popular, I decided it would be a good learning exercise to rewrite TextAnalysisTool to run on .NET as a way to learn the Framework and make some architectural improvements to the application. TextAnalysisTool.NET was released in February of 2003 as a fully managed .NET 1.0 C# application with the same functionality of the C++ application it replaced. TextAnalysisTool.NET has gone through a few revisions since then and has slowly made its way across parts of the company. (It's always neat to get an email from someone in a group I had no idea was using TextAnalysisTool.NET!) TextAnalysisTool.NET is popular enough among its users that I started getting requests to make it available outside the company so that customers could use it to help with investigations.

The effort of getting something posted to Microsoft.com seemed overwhelming at the time, so TextAnalysisTool.NET stayed internal until now. With the latest request, I realized my blog would be a great way to help internal groups and customers by making TextAnalysisTool.NET available to the public!

You can download the latest version of TextAnalysisTool.NET by clicking here (or on the image above).

In the above demonstration of identifying the errors and warnings from sample build output, note how the use of regular expression text filters and selective hiding of surrounding content make it easy to zoom in on the interesting parts of the file - and then zoom out to get context.

Additional information can be found in the TextAnalysisTool.NET.txt file that's included in the ZIP download (or from within the application via Help | Documentation). The first section of that file is a tutorial and the second section gives a more detailed overview of TextAnalysisTool.NET (excerpted below). The download also includes a ReadMe.txt with release notes and a few other things worth reading.

The Problem: For those times when you have to analyze a large amount of textual data, picking out the relevant line(s) of interest can be quite difficult. Standard text editors usually provide a generic "find" function, but the limitations of that simple approach quickly become apparent (e.g., when it is necessary to compare two or more widely separated lines). Some more sophisticated editors do better by allowing you to "bookmark" lines of interest; this can be a big help, but is often not enough.

The Solution: TextAnalysisTool.NET - a program designed from the start to excel at viewing, searching, and navigating large files quickly and efficiently. TextAnalysisTool.NET provides a view of the file that you can easily manipulate (through the use of various filters) to display exactly the information you need - as you need it.

Filters: Before displaying the lines of a file, TextAnalysisTool.NET passes the lines of that file through a set of user-defined filters, dimming or hiding all lines that do not satisfy any of the filters. Filters can select only the lines that contain a sub-string, those that have been marked with a particular marker type, or those that match a regular expression. A color can be associated with each filter so lines matching a particular filter stand out and so lines matching different filters can be easily distinguished. In addition to the normal "including" filters that isolate lines of text you DO want to see, there are also "excluding" filters that can be used to suppress lines you do NOT want to see. Excluding filters are configured just like including filters but are processed afterward and remove all matching lines from the set. Excluding filters allow you to easily refine your search even further.

Markers: Markers are another way that TextAnalysisTool.NET makes it easy to navigate a file; you can mark any line with one or more of eight different marker types. Once lines have been marked, you can quickly navigate between similarly marked lines - or add a "marked by" filter to view only those lines.

Find: TextAnalysisTool.NET also provides a flexible "find" function that allows you to search for text anywhere within a file. This text can be a literal string or a regular expression, so it's easy to find a specific line. If you decide to turn a find string into a filter, the history feature of both dialogs makes it easy.

Summary: TextAnalysisTool.NET was written with speed and ease of use in mind throughout. It saves you time by allowing you to save and load filter sets; it lets you import text by opening a file, dragging-and-dropping a file or text from another application, or by pasting text from the clipboard; and it allows you to share the results of your filters by copying lines to the clipboard or by saving the current lines to a file. TextAnalysisTool.NET supports files encoded with ANSI, UTF-8, Unicode, and big-endian Unicode and is designed to handle large files efficiently.

I maintain a TODO list with a variety of user requests, but I thought I'd see what kind of feedback I got from releasing TextAnalysisTool.NET to the public before I decide where to go with the next release. I welcome suggestions - and problem reports - so please share them with me if you've got any!

I hope you find TextAnalysisTool.NET useful as I have!