The blog of dlaa.me

Computing the size of your boat [Sample code to help analyze storage space requirements]

Yesterday I mentioned a quick C# program I wrote to help analyze storage space requirements. There was some interest in how that program worked, so I'm posting the complete source code for anyone to use.

using System;
using System.Collections.Generic;
using System.IO;

class SizeOfFilesCreatedOnDate
{
    
private const string outputFileName = "SizeOfFilesCreatedOnDate.csv";

    
static void Main(string[] args)
    {
        
// Create a dictionary to hold the date/size pairs (sorted for subsequent output)
        SortedDictionary<DateTime, long> sizeOfFilesCreatedOnDate = new SortedDictionary<DateTime, long>();

        
// Tally the contents of each specified directory
        // * If no command-line argument was given, default to the current directory
        if (0 == args.Length)
        {
            args =
new string[] { Environment.CurrentDirectory };
        }
        
foreach (string directory in args)
        {
            AddDirectoryContents(directory,
ref sizeOfFilesCreatedOnDate);
        }

        
// Output all date/size pairs to a CSV file in the current directory
        using (StreamWriter writer = File.CreateText(outputFileName))
        {
            writer.WriteLine(
"Date,Size,Cumulative");
            
long cumulative = 0;
            
foreach (DateTime date in sizeOfFilesCreatedOnDate.Keys)
            {
                
long size = sizeOfFilesCreatedOnDate[date];
                cumulative += size;
                writer.WriteLine(
"{0},{1},{2}", date.ToShortDateString(), size, cumulative);
            }
        }
        
Console.WriteLine("Output: {0}", outputFileName);
    }

    
private static void AddDirectoryContents(string directory, ref SortedDictionary<DateTime, long> sizeOfFilesCreatedOnDate)
    {
        
// Display status
        Console.WriteLine("Scanning: {0}", directory);

        
// Tally each child file in the parent directory
        foreach (string file in Directory.GetFiles(directory))
        {
            
// Get a FileInfo for the file
            FileInfo fileInfo = new FileInfo(file);

            
// Get the creation time of the file
            // * If last write < creation, then the file was moved at least once; use the earlier date
            // * The difference between local/UTC (~hours) is unimportant at this scale (~years); use local
            DateTime date = fileInfo.CreationTime.Date;
            
if (fileInfo.LastWriteTime.Date < date)
            {
                date = fileInfo.LastWriteTime.Date;
            }

            
// Update the relevant date/size pair
            long size;
            
if (!sizeOfFilesCreatedOnDate.TryGetValue(date, out size))
            {
                size = 0;
            }
            sizeOfFilesCreatedOnDate[date] = size + fileInfo.Length;
        }

        
// Recursively tally each child directory in the parent directory
        foreach (string childDirectory in Directory.GetDirectories(directory))
        {
            AddDirectoryContents(childDirectory,
ref sizeOfFilesCreatedOnDate);
        }
    }
}

Notes:

  • I wrote this code for a simple one-time purpose, so there's no fancy/friendly user interface.
  • There's also no error-checking. In particular, if it bumps into a directory/file that it doesn't have permission to access, the resulting UnauthorizedAccessException will bubble up and terminate the process. (While this is unlikely to occur when using the program for its intended purpose of examining your data files, it is pretty likely to occur if playing around and pointing it at C:\.)
  • Other than adding comments and support for specifying multiple directories on the command-line, this is the same code I used to generate my chart.
  • The code for handling last write time being earlier than creation time was something I discovered a need for experimentally when I noticed that considering only creation time reported that none of my files were older than a couple of years. Apparently when I moved stuff around a couple of years ago, the copy to my current drive preserved the file's last write time, but reset its creation time (perhaps because of the FAT->NTFS transition).

Enjoy!

Tags: Technical

"You're gonna need a bigger boat." [A brief look at data storage requirements in today's world]

I've previously blogged about my data storage/backup strategy. Briefly, I've got one big drive in my home server that stores all the data my family cares about: mostly music, pictures, and videos (with a little bit of other stuff for good measure). To protect the data, I've got another equally big external drive that I connect occasionally and use for backups by simply mirroring the content of the internal drive.

As things stand today, the internal drive is 320GB and the external drive is 300GB, but I've hit the wall and am almost out of space to add new files. Looking at hard drive prices these days, the sweet spot (measured in $/GB) seems to be with 500GB drives at about $140 (PATA or SATA). Any smaller than that and the delta from 300GB isn't enough to be interesting - any larger than that and the cost really goes up.

I was already prepared to buy a new drive every year or so to allow for growth, so I was curious if getting a 500GB drive now would do the trick. I wrote a quick program to look at every file I backup and tally up the size according to the date the file was created. The C# program walks the whole directory tree, sums the sizes by date, and writes out a simple CSV file with the results. The idea here is to chart the rate at which I'm adding data in order to predict when I'd run out of space next. (Yes, it's easy to come up with more sophisticated heuristics, but this is really just a back-of-the-envelope calculation and doesn't need to be perfect to be meaningful.)

Last night I opened the CSV file in Excel and charted the data. The resulting chart looks like this:

Data Storage Space (GB)

The blue line represents the cumulative size of the data I had at each point in time (horizontal axis) measured in GB (vertical axis) - you can see that I'm just above 300GB today. The red line is Excel's exponential trend line for the same data - it matches the blue line almost perfectly, so it seems pretty safe to say that my data storage needs are increasing exponentially. I was kind of afraid of that, because it means the 500GB drives I've been considering are likely to fill up within the next 8 months!

Clearly, I need to be prepared to spend more on hard drives than I'd initially planned to - or else I'm going to need to significantly change how I do things. I've got some ideas I'm still considering, but charting this data was a good wake-up call that drive capacity isn't increasing as rapidly as I might like. :)

I think that data storage and backup are issues that will affect all of us pretty soon (if they're not already). Backing up to DVDs doesn't scale well once you need more than 10 or so DVDs, and backing up over the network just doesn't seem practical when you're talking about numbers this large. Even ignoring the need to backup, simply storing all the data you have is rapidly becoming an issue. With downloadable HD movie/TV content becoming popular, high megapixel still/video cameras being commonplace, and fast Internet connections becoming the norm, it seems to me that content is outpacing storage right now.

Here's hoping for a quantum leap in storage technology!

Updated on 2007/03/14: I've just posted the source code for the program I wrote to gather this data.

Tags: Technical

Lost (Sk)in Translation [AJAX Control Toolkit update!]

A short while ago we made available the 10301 release of the AJAX Control Toolkit. With this release, we managed to add some great core functionality, a couple of new controls, and a bunch of bug fixes for popular issues (as identified by our user community in the support forum and online issue tracker).

The 10301 release includes two new controls:

  • ListSearch (by contributor Damian Mehers) - A handy enhancement to ListBox/DropDownList controls that allows the user to select items by typing the first few characters
  • SlideShow (by our own Kirti) - A visually pleasing slide show of automatically sized images and captions with optional automatic looping or manual control

We also managed to add three pieces of core functionality that users have been asking for:

  • Localization support - The Toolkit now has an infrastructure for localizing the text in JavaScript files using the standard ASP.NET/AJAX support for resource files. There are placeholder files for the following 14 languages: Arabic, German, English, Spanish, French, Hebrew, Hindi, Italian, Japanese, Korean, Portuguese, Russian, Chinese (Simplified), and Chinese (Traditional). Complete localization of all strings hasn't been done yet, but our most popular request by far is for the string "Today:" at the bottom of the Calendar's popup - and that has already been localized to all of the above languages. Just set your system/browser's language settings to one of the supported languages and visit the Calendar sample page for a demonstration.
  • JavaScript comment/whitespace stripping - People are always interested in keeping the download size of their pages as low as possible, and the Toolkit goes the extra mile to help. When compiled in "Release" mode (as the official releases on CodePlex always are), all of the JavaScript files in the Toolkit automatically have all comments and unnecessary whitespace removed. Ted, the author of this functionality, was careful to adhere to the ECMA-262 ECMAScript Language Specification, so the modifications are both safe and effective.
  • ASP.NET Theme/Skin support - Central control is usually best, so it's nice to be able to use ASP.NET's theme/skin support to specify properties for controls across an entire web site in one central location. Toolkit controls didn't used to be skinnable (i.e., support the SkinID property) but now they are, so go ahead and start skinning your site. In fact, we used this new ability to style the CollapsiblePanels that we use for descriptions/properties/known issues on every sample page.

And, with the help of our contributors, we fixed a bunch of bugs along the way...

We hope you like the new release!!

Recall that you can sample any of the controls right now (no install required). You can also browse the project web site, download the latest Toolkit, and start creating your own controls and/or contributing to the project!

If you have any feedback, please share it with us on the support forum (or email me)!

Toolkit talk, take two [Spoke at the TechReady4 conference]

Earlier today I presented the DEV304: ASP.NET AJAX Control Toolkit Unleashed: Creating Rich Client-Side Controls and Components session at Microsoft's TechReady4 conference here in Seattle. TechReady is a Microsoft-only conference that non-Redmond-based employees attend to get an opportunity to meet with product team members (typically Redmond-based) and catch up on all the new technology each year. Most of the session content is Microsoft-only, but because the presentation I gave today had no private information, I'm posting the (sanitized) slide deck and demos here for everyone to use (note: the attachment is at the bottom of this post).

If you saw the content for my ASP.NET Connections talk a few months ago, the slide content for today's TechReady4 talk was very similar. The first demo was identical and the second (new) demo was just a quick overview of the sample web site that comes with the Toolkit. However, the third demo was completely new and demonstrated how to encapsulate existing script into a new Toolkit control. For demonstration purposes, I took a simple web page with a WPF/E (February CTP) control (the use of which requires two <script>/src tags and a <script>/code block) and showed how to wrap that all into a single "one-liner" Toolkit component which can be more easily used/maintained/added to a page. Those of you who are paying close attention will realize that I took my inspiration from a similar effort by my manager, Shawn Burke - though I wrote all my demo code from scratch so I'd be more familiar with it. :) The final result is a pretty handy way of handling WPF/E controls and folks are welcome to use it however they want!

I hope those of you who attended today enjoyed the talk and learned more about the Toolkit - it was great to have an opportunity to meet with you!

PS - We're always looking for more contributors and additional control ideas, so please let me know if you want to contribute!

[TechReady4-DEV304-AjaxControlToolkit.zip]

Safely avoiding the "access denied" dialog [How to: Work around the access denied cross-domain IFRAME issue in the AJAX Control Toolkit]

Bertrand recently blogged "How to work around the access denied cross-domain frame issue in ASP.NET Ajax 1.0". Unfortunately, a similar issue exists in the AJAX Control Toolkit - with a similar workaround that's detailed in this post.

Background: The AJAX Control Toolkit attempts to use as much of the ASP.NET AJAX infrastructure as possible in order to keep things simple and benefit from the established architecture of ASP.NET AJAX. However, the ASP.NET AJAX 1.0 implementation of Sys.UI.getLocation was not accurate for IFRAME content with certain page layouts in IE6 - one of which was used by the Toolkit test harness. We couldn't have our automated tests failing - and the ASP.NET AJAX team wasn't able to include a fix in their 1.0 release - so we needed to address this somehow. Since we had developed our own getLocation implementation that was accurate in IE6, we decided to route all Toolkit calls to getLocation through our own wrapper function - which used our improved implementation for IE6 and called through to the ASP.NET AJAX implementation for everything else. This works pretty well - except that our version suffers from the same cross-domain permissions issue that the ASP.NET AJAX version suffers from (please refer to Bertrand's post for more details).

Fix: We've just checked in a fix to the Toolkit's Development branch to add the same try/catch wrapper that Bertrand suggests; this fix will be included with the next Toolkit release (in about a month). In the meantime, people who want the fix sooner are encouraged to apply the changes themselves. To do so, open AjaxControlToolkit.sln in Visual Studio, open the file AjaxControlToolkit\Common\Common.js, find the getLocation function, add the yellow, italic block below, and build the project to get a new AjaxControlToolkit.dll with the fix. The complete function definition with the fix applied is included here to make this as easy as possible:

getLocation : function(element) {
    
/// <summary>Gets the coordinates of a DOM element.</summary>
    /// <param name="element" domElement="true"/>
    /// <returns type="Sys.UI.Point">
    ///   A Point object with two fields, x and y, which contain the pixel coordinates of the element.
    /// </returns>

    // workaround for an issue in getLocation where it will compute the location of the document element.
    // this will return an offset if scrolled.
    //
    if (element === document.documentElement) {
        
return new Sys.UI.Point(0,0);
    }

    
// Workaround for IE6 bug in getLocation (also required patching getBounds - remove that fix when this is removed)
    if (Sys.Browser.agent == Sys.Browser.InternetExplorer && Sys.Browser.version < 7) {
        
if (element.window === element || element.nodeType === 9 || !element.getClientRects || !element.getBoundingClientRect) return new Sys.UI.Point(0,0);

        
// Get the first bounding rectangle in screen coordinates
        var screenRects = element.getClientRects();
        
if (!screenRects || !screenRects.length) {
            
return new Sys.UI.Point(0,0);
        }
        
var first = screenRects[0];

        
// Delta between client coords and screen coords
        var dLeft = 0;
        
var dTop = 0;

        var inFrame = false;
        
try {
            inFrame = element.ownerDocument.parentWindow.frameElement;
        }
catch(ex) {
            
// If accessing the frameElement fails, a frame is probably in a different
            // domain than its parent - and we still want to do the calculation below
            inFrame = true;
        }

        
// If we're in a frame, get client coordinates too so we can compute the delta
        if (inFrame) {
            // Get the bounding rectangle in client coords
            var clientRect = element.getBoundingClientRect();
            
if (!clientRect) {
                
return new Sys.UI.Point(0,0);
            }

            
// Find the minima in screen coords
            var minLeft = first.left;
            
var minTop = first.top;
            
for (var i = 1; i < screenRects.length; i++) {
                
var r = screenRects[i];
                
if (r.left < minLeft) {
                    minLeft = r.left;
                }
                
if (r.top < minTop) {
                    minTop = r.top;
                }
            }

            
// Compute the delta between screen and client coords
            dLeft = minLeft - clientRect.left;
            dTop = minTop - clientRect.top;
        }

        
// Subtract 2px, the border of the viewport (It can be changed in IE6 by applying a border style to the HTML element,
        // but this is not supported by ASP.NET AJAX, and it cannot be changed in IE7.), and also subtract the delta between
        // screen coords and client coords
        var ownerDocument = element.document.documentElement;
        
return new Sys.UI.Point(first.left - 2 - dLeft + ownerDocument.scrollLeft, first.top - 2 - dTop + ownerDocument.scrollTop);
    }

    
return Sys.UI.DomElement.getLocation(element);
},

Sorry for the trouble - we hope this patch helps any people who might be running into this issue!

Updated at 6pm: Modified the getLocation implementation to return the correct value in all cases.

A quick update to address some top customer issues [AJAX Control Toolkit update!]

Just a few moments ago we made available the 10201 release of the AJAX Control Toolkit to address some of the most significant issues reported since the 10123 release last week. We chose these issues based on feedback from our users in our support forum and online issue tracker and tried to prioritize issues with widespread benefits while minimizing the risks of breaking anything.

In particular, we:

  • Fixed problems that occurred when using the new AutoComplete or Tabs controls inside an UpdatePanel
  • Improved the localization behavior of the Calendar so that it works better in other countries/time zones
  • Simplified and improved the implementation of the new "draggable popup" feature of ModalPopup
  • Addressed a TextBoxWatermark focus issue that was inconveniencing users

Recall that you can sample any of the controls right now (no install required). You can also browse the project web site, download the latest Toolkit, and start creating your own controls and/or contributing to the project!

If you have any feedback, please share it with us on the support forum (or email me)!

ASP.NET AJAX 1.0 is on the loose! [AJAX Control Toolkit update!]

Earlier today we made available the 10123 release of the AJAX Control Toolkit. We did this in parallel with the official release of ASP.NET AJAX v1.0 and added a bunch of good stuff along the way.

As I note in the release teaser, "This release of the AJAX Control Toolkit includes four new controls (AutoComplete (formerly in the Futures CTP), Calendar, MaskedEdit, and Tabs), fixes for over 75 issues reported by the community, and new functionality for ModalPopup, RoundedCorners, and the ExtenderBase framework." Additionally, we spent time making sure that all 32 Toolkit controls work well on the latest version of the Opera browser. Accepting that every browser has its little quirks, we now offer the same level of "pretty much everything works" support for Internet Explorer 6, Internet Explorer 7, Firefox 1.5, Firefox 2.0, Safari 2.0.4, and Opera 9.10. Those are the top six browsers on the Internet, so there's an very good chance that visitors to a Toolkit-enabled site will be running a supported browser.

Of note, three of the four new controls for this release were submitted by contributors! Ron Buckton wrote both Calendar and Tabs while Fernando Cerqueira wrote MaskedEdit. These are some very cool new controls and we're fortunate to have such great contributors helping out with the Toolkit! As a teaser, I created two little "movies" to show off Calendar and MaskedEdit in action - as well as our Opera support. :)

Calendar Demonstration

MaskedEdit Demonstration

Recall that you can sample any of the controls right now (no install required). You can also browse the project web site, download the latest Toolkit, and start creating your own controls and/or contributing to the project!

If you have any feedback, please share it with us on the support forum (or email me)!

Runnin' on the RC [AJAX Control Toolkit update!]

A few minutes ago we made available the 61214 release of the AJAX Control Toolkit. We did this in parallel with the release of the ASP.NET AJAX v1.0 RC in order to accommodate some breaking changes introduced in that release. Though we did not make any functional changes to the Toolkit in this release, we did take the opportunity to fix a handful of bugs that we'd found.

Recall that you can sample any of the controls right now (no install required). Then you can browse the project web site, download the latest Toolkit, and start creating your own controls and/or contributing to the project!

If you have any feedback, please share it with us on the support forum (or email me)!

Dynamic content made easy *remix* [How to: Use the new dynamic population support for Toolkit controls]

I previously blogged an update to my sample demonstrating how to use the AJAX Control Toolkit's dynamic population functionality with a data-bound ModalPopup. Someone asked privately what the C# page method would look like in VB, so I figured I'd post a small follow-up to show the straightforward translation to VB.

Here's the original C# code for reference:

<script runat="server">
[System.Web.Services.WebMethod]
[Microsoft.Web.Script.Services.
ScriptMethod]
public static string GetContent(string contextKey)
{
    
// Create a random color
    string color = (new Random()).Next(0xffffff).ToString("x6");
    
// Use the style specified by the page author
    string style = contextKey;
    
// Display the current time
    string time = DateTime.Now.ToLongTimeString();
    
// Compose the content to return
    return "<span style='color:#" + color + "; " + style + "'>" + time + "</span> ";
}
</script>

Translating that to VB is largely a matter of changing the syntactic elements from their C# versions to their VB versions. So "[" becomes "<", "}" becomes "End Function", and so on... Two things that tripped me up because I don't regularly use VB were the translation of the hexadecimal prefix "0x" to "&H" and the translation of "static" to "Shared" - but a quick look at the online documentation for VB sorted me out on both counts. After a short while, I ended up with this:

<script runat="server">
    <System.Web.Services.WebMethod()> _
    <Microsoft.Web.Script.Services.ScriptMethod()> _
    
Public Shared Function GetContent(ByVal contextKey As String) As String
        ' Create a random color
        Dim color As String = (New Random()).Next(&HFFFFFF).ToString("x6")
        
' Use the style specified by the page author
        Dim style As String = contextKey
        
' Display the current time
        Dim time As String = DateTime.Now.ToLongTimeString()
        
' Compose the content to return
        Return "<span style='color:#" + color + "; " + style + "'>" + time + "</span> "
    End Function
</
script>

I completed the translation by changing the sample page's Page/Language property from "C#" to "VB", ran the sample page, and verified that it worked just like it did before. Of course! :)

Another quick, small-scale release [AJAX Control Toolkit update!]

Just a short while ago we made available the 61121 release of the AJAX Control Toolkit. I'd barely finished posting the release bits to CodePlex and was still in the process of verifying them when I saw that we'd already had our first eight customer downloads. Wow, those people must really want the latest stuff! :)

The 61121 release was all about addressing some of the biggest pain points for our customers. Of particular note were two workarounds for ASP.NET AJAX Beta 2 issues:

  • The first addresses an issue where the Toolkit had a hard dependency on Microsoft.Web.Extensions.Design.dll, a file that isn't installed by ASP.NET AJAX Beta 2 on machines that don't have some flavor of Visual Studio already installed. This dependency can cause obvious issues for web servers since they usually do not have development tools like Visual Studio installed.
  • The second addresses an issue where ASP.NET AJAX Beta 2 doesn't render the ScriptManager/extenders out to the page during an async postback if EnableEventValidation=false - meaning that the relevant extenders simply don't get recreated after an UpdatePanel postback. We reported this issue to the ASP.NET AJAX team prior to Beta 2, but there wasn't time for them to fix it in that release (don't worry, they'll be fixing it in their next release!). We'd implemented a workaround for this issue in our earlier 61106 release, but it applied only to CascadingDropDown (which is the only control that requires EnableEventValidation=false). Subsequent customer feedback prompted us to move that workaround up a level for this release so that it now applies to the entire Toolkit.

In addition to those two "big ticket" items, we also fixed a handful of other issues, most of which we chose based on customer feedback and work item voting on CodePlex. (Voting is a fairly recent addition - have a look at how it focuses attention on issues that are important to the community!). The other fixes in the 61121 release address the most popular issues in our CodePlex queue and include fixes for ModalPopup server-side show/hide, ClientState within an UpdatePanel, Accordion pane size updates, multiple DragDrop panels, CommonToolkitScripts, and web.config updates to get things working seamlessly with IIS7.

This being small-scale release, we didn't have time to include any new controls this time around. :( However, that's an explicit goal for an upcoming release which we hope to make available before the end of the year. We've got some neat stuff in the prototype stage and it'll be great to get some of it polished and released to the world!

Recall that you can sample any of the controls right now (no install required). Then you can browse the project web site, download the latest Toolkit, and start creating your own controls and/or contributing to the project!

If you have any feedback, please share it with us on the support forum (or email me)!