Organize Files Monthly by Last Modified Date

I recently had a situation where I had a single folder with 50,000+ files in it (uploaded motion detection JPEG images from a surveillance camera), and I needed to split them up into folders by month to make it manageable. I wanted to look at the Last Modified Date of the file, and send it into its respective folder (i.e. 2009-02, 2009-03, etc).

Out of that came this simple little utility that does just that. I’ve ended up using it for a few situations so thought I’d go ahead and share it. This is written in .NET, so the .NET framework is required.

Download the utility

Instructions (also in instructions.txt):

This utility will take files in a folder, say, C:\Temp, and move files based on their Last Modified date into “monthly” folders.

For example, if two files exist in C:\Temp, file1.txt, and file2.txt, and the first was last modified in January 2009, and the second last modified in April 2009, this utility will move them into two respective folders:

2009-01
2009-04

By default, it will run in read-only mode, and not actually do anything with the files unless the -noReadOnly parameter is used.

Usage: filefolderdateorganizer “C:\Temp” -noReadOnly

Random Contact Data Generation in C# / VB.NET

I wanted to generate random contact data (you know, first names, last names, city/state/zip and phone numbers) in .NET, and I found a few little snippets, but nothing really comprehensive.

I wrote these little methods to do this to save me (and maybe you) some time in the future. It’s obviously very easy to add more data to them depending on your needs. It generates things like this (just a few examples):

Michael Jones
461 Second Dr
Los Angeles, AK 12422
124-851-3783

James Smith
532 Sixth St
Columbus, LA 44714
446-576-5437

Richard Harris
261 Elm Pkwy
Las Vegas, OH 77006
769-302-7091

Patricia Thomas
68218 Maple Pkwy
Boston, NE 62280
944-388-4109

Elizabeth Martinez
332 Ninth Blvd
Sacramento, VT 79846
797-749-8472

Daniel Martin
65363 Elm Blvd
Tucson, CT 22140
221-474-1127

Robert Williams
403 First Dr
Milwaukee, MN 54432
543-200-2781

Joseph Taylor
133 Seventh St
Jacksonville, IA 39706
396-561-7144

David Miller
5542 Eighth Pkwy
Portland, NC 71998
719-286-8798

Linda White
844 Second Dr
Los Angeles, AZ 14291
142-911-1453

In C#… (see VB.NET down further)

        private Random GetRandomizer()
        {
            System.Threading.Thread.Sleep(1);
            return new Random(DateTime.Now.Millisecond);
        }

        private string GenerateName()
        {
            return GenerateFirstName() + " " + GenerateLastName();
        }

        private string GenerateFirstName()
        {
            string[] fNames = { "James", "John", "Robert", "Michael", "Mary", "William", "David", "Richard", "Charles", "Joseph",
                    "Thomas", "Patricia", "Christopher", "Linda", "Barbara", "Daniel", "Paul", "Mark", "Elizabeth", "Jennifer"};

            return GetRandomValueFromStringArray(fNames);
        }

        private string GenerateLastName()
        {
            string[] lNames = { "Smith", "Johnson", "Williams", "Jones", "Brown", "Davis", "Miller", "Wilson", "Moore", "Taylor",
                    "Anderson", "Thomas", "Jackson", "White", "Harris", "Martin", "Thompson", "Garcia", "Martinez", "Robinson"};

            return GetRandomValueFromStringArray(lNames);
        }

        private string GenerateStreetNumber()
        {
            Random r = GetRandomizer();

            StringBuilder sB = new StringBuilder();

            for (int i = 0; i <= r.Next(2, 5); i++)
            {
                if (i == 0)
                {
                    // don't use a 0
                    sB.Append(r.Next(1, 9).ToString());
                }
                else
                {
                    sB.Append(r.Next(0, 9));
                }
            }

            return sB.ToString();
        }

        private string GeneratePhoneNumber()
        {
            Random r = GetRandomizer();
            int areaCode = r.Next(100, 999);

            if (areaCode.ToString().EndsWith("11"))
            {
                areaCode = areaCode + 1;
            }

            return areaCode.ToString() + "-" + r.Next(100, 999).ToString() + "-" + r.Next(1000, 9999).ToString();
        }

        private string GenerateStreetName()
        {
            string[] popularStreetNames = { "Second", "Third", "First", "Fourth", "Park", "Fifth", "Main", "Sixth", "Oak", "Seventh",
                            "Pine", "Maple", "Cedar", "Eighth", "Elm", "View", "Washington", "Ninth", "Wall" };
            string[] streetSuffixes = { "Dr", "St", "Pkwy", "Blvd" };

            return GetRandomValueFromStringArray(popularStreetNames) + " " + GetRandomValueFromStringArray(streetSuffixes);
        }

        private string GenerateAptOrSuite()
        {
            Random r = GetRandomizer();

            string[] prefixes = { "Apt", "Ste", "Bldg" };
            string[] suffixes = { "A", "B", "C", "D" };
            int numericalVal = r.Next(1, 300);

            return GetRandomValueFromStringArray(prefixes) + " " + numericalVal.ToString() + GetRandomValueFromStringArray(suffixes);
        }

        private string GenerateCity()
        {
            string[] cities = { "New York", "Los Angeles", "Chicago", "Houston", "Philadelphia", "Phoenix", "San Diego", "San Antonio", "Dallas", "Detroit",
                         "San Jose", "Indianapolis", "Jacksonville", "San Francisco", "Columbus", "Austin", "Memphis", "Baltimore", "Milwaukee", "Fort Worth",
                         "Charlotte", "El Paso", "Boston", "Seattle", "Washington", "Denver", "Portland", "Oklahoma", "Las Vegas", "Tucson",
                         "Long Beach", "Albuquerque", "New Orleans", "Cleveland", "Fresno", "Sacramento", "Mesa", "Atlanta" };

            return GetRandomValueFromStringArray(cities);
        }

        private string GenerateState(bool abbreviationOnly)
        {
            string[] states = { "Alabama", "Alaska", "Arizona", "Arkansas", "California", "Colorado", "Connecticut", "Delaware", "District of Columbia", "Florida",
                         "Georgia", "Hawaii", "Idaho", "Illinois", "Indiana", "Iowa", "Kansas", "Kentucky", "Louisiana", "Maine",
                         "Maryland", "Massachusetts", "Michigan", "Minnesota", "Mississippi", "Missouri", "Montana", "Nebraska", "Nevada", "New Hampshire",
                         "New Jersey", "New Mexico", "New York", "North Carolina", "North Dakota", "Ohio", "Oklahoma", "Oregon", "Pennsylvania", "Rhode Island",
                         "South Carolina", "South Dakota", "Tennessee", "Texas", "Utah", "Vermont", "Virginia", "Washington", "West Virginia", "Wisconsin",
                         "Wyoming" };
            string[] stateAbbr = { "AL", "AK", "AZ", "AR", "CA", "CO", "CT", "DE", "DC", "FL",
                         "GA", "HI", "ID", "IL", "IN", "IA", "KS", "KT", "LA", "MN",
                         "MD", "MA", "MI", "MN", "MS", "MO", "MT", "NE", "NV", "NH",
                         "NJ", "NM", "NY", "NC", "ND", "OH", "OK", "OR", "PA", "RI",
                         "SC", "SD", "TN", "TX", "UT", "VT", "VA", "WA" };

            if (abbreviationOnly)
            {
                return GetRandomValueFromStringArray(stateAbbr);
            }
            else
            {
                return GetRandomValueFromStringArray(states);
            }
        }

        private string GenerateZipCode()
        {
            Random r = GetRandomizer();

            return r.Next(10000, 99999).ToString();
        }

        private string GetRandomValueFromStringArray(string[] sArray)
        {
            Random r = GetRandomizer();

            return sArray[r.Next(sArray.Length)];
        }

How To: Move File Shares to new location on a Windows Server

This is fairly Network 101, but I created it for our internal knowledge base anyway, and figured it might be useful.

Follow the following procedure to move a shared folder from one drive to another.

Note: The hard part about this is migrating permissions with the files. Using normal “copy” utilities to copy or move the data without the correct options can result in permissions not being applied correctly at the destination.

Part 1 – Copy the Data
Do not move the data, as if something occurs in a “move” operation, half the file structure can be located at Point A and the other half coule be located at Point B, leaving you a mess to clean up. Better to do a successful Copy then delete the source after a few days once you’re confident everything has been transferred successfully.

1. Copy the directory / permissions structure using XCOPY.Go to a command prompt, and issue the following command (tailor to your needs):

   xcopy “C:\Source Folder” “D:\Destination Folder” /O /E /F /Y /T

   The /T command ensures that you are only copying the folder structure and not actual files within the folders. For more XCOPY commands, use xcopy /?

2. Using SyncToy (google for “SyncToy” and download the utility), synchronize the source and destination folders together.

Part 2 – Move the Shares

1. Remove the current Share names.Go to Start > Run… and type compmgmt.msc

   Under the Shared Folders node, expand it to show “Shares“. Right click on the Shares node and click the Export List… menu option.

   IMPORTANT: If there are specific permissions settings on each Share entry, make a note of them.

   Once you’ve saved the list of shares to a file, right click the shares that were previously associated with the source folder you copied, and click Stop Sharing. This will remove the share from the list.

2. Recreate the exact share names again, with the destination folders.Open Windows Explorer (right click Start and go to Explore) and navigate to your newly copied folders. Refer to the exported list you created in Step 1 to re-create the shares with their exact names.
3. Set the correct Share permissions (not normal Windows Security) on the “Sharing” tab of the folder properties. This is normally “Full Control” for “Authenticated Users”.

If you have moved the shares correctly, using the same names, clients mapped to drives using the \\server\ShareName will not notice any interruption or change, and no change will need to be made on the client side.

MOSS Content Type Propagation – Part 2 – The Code

(This is a follow-up to the previous post: Propagating Content Type Changes in MOSS 2007)

So far, I have propagated a few Content Type fields in a live production environment of ~65,000 webs using this method. Mind you, this does not mean that I have propagated a content type to 65,000 lists or items, just that the environment is large.

The whole game here revolves around Field Links (see the SPFieldLink class on MSDN). And to recap, the goal here is to be able to maintain your Content Types in in a Solution (WSP) feature. MOSS does not support updating content types “declaratively via XML” (i.e. through a solution package).

Here’s what MOSS will do: In most cases, MOSS will add new fields to the root / site collection content type (make sure you do an RDAD deployment – Retract / Delete / Add / Deploy). What this actually means (and this is important) – is that MOSS will add the field and create a SPFieldLink which links the field to the root content type. This is why when you create a new list item based on the content type, that it will show up, but there are no field links automatically created for existing items.

So here’s the work we need to do to propagate the field:

1. If working with a large site collection (more than 1,000 webs), determine the usage count of the content type you’re updating. This will give you an approximate idea of how long this is going to take to propagate (more on this below).
2. Break the SPFieldLink at the root level, so that the field is no longer linked to the content type, and update the content type.
3. Add the SPFieldLink to the content type, but this time, call .Update(true) on the content type, so that SharePoint will propagate the changes to all inherited items (including lists items).

Let’s step through these…

1. Determine Usage Count – This determines where the content type is actually used, and therefore, the # of updates that will take place. A content type usage is defined as the # of items that actually use / inherit the content type.

This is simple – if you already have a SPWeb object called “web” and the Content Type name string called “contentTypeName”, just use:

int contentTypeUsages = Microsoft.SharePoint.SPContentTypeUsage.GetUsages(web.ContentTypes[contentTypeName]).Count;

In testing so far, here are my performance findings as to how long it will take to propagate. Bear in mind that for smaller usage counts there is still some overhead time in establishing a context, retrieving web, etc. However, the results are not very linear, and I couldn’t tell you why. This will just give you a very general and vague idea to set your expectations.

• 1 usage – 0.66 seconds per web (Total Time: 28 seconds)
• 6 usages – 0.01 seconds per web (Total Time: 19 seconds)
• 1,876 usages – 0.64 seconds per web (Total Time: 1192 seconds / 19 min, 52 sec)
• 13,138 usages – 0.53 seconds per web (Total Time: 1003 seconds / 16 min, 43 sec)
• 15,031 usages – 0.66 seconds per web (Total Time: 1234 seconds / 20 min, 34 sec)

2. Break the SPFieldLink at the root level – If you already see the field when you go to Site Collection Settings > Content Types, then you will need to break the SPFieldLink. Find the field link by looking in the SPContentType.FieldLinks collection (in the below example, this is the selectedFieldLink object, and selectedSiteColl is the SPSite site collection object).

SPFieldCollection fieldColl = selectedSiteColl.RootWeb.Fields;

this.selectedContentType.FieldLinks.Delete(selectedFieldLink.Id);
this.selectedContentType.Update(true);

fieldColl.Delete(selectedFieldLink.Name);
selectedSiteColl.RootWeb.Update();

3. Propagate the field – Example below – In this case I have created a FieldLinkInstance class that just holds some properties like the Site Collection, Content Type name, etc, but you should be able to see the general idea here. The most critical thing is the last line where you call c.Update(true).

public void PropagateFieldToSiteCollection(FieldLinkInstance fieldLink)
{
SPContentTypeCollection contTypes = fieldLink.SiteCollection.RootWeb.ContentTypes;
SPFieldCollection fieldColl = fieldLink.SiteCollection.RootWeb.Fields;
SPContentType c = contTypes[fieldLink.FieldLink.ContentType.ContentTypeName];
SPField newField = new SPField(fieldColl, fieldLink.FieldLink.Field.FieldType, fieldLink.FieldLink.Field.Name);
newField.Title = fieldLink.FieldLink.Field.DisplayName;
newField.StaticName = fieldLink.FieldLink.Field.StaticName;
newField.PushChangesToLists = true;

string strNewColumn = fieldColl.Add(newField);
SPField targetField = fieldColl.GetFieldByInternalName(strNewColumn);

SPFieldLink oFieldLink = new SPFieldLink(targetField);
c.FieldLinks.Add(oFieldLink);

c.Update(true);
}

Hopefully this will give you an idea and a little code to propagate content types using this method. I have created a class library that actually parses WSP solution packages to auto-detect unpropagated content types between a Solution and live MOSS farm and propagate them, but can’t share the source code for that at the moment. In general, you can extract the WSP package as a CAB, then parse the XML files to extract the Content Types.

ASP.NET Gotcha: Service Accounts with SQL Express

This happened a few weeks ago, on a short one-off application that I wrote.

Here’s the situation: ASP.NET web application hosting a web service. I just needed a minimal database, so used a SQL Express database, with the .MDF file contained within the ASP.NET application (connection string .\SQLEXPRESS… yada yada).

On our production domain, we use a service account for Windows Authentication, which does not have interactive logon privileges (i.e. this is not an account we use to log on to the servers).

Gotcha #1 – SQL Express needs to rock some temporary data in C:\Documents and Settings

My first surprise was that SQL Express could not create an instance because it needed to do some magic in the User folder (in this case it would have been C:\Documents and Settings\<service account here>\ etc etc). No problem, I enabled interactive logon for the service account just to get it up and running, logged on once to set up the profile goodness, and off we go.

Gotcha #2 – This is only temporary

To my surprise, I thought after logging on once with the service account once, creating the user profile, et al, we’d be good. No, that’s not how technology works. A few days later it stopped working again. Log back on, and we’re good.

So, when using SQL Server Express with service accounts, be careful and make sure that the user profile path is available!

MOSS Web.Config Modification – Beware the XPath

Today I finally got around to diagnosing an interesting and simple (but critical) problem with web.config settings. Here’s the scenario:

Every time web.config modifications are done, multiple entries get added to the web.config file. In this case, to retrieve data from a web service, the MOSS farm uses a <proxy> entry to specify the outbound proxy server for the application.

However, every time SPWeb.ApplyWebConfigModifications() was called, an additional <proxy> entry is added to the web.config file. You can’t have more than one proxy entry, and therefore causes these components (or the entire requests) to fail because of an invalid web.config.

(For more information about Web.Config Modifications in MOSS – http://sharepointsolutions.blogspot.com/2006/12/using-spwebconfigmodificat_116736917110571614.html)

Well, the MOSS web.config modification entries looked something like this:

Owner: ProxySettings
Name: proxy[@autoDetect='true']
Value: <proxy proxyaddress=”http://someproxy:8080″ />

The problem lies in the fact that MOSS does not do any validation to make sure that the value you’re putting in matches the name’s XML Xpath. So, MOSS says “OK, got a new modification for me? Alright… I’ll put it in. No problem.” and it works fine.

The next time SPWeb.ApplyWebConfigModifications() is called, MOSS does the following:

1. Looks at the existing web.config file, and makes an XML DOM inventory of what is already in it.
2. Enumerates through the Web.Config Modifications (which, by the way, are persistently stored in the SharepointConfig database) that should be applied.
3. Gets to this proxy modification entry, and tries to match the name – proxy[@autoDetect='true'] – which in XML-ese is slang for: “Let me see if I already have a tag (element) that starts with <proxy and has an attribute in it of autoDetect=’true’.

   So, if our modification value looked like this: <proxy autoDetect=’true’ /> then MOSS would have found this entry and not added a new one. The problem is, our value has no autoDetect=’true’ attribute inside it, so every time MOSS tries to match the value, it never will match.

4. Match fails, so MOSS assumes that the entry doesn’t exist yet, and adds another one.
5. MOSS site goes down in flames due to multiple proxy entries, users scream, riots happen, etc.

Really, the setting should have looked like this:

Owner: ProxySettings
Name: proxy
Value: <proxy proxyaddress=”http://someproxy:8080″ />

That way, any element/tag starting with <proxy… would be matched and correctly handled.

Unfortunately, even before or after this is corrected, there is one more important thing: The original never matched, so it’s now orphaned and MOSS will not remove it. Once the web.config modifications are removed from the persisted MOSS object model, the entries are still there in the web.config. The proper way to do this is as follows:

1. Delete the SPWebConfigModification entry in the object model.
2. Apply web config modification settings with SPWeb.ApplyWebConfigModifications().
3. Manually remove all entries (in this case, all entries starting with <proxy…) from the web.config file and save.
4. Add the correct SPWebConfigModification entry and apply modifications again.

Now, the web.config modifications should be matched by MOSS.

Changing Hostname on CentOS

I needed this information earlier for CentOS (which Trixbox uses) and found it on this page: http://rutmandal.info/infotalk/2008/07/10/change-host-name-for-trixbox-shell/

Re-posting here so I don’t forget…

Changing hostname commands –

[trixbox1.localdomain ~]# sysctl kernel.hostname

kernel.hostname = trixbox1.localdomain

[trixbox1.localdomain ~]# sysctl kernel.hostname=veejansh.ivr

kernel.hostname = veejansh.ivr

You also need to edit your network file to make this change permanent i.e. persistent across reboots.

[veejansh.ivr ~]# vi /etc/sysconfig/netowrk

Here, edit “HOSTNAME” to “veejansh.ivr”, and save the file.

Now, restart your shell and you will have new name for your shell prompt or HOSTNAME.

MOSS Farm Monitoring – Everybody online?

Before a deployment, I’m working on a “health check” to determine whether or not some common ‘gotcha’s in a farm are going to be a problem. One problem in particular that I needed to solve is determining whether or not a MOSS Farm Member server is actually online and functioning – specifically its Timer Service.

(Slight detour – Solution Deployment in a farm environment totally relies on Timer Jobs – when you deploy a solution, MOSS creates a Solution Deployment Timer Job that every server in the farm executes. If a server is offline, that solution deployment job will hang indefinitely until the offline server comes back – by design, so that you don’t consider a solution deployment “successful” without all servers receiving the deployment.)

There are obviously the normal “Windows” ways to figure this out – can you ping each member, access a file share, etc. But I actually wanted to know that the member was not only running, but actively communicating with MOSS.

My goal was to come up with, for lack of a better term, a “Farm Ping” — Whereas I would kick off some kind of “dummy” timer job that all farm members would need to run in order to consider the “ping” completed. If I queued this job, and all member servers ran it fine, then I would know everybody was online. If it didn’t finish, then I’d know I have a problem.

The only issue with this technique is that I would need to create some type of custom timer job, package it up in a solution, and deploy it whenever I wanted to test this. But really, that just adds another moving piece to my equation. However, looking around, I found a great alternative that MOSS already does – the Config Refresh Timer Job.

After watching my Timer Job Monitoring Utility for a couple minutes, I noticed a pattern. Every farm member that runs Timer Jobs runs the “Config Refresh” job every 30 seconds or so. If I stop the Timer service on a member server, the Config Refresh job will continue to age. Start the Timer service again, and within a minute or so the Config Refresh job gets run and back on its 30-second schedule.

What’s this mean? Well, we can simply get SPFarm.Local.TimerService.RunningJobs, take a look at the “Config Refresh” jobs in that collection, and if they’re more than 30 seconds old, we know that the server it belongs to is not online. Simple, easy, and surprisingly accurate.

SQL Server Connectivity when Configuration Manager is broken

An awesome admin told me about a cool trick today… Needed to add a SQL Alias to SQL 2005 via SQL Server Configuration Manager. However, upon starting SQL Server Configuration Manager, received the following error:

Cannot connect to WMI provider. You do not have permission or the server is unreachable. Note that you can only manage SQL Server 2005 servers with SQL Server Configuration Manager.

Invalid Class [0x80041010]

Well, removing and reinstalling the SQL components was not a good option in this case. The solution?

Run c:\windows\system32\cliconfg.exe

This is a SQL Server Client Network Utility that allows you to access some of the basic SQL Server Configuration options.

Sometimes the ancient way just works better

For an application in .NET Compact Framework, I need to load about 10MB worth of data stored in XML files into datasets. On a desktop, it takes approximately no time to load that data into a dataset, or write it back out to a XML file.

I thought to myself, “since this worked in less than 1 second on a PC, I should be able to do this on a mobile device in maybe 5-10 seconds” – acceptable for an application startup time. Right? Wrong. Try over 4 minutes. Nothing fancy, just a DataSet.ReadXml() and then iterating through and validating data.

Right now, you’re thinking to yourself (because I’m writing this I get to choose), “why not use SQL CE or something?” – because the sole purpose of this data is to get enough data loaded into memory to get a head start on synchronizing it with the server, and SQL CE is not worth getting into as a dependency for this task. It’s supposed to be as simple of a process as possible.

“Oh, that makes sense,” you think. Thank you.

Knowing that 4 minutes for an applicaiton startup was a little bit excessive, I spent a few minutes turning it into a tab-delimited file instead.

The results? All the data loaded in 20 seconds. It’s not fancy, it’s not flexible, but it works. And that’s what I care about at the end of the day.