In order to create the file management software for our website, we need to consider topics such as a single or multi-file upload, file system management, and image manipulation in the case of photos. In addition to this we will cover creation of pages for displaying the user’s photo albums, their friends’ photo albums, as well as a few data management pages.
What’s the problem?
Apart from the standard infrastructure issues that we have to consider when building a system such as this, one of the core issues in any web-based file management system is file upload. As we all know, most server side technologies allow only one file to be uploaded at a time and ASP.NET is no different. And while we could easily buy a third-party plug-in to handle multiple files at once, we decided to provide for options to upload the files either via Silverlight or via Flash.
Once we get our file upload process working we are only one-third of the way there! As we are going to be mostly concerned with uploading images, we need to consider that we will need to provide some image manipulation. With each file that is uploaded to our system we need to create a handful of different sizes of each image to be used in various scenarios across our site. To start with, we will create thumbnail, small, medium, large, and original size photos.
Now while creating different size files is technically working with the file storage system, we wanted to take an extra breath with regards to the file storage concepts. We can choose to store the files on the file system or in a database. For avatars it made sense to store each with the profile data whereas for image galleries it makes more sense to store the file on the file system. While storing files to the file system we need to be very cautious as to how the file structure is defined and where and how the individual files are stored. In our case we will use system-generated GUIDs as our file names with extensions to define the different sizes that we are storing. We will dig into this more as we start to understand the details of this system.
Once we have uploaded the files to the server and they are ready for our use across the system, we will take up the concept of user files versus system files. If we build the system with some forethought regarding this topic we can have a very generic file management system that can be extended for future use. We will build a personal system in this article. But as you will see with some flags in just the right places, we could just as easily build a system file manager or a group file manager.
Design
Let’s take a look at the design for this feature.
Files
For our site, as we are not storing our files in the database, we need to take a closer look at what actually needs to be managed in the database so as to keep track of what is going on in the file system. In addition to standard file metadata, we need to keep a close eye on where the file actually lives—specifically which file system folder (directory on the hard drive) the file will reside in. We also need to be able to maintain which accounts own which files, or in the case of system files, which files can be viewed by anyone.
Folders
You may be wondering why we have a separate section regarding folders when we just touched upon the fact that we will be managing which file system folder we will be storing files in. In this section we are going to discuss folder management from a site perspective rather than a file system perspective—user folders or virtual folders if you desire.
Very similar to file storage, we will be storing various metadata about each folder. We will also have to keep track of who owns which folder, who can see which folder, or in the case of system folders whether everyone can see that folder. And of course as each folder is a virtual container for a file, we will have to maintain the relationship between folders and files.
File upload
The file upload process will be handled by a Silverlight/Flash client. While this is not really an article about either Silverlight or Flash, we will show you how simple it is to create this Flash client, that is really just providing a way to store many files that need to be uploaded, and then uploading them one at a time in a way that the server can handle each file. For the Silverlight option, we are using code from Codeplex—http://silverlightfileupld.codeplex.com/.
File system management
Managing the file system may seem like a non-issue to begin with. However, keep in mind that for a community site to be successful we will need at least 10,000 or so unique users. Given that sharing photos and other files is such a popular feature of most of today’s community sites, this could easily translate into a lot of uploaded files.
While you could technically store a large number of files in one directory on your web server, you will find that over time your application becomes more and more sluggish. You might also run into files being uploaded with the same name using this approach. Also, you may find that you will have storage issues and need to split off some of your files to another disk or another server.
Many of these issues are easily handled if we think about and address them up front. In our case we will use a unique file name for each uploaded file. We will store each file in subdirectories that are also uniquely named based on the year and month in which the file was uploaded. If you find that you have a high volume of files being uploaded each day, you may want to store your files in a folder with the year and month in the name of the folder and then in another subdirectory for each day of that month.
In addition to a good naming convention on the file system, we will store the root directory for each file in the database. Initially you may only have one root for your photos, one for videos, and so on. But storing it now will allow you to have multiple roots for your file storage—one root location per file. This gives you a lot of extensibility points over time meaning that you could easily relocate entire sections of your file gallery to a separate disk or even a separate server.
Data management screens
Once we have all of the infrastructure in place we will need to discuss all the data management screens that will be needed—everything from the UI for uploading files to the screens for managing file metadata, to screens for creating new albums. Then we will need to tie into the rest of the framework and allow users to view their friends’ uploaded file albums.
The solution
Let’s take a look at our solution.
Implementing the database
First let’s take a look at the tables required for these features (see the following screenshot).
Files
The most important thing to consider storing while in the database is of course our primary interest files. As with most other conversations regarding a physical binary file we always have to consider if we want to store the file in the database or on the file system. In this case we think it makes sense to store the file (and in the case of a photo, its various generated sizes) on the file system. This means that we will only be storing metadata about each file in our database.
The most important field here to discuss is the FileSystemName. As you can see this is a GUID value. We will be renaming uploaded files to GUIDs in addition to the original extension. This allows us to ensure that all the files in any given folder are uniquely named. This removes the need for us to have to worry about overwriting other files.
Then we see the FileSystemFolderID. This is a reference to the FileSystemFolders table, that lets us know the root folder location where the file is stored.
Next on our list of items to discuss is the IsPublicResource flag. By its name it is quite clear that this flag will set a file as public or private and can therefore be seen by all or by its owner (AccountID).
We then come to a field that may be somewhat confusing: DefaultFolderID. This has nothing to do with the file system folders. This is a user created folder. When files are uploaded initially they are put in a virtual folder. That initial virtual folder becomes the file’s permanent home. This doesn’t mean that it is the file’s only home. As you will see later we have the concept that files can live in many virtual folders by way of subscription to the other folders.
File system folders
As mentioned previously, the FileSystemFolders table is responsible for letting us know where our file’s root directory is. This allows us to expand our system down the road to have multiple roots, that could live on the same server but different disks, or on totally different servers. The fields in the table are Key, Path (URL), and a Timestamp.
File types
The FileTypes table will help us to keep track of what sort of files we are storing and working with. This is a simple lookup table that tells us the extension of a given file.
Folders
Folders are virtual in this case. They provide us with a way to specify a container of files. In our case we will be containing photos, in which case folders will act as photo albums. The only field worth explaining here is the flag IsPublicResource, which allows us to specify whether a folder and its resources are public or private, that is, viewable by all or viewable only by the owner.
Folder types
The FolderTypes table allows us a way to specify the type of folder. Currently this will simply be Name, photos, movies, and so on. However, down the road you may want to specify an icon for each folder type in which case this is the place where you would want to assign that specification.
Account folders
In the AccountFolders table we are able to specify additional ownership of a folder. So in the case that a folder is a public resource and external resources can own folders, we simply create the new ownership relationship here. This is not permanent ownership. It is still specified with the Folders table’s AccountID. This is a temporary ownership across many Accounts.
As you can see in the previous screenshot we have the owner (AccountID) and the folder that is to be owned (FolderID).
Account files
Similar to the AccountFolders table, the AccountFiles table allows someone to subscribe to a specific file. This could be used for the purposes of Favorites or similar concepts. The makeup of this table is identical to AccountFolders. You have the owner and the file being owned.
Folder files
The FolderFiles table allows an Account to not only subscribe to a file, similar to the Favorites concept, but it also allows a user to take one of my files and put it into one of their folders as though the file itself belonged to them.
As you can see in the previous screenshot this is primarily a table that holds the keys to the other tables. We have the FolderID, FileID, and AccountID for each file. This clearly specifies who is taking ownership of what and where they want it to be placed.
Creating the relationships
Once all the tables are created we can then create all the relationships.
For this set of tables we have relationships between the following tables:
- Files and FileSystemFolders
- Files and FileTypes
- Files and Folders
- Files and Accounts
- Folders and Accounts
- Folders and FolderTypes
- AccountFolders and Accounts
- AccountFolders and Folders
- AccountFiles and Accounts
- AccountFiles and Files
- FolderFiles and Accounts
- FolderFiles and Folders
- FolderFiles and Files
