12 min read

(For more resources related to this topic, see here.)

Files are the building blocks of any computer system. This article deals with portable handling of read-only application resources, and provides recipes to store the application data.

Let us briefly consider the problems covered in this article. The first one is the access to application data files. Often, application data for desktop operating systems resides in the same folder as the executable file. With Android, things get a little more complicated. The application files are packaged in the .apk file, and we simply cannot use the standard fopen()-like functions, or the std::ifstream and std::ofstream classes.

The second problem results from the different rules for the filenames and paths. Windows and Linux-based systems use different path separator characters, and provide different low-level file access APIs.

The third problem comes from the fact that file I/O operations can easily become the slowest part in the whole application. User experience can become problematic if interaction lags are involved. To avoid delays, we should perform the I/O on a separate thread and handle the results of the Read() operation on yet another thread. To implement this, we have all the tools required.

We start with abstract I/O interfaces, implement a portable .zip archives handling approach, and proceed to asynchronous resources loading.

Abstracting file streams

File I/O APIs differ slightly between Windows and Android (POSIX) operating systems, and we have to hide these differences behind a consistent set of C++ interfaces.

Getting ready

Please make sure you are familiar with the UNIX concept of the file and memory mapping. Wikipedia may be a good start (http://en.wikipedia.org/wiki/Memory-mapped_file).

How to do it…

  1. From now on, our programs will read input data using the following simple interface. The base class iObject is used to add an intrusive reference counter to instances of this class:

    class iIStream: public iObject { public: virtual void Seek( const uint64 Position ) = 0; virtual uint64 Read( void* Buf, const uint64 Size ) = 0; virtual bool Eof() const = 0; virtual uint64 GetSize() const = 0; virtual uint64 GetPos() const = 0;

    The following are a few methods that take advantage of memory-mapped files:

    virtual const ubyte* MapStream() const = 0; virtual const ubyte* MapStreamFromCurrentPos() const = 0; };

    This interface supports both memory-mapped access using the MapStream() and MapStreamFromCurrentPos() member functions, and sequential access with the BlockRead() and Seek() methods.

  2. To write some data to the storage, we use an output stream interface, as follows (again, the base class iObject is used to add a reference counter):

    class iOStream: public iObject { public: virtual void Seek( const uint64 Position ) = 0; virtual uint64 GetFilePos() const = 0; virtual uint64 Write( const void* B, const uint64 Size ) = 0; };

  3. The Seek(), GetFileSize(), GetFilePos(), and filename-related methods of the iIStream interface can be implemented in a single class called FileMapper:

    class FileMapper: public iIStream { public: explicit FileMapper( clPtr<iRawFile> File ); virtual ~FileMapper(); virtual std::string GetVirtualFileName() const{
    return FFile->GetVirtualFileName(); } virtual std::string GetFileName() const{ return FFile->GetFileName(); }

  4. Read a continuous block of data from this stream and return the number of bytes actually read:

    virtual uint64 BlockRead( void* Buf, const uint64 Size ) { uint64 RealSize =( Size > GetBytesLeft() ) ? GetBytesLeft() : Size;

  5. Return zero if we have already read everything:

    if ( RealSize < 0 ) { return 0; } memcpy( Buf, ( FFile->GetFileData() + FPosition ),static_cast<size_t>
    ( RealSize ) );

  6. Advance the current position and return the number of copied bytes:

    FPosition += RealSize; return RealSize; } virtual void Seek( const uint64 Position ) { FPosition = Position; } virtual uint64 GetFileSize() const { return FFile->GetFileSize(); } virtual uint64 GetFilePos() const { return FPosition; } virtual bool Eof() const { return ( FPosition >= GetFileSize() ); } virtual const ubyte* MapStream() const { return FFile->GetFileData(); } virtual const ubyte* MapStreamFromCurrentPos() const { return ( FFile->GetFileData() + FPosition ); } private: clPtr<iRawFile> FFile; uint64 FPosition; };

  7. The FileMapper uses the following iRawFile interface to abstract the data access:

    class iRawFile: public iObject { public: iRawFile() {}; virtual ~iRawFile() {}; void SetVirtualFileName( const std::string& VFName );void
    SetFileName( const std::string& FName );std::string GetVirtualFileName() const; std::string GetFileName(); virtual const ubyte* GetFileData() const = 0; virtual uint64 GetFileSize() const = 0; protected: std::string FFileName; std::string FVirtualFileName; };

Along with the trivial GetFileName() and SetFileName() methods implemented here, in the following recipes we implement the GetFileData() and GetFileSize() methods.

How it works…

The iIStream::BlockRead() method is useful when handling non-seekable streams. For the fastest access possible, we use memory-mapped files implemented in the following recipe. The MapStream() and MapStreamFromCurrentPos() methods are there to provide access to memory-mapped files in a convenient way. These methods return a pointer to the memory where your file, or a part of it, is mapped to. The iOStream::Write() method works similar to the standard ofstream::write() function. Refer to the project 1_AbstractStreams for the full source code of this and the following recipe.

There’s more…

The important problem while programming for multiple platforms, in our case for Windows and Linux-based Android, is the conversion of filenames.

We define the following PATH_SEPARATOR constant, using OS-specific macros, to determine the path separator character in the following way:

#if defined( _WIN32 ) const char PATH_SEPARATOR = '\'; #else const char PATH_SEPARATOR = '/'; #endif

The following simple function helps us to make sure we use valid filenames for our operating system:

inline std::string Arch_FixFileName(const std::string& VName) { std::string s( VName ); std::replace( s.begin(), s.end(), '\', PATH_SEPARATOR ); std::replace( s.begin(), s.end(), '/', PATH_SEPARATOR ); return s; }

See also

  • Implementing portable memory-mapped files
  • Working with in-memory files

Implementing portable memory-mapped files

Modern operating systems provide a powerful mechanism called the memory-mapped files. In short, it allows us to map the contents of the file into the application address space. In practice, this means we can treat files as usual arrays and access them using C pointers.

Getting ready

To understand the implementation of the interfaces from the previous recipe we recommend to read about memory mapping. The overview of this mechanism implementation in Windows can be found on the MSDN page at http://msdn.microsoft.com/en-us/library/ms810613.aspx.

To find out more about memory mapping, the reader may refer to the mmap() function documentation.

How to do it…

  1. In Windows, memory-mapped files are created using the CreateFileMapping() and MapViewOfFile() API calls. Android uses the mmap() function, which works pretty much the same way. Here we declare the RawFile class implementing the iRawFile interface.

    RawFile holds a pointer to a memory-mapped file and its size:

    ubyte* FFileData; uint64 FSize;

  2. For the Windows version, we use two handles for the file and memory-mapping object, and for the Android, we use only the file handle:

    #ifdef _WIN32 HANDLE FMapFile; HANDLE FMapHandle; #else int FFileHandle; #endif

  3. We use the following function to open the file and create the memory mapping:

    bool RawFile::Open( const string& FileName,const string& VirtualFileName ) {

  4. At first, we need to obtain a valid file descriptor associated with the file:

    #ifdef OS_WINDOWS FMapFile = (void*)CreateFileA( FFileName.c_str(),GENERIC_READ,
    FILE_SHARE_READ,NULL, OPEN_EXISTING,FILE_ATTRIBUTE_NORMAL |
    FILE_FLAG_RANDOM_ACCESS,NULL ); #else FFileHandle = open( FileName.c_str(), O_RDONLY ); if ( FFileHandle == -1 ) { FFileData = NULL; FSize = 0; } #endif

  5. Using the file descriptor, we can create a file mapping. Here we omit error checks for the sake of clarity. However, the example in the supplementary materials contains more error checks:

    #ifdef OS_WINDOWS FMapHandle = (void*)CreateFileMapping( ( HANDLE )FMapFile,NULL, PAGE_READONLY,
    0, 0, NULL ); FFileData = (Lubyte*)MapViewOfFile((HANDLE)FMapHandle,FILE_MAP_READ, 0,0,0 ); DWORD dwSizeLow = 0, dwSizeHigh = 0; dwSizeLow = ::GetFileSize( FMapFile, &dwSizeHigh ); FSize = ((uint64)dwSizeHigh << 32) | (uint64)dwSizeLow; #else struct stat FileInfo; fstat( FFileHandle, &FileInfo ); FSize = static_cast<uint64>( FileInfo.st_size ); FFileData = (Lubyte*) mmap(NULL, FSize, PROT_READ,MAP_PRIVATE, FFileHandle, 0); close( FFileHandle ); #endif return true; }

  6. The correct deinitialization function closes all the handles:

    bool RawFile::Close() { #ifdef OS_WINDOWS if ( FFileData ) UnmapViewOfFile( FFileData ); if ( FMapHandle ) CloseHandle( (HANDLE)FMapHandle ); CloseHandle( (HANDLE)FMapFile ); #else if ( FFileData ) munmap( (void*)FFileData, FSize ); #endif return true; }

  7. The main functions of the iRawFile interface, GetFileData and GetFileSize, have trivial implementation here:

    virtual const ubyte* GetFileData() { return FFileData; } virtual uint64 GetFileSize() { return FSize; }

How it works…

To use the RawFile class we create an instance and wrap it into a FileMapper class instance:

clPtr<RawFile> F = new RawFile(); F->Open("SomeFileName"); clPtr<FileMapper> FM = new FileMapper(F);

The FM object can be used with any function supporting the iIStream interface. The hierarchy of all our iRawFile implementations looks like what is shown in the following figure:

Implementing file writers

Quite frequently, our application might want to store some of its data on the disk. Another typical use case we have already encountered is the downloading of some file from the network into a memory buffer. Here, we implement two variations of the iOStream interface for the ordinary and in-memory files.

How to do it…

  1. Let us derive the FileWriter class from the iOStream interface. We add the Open() and Close() member functions on top of the iOStream interface and carefully implement the Write() operation. Our output stream implementation does not use memory-mapped files and uses ordinary file descriptors, as shown in the following code:

    class FileWriter: public iOStream { public: FileWriter(): FPosition( 0 ) {} virtual ~FileWriter() { Close(); } bool Open( const std::string& FileName ) { FFileName = FileName;

  2. We split Android and Windows-specific code paths using defines:

    #ifdef _WIN32 FMapFile = CreateFile( FFileName.c_str(),GENERIC_WRITE, FILE_SHARE_READ,
    NULL, CREATE_ALWAYS,FILE_ATTRIBUTE_NORMAL, NULL ); return !( FMapFile == ( void* )INVALID_HANDLE_VALUE ); #else FMapFile = open( FFileName.c_str(), O_WRONLY|O_CREAT ); FPosition = 0; return !( FMapFile == -1 ); #endif }

  3. The same technique is used in the other methods. The difference between both OS systems is is trivial, so we decided to keep everything inside a single class and separate the code using defines:

    void Close() { #ifdef _WIN32 CloseHandle( FMapFile ); #else if ( FMapFile != -1 ) { close( FMapFile ); } #endif } virtual std::string GetFileName() const { return FFileName; } virtual uint64 GetFilePos() const { return FPosition; } virtual void Seek( const uint64 Position ) { #ifdef _WIN32 SetFilePointerEx( FMapFile,*reinterpret_cast<const LARGE_INTEGER*>(
    &Position ),NULL, FILE_BEGIN ); #else if ( FMapFile != -1 ) { lseek( FMapFile, Position, SEEK_SET ); } #endif FPosition = Position; }

    However, things may get more complex if you decide to support more operating systems. It can be a good refactoring exercise.

    virtual uint64 Write( const void* Buf, const uint64 Size ) { #ifdef _WIN32 DWORD written; WriteFile( FMapFile, Buf, DWORD( Size ),&written, NULL ); #else if ( FMapFile != -1 ) { write( FMapFile, Buf, Size ); } #endif FPosition += Size; return Size; } private: std::string FFileName; #ifdef _WIN32 HANDLE FMapFile; #else int FMapFile; #endif uint64 FPosition; };

How it works…

Now we can also present an implementation of the iOStream that stores everything in a memory block. To store arbitrary data in a memory block, we declare the Blob class, as shown in the following code:

class Blob: public iObject { public: Blob(); virtual ~Blob();

Set the blob data pointer to some external memory block:

void SetExternalData( void* Ptr, size_t Sz );

Direct access to data inside this blob:

void* GetData(); …

Get the current size of the blob:

size_t GetSize() const;

Check if this blob is responsible for managing the dynamic memory it uses:

bool OwnsData() const; …

Increase the size of the blob and add more data to it. This method is very useful in a network downloader:

bool AppendBytes( void* Data, size_t Size ); … };

There are lots of other methods in this class. You can find the full source code in the Blob.h file. We use this Blob class, and declare the MemFileWriter class, which implements our iOStream interface, in the following way:

class MemFileWriter: public iOStream { public: MemFileWriter(clPtr<Blob> Container);

Change the absolute position inside a file, where new data will be written to:

virtual void Seek( const uint64 Position ) { if ( Position > FContainer->GetSize() ) {

Check if we are allowed to resize the blob:

if ( Position > FMaxSize - 1 ) { return; }

And try to resize it:

if ( !FContainer->SafeResize(static_cast<size_t>( Position ) + 1 )) { return; } } FPosition = Position; }

Write data to the current position of this file:

virtual uint64 Write( const void* Buf, const uint64 Size ) { uint64 ThisPos = FPosition;

Ensure there is enough space:

Seek( ThisPos + Size ); if ( FPosition + Size > FMaxSize ) { return 0; } void* DestPtr = ( void* )( &( ( ( ubyte* )(FContainer->GetData() )
)[ThisPos] ) );

Write the actual data:

memcpy( DestPtr, Buf, static_cast<size_t>( Size ) ); return Size; } } private: … };

We omit the trivial implementations of GetFileName(), GetFilePos(), GetMaxSize(), SetContainer(), GetContainer(), GetMaxSize(), and SetMaxSize() member functions, along with fields declarations. You will find the full source code of them in the code bundle of the book.

See also

  • Working with in-memory files

Working with in-memory files

Sometimes it is very convenient to be able to treat some arbitrary in-memory runtime generated data as if it were in a file. As an example, let’s consider using a JPEG image downloaded from a photo hosting, as an OpenGL texture. We do not need to save it into the internal storage, as it is a waste of CPU time. We also do not want to write separate code for loading images from memory. Since we have our abstract iIStream and iRawFile interfaces, we just implement the latter to support memory blocks as the data source.

Getting ready

In the previous recipes, we already used the Blob class, which is a simple wrapper around a void* buffer.

How to do it…

  1. Our iRawFile interface consists of two methods: GetFileData() and GetFileSize(). We just delegate these calls to an instance of Blob:

    class ManagedMemRawFile: public iRawFile { public: ManagedMemRawFile(): FBlob( NULL ) {} virtual const ubyte* GetFileData() const { return ( const ubyte* )FBlob->GetData(); } virtual uint64 GetFileSize() const { return FBlob->GetSize(); } void SetBlob( const clPtr<Blob>& Ptr ) { FBlob = Ptr; } private: clPtr<Blob> FBlob; };

  2. Sometimes it is useful to avoid the overhead of using a Blob object, and for such cases we provide another class, MemRawFile, that holds a raw pointer to a memory block and optionally takes care of the memory allocation:

    class MemRawFile: public iRawFile { public: virtual const ubyte* GetFileData() const { return (const ubyte*) FBuffer; } virtual uint64 GetFileSize() const { return FBufferSize; } void CreateFromString( const std::string& InString ); void CreateFromBuffer( const void* Buf, uint64 Size ); void CreateFromManagedBuffer( const void* Buf, uint64 Size ); private: bool FOwnsBuffer; const void* FBuffer; uint64 FBufferSize; };

How it works…

We use the MemRawFile as an adapter for the memory block extracted from a .zip file and ManagedMemRawFile as the container for data downloaded from photo sites.

LEAVE A REPLY

Please enter your comment!
Please enter your name here