Design of client/server application

[FreeDelphiPascal 19]

I want to create a client/server application.

The server has multiple binary files in one folder on the server. There can be folders within folders. Basically, I simulate some kind of file system with per-file and per-folder permissions.

I want the client to be able to send a command to the server, to retrieve the file/folder tree and then request a specific file (by ID) for editing. In return, the server will read that binary file from disk and send it to the client as a binary stream.

The client modifies the stream and presses 'Save' which will send the file back to the server, which will store it back on disk.

    TFolder= class
      private
        procedure load(Stream: TBuffStream);
        procedure save(Stream: TBuffStream);
      public
        Users      : TPermissions;    // Users participating in this folder (users that can see the list of files in this folder)
        FolderType : TFolderType;
        ParentList : TFolders;
        SubFolders : TFolders;
        Files: TFiles;
    
        class function CreateNew(CONST aName: string; aOwner: TUser; ParentLst: TFolders; aCore: TObject; aFolderType: TFolderType): TFolder; static;
        constructor Create      (CONST aName: string; aOwner: TUser; ParentLst: TFolders; aCore: TObject; aFolderType: TFolderType);
    
        property ID: Integer read FID;
        property Name: string read FName;
    
        function GetFolder: string;
      end;
      
      TFolders= class(TObjectList<TFolder>)  
      

I also need support for file synchronization. I want the server to be able to transmit some kind of flag to indicate to other clients that a certain file is already open (in use) by a client and tell which client (ID) has locked the file. If the file is locked, other clients can still read it but cannot write to it and send it back to the server.
In other words, the first client that requests the file, gets read-write access. The rest of the clients only get read access.

I want to allow the clients that have read-only permissions to send a text message to the client that locked the file to ask for the file's liberation.

The server should create a list of records in which it keeps track of the status of each file.

    TPermissions = class(TDictionary<TUser, TAccPermissionSet>)
      
    TUserLock = Record
      Read: boolean;
      Write: boolean;
      Changed: boolean; // The user changed the file
    end;

    TFileStatus = Record
      FileName: string;
      ByUser: TList<TUserLock>;
    end;

 

Of course, security needs to be considered.
Maybe something like this:

     SSLHandler := TIdServerIOHandlerSSLOpenSSL.Create(TCPServer);

 

 

NTFS-like user permissions

In theory, all the users of a top folder can see down on lower folders. BUT could be also the case that each file/folder has a VERY SPECIFIC list of users that can read/write the documents. So, you could prevent "top users" from seeing lower files. Additionally, to that list of users, I need a "master" user that "owns" the folder and its documents. It is pretty much like the permissions of the NTFS.

 

File-centric

I won't allow users to move files around. The work will be mostly done inside the files (RTF). I am thinking about splitting the document in multiple RTF sections and locking only one section. This way, I could get something like "Google Documents" where multiple users edit the same doc in parallel.

I could give to each folder/document (even document section) a unique ID. The client could use these IDs to ask for a specific document (or document section) from the server.

 

WebDev as solution?

I have yet to see if WebDev allows this kind of user access. I also see a problem with the speed if you have thousands of files and folders. I don't know how much is webdev like ftp, but navigating via FTP through a dense structure of files/folders can be quite slow.

 

Folder tree

Now, I store the information about files and folders in individual binary files. But this could create a lot of I/O activity. It seems more reasonable now to send the file/folder structure as a whole to the clients (but not the file contents, since the files are big).

I am considering sending the whole folder tree to the clients. Fortunately, the users are split to groups (a group is like a "root" but I can have multiple of these groups/roots). I only have to send them the folder structure of a single group.

 

Existing code

Actually, I have the program above, but it is for single-user. No server side whatsoever. So, I want to convert it to multi-user.
I intend to use Indy, but I only have a tiny amount of experience with it.

 

Other requirements

The server should work with multiple clients.

The program should work over the Internet.

The user should be able to work offline (this could be implemented at a later date, but must be implemented at one point).

 

--------------------

If you ever designed such a system, please share your thoughts, before I go in 🙂  

Maybe there is already some template application like this....
 

 

Edited November 13, 2023 by FreeDelphiPascal

[Anders Melander 1802]

It sounds like you are reinventing WebDAV...

 

40 minutes ago, FreeDelphiPascal said:

the first client that requests the file, gets read-write access. The rest of the clients only get read access.

Bad idea, IMO.

Supposedly a client that has acquired read-access will expect some kind of file stability. It would make more sense to make write locks exclusive and have read locks block the acquisition of write locks.

[FreeDelphiPascal 19]

13 hours ago, Anders Melander said:

Bad idea, IMO.

Supposedly a client that has acquired read-access will expect some kind of file stability. It would make more sense to make write locks exclusive and have read locks block the acquisition of write locks.

I thought about that. When the second client opens the document (in read-only mode) it will be informed that the document is being changed by somebody else (big red label on top of the form).

 

The idea is that I don't want to prevent the second client from reading the information in the document. This will block his work/progress.

 

In most cases, the first user will mostly add content at the end of the document, not change the old content.

[FreeDelphiPascal 19]

I don't think I need to implement the full protocol in Delphi/Indy. I don't want/need to be compatible with other WebDav programs.

But I can learn from it!
Thanks!

 

 

 

[Anders Melander 1802]

19 minutes ago, FreeDelphiPascal said:

I don't think I need to implement the full protocol in Delphi/Indy. I don't want/need to be compatible with other WebDav programs.

But I can learn from it!
Thanks!

 

 

 

I don't know why you quoted that specific document without specifying the source:

https://support.netdocuments.com/s/article/360044231332

It's just some CMS service provider notifying its users that it no longer supports WebDAV. WebDAV isn't going anywhere; It's just a protocol standard.

[FreeDelphiPascal 19]

 

Sorry. I took only one snippet, but you can find many

_____

I will look anyway if Indy has support for it. Just informational, in the beginning 🙂

Update: it seems there is: https://stackoverflow.com/questions/8966787/getting-webdav-content-via-delphi-and-indy-10-tidhttp

 

But thank you very much for pointing to it. I never had a clue it exists.

 

Edited November 13, 2023 by FreeDelphiPascal

[Pat Foley 51]

Here's some more Information for your consideration.

 

Quote

The problem with the lock-modify-unlock model is that it's a bit restrictive and often becomes a roadblock for users:

• Locking may cause administrative problems. Sometimes Harry will lock a file and then forget about it. Meanwhile, because Sally is still waiting to edit the file, her hands are tied. And then Harry goes on vacation. Now Sally has to get an administrator to release Harry's lock. The situation ends up causing a lot of unnecessary delay and wasted time...

https://svnbook.red-bean.com/en/1.7/svn-book.pdf  

 

Surely only copies of your documents are being looked at and double checked before changes are recorded into the system.  

[FreeDelphiPascal 19]

22 minutes ago, Pat Foley said:

The problem with the lock-modify-unlock model is that it's a bit restrictive and often becomes a roadblock for users:

• Locking may cause administrative problems. Sometimes Harry will lock a file and then forget about it. Meanwhile, because Sally is still waiting to edit the file, her hands are tied. And then Harry goes on vacation. Now Sally has to get an administrator to release Harry's lock. The situation ends up causing a lot of unnecessary delay and wasted time...

Thanks, Pat.

Exactly. I was considering exactly that! Imagine a user goes 2 weeks in holidays and he/she forgets to unlock a file.

Or a user needs to work but the information it needs is in one of those locked documents, locked for 1-2 hours, or god knows how much....

I am not criticizing WebDav, I just think my requirements are a bit more demanding....

__

@Anders. I think WebDev is cool, but it might not fulfill my very specific requirements.
In my app, in theory, all the users of a top folder can see down on lowerfolders. BUT could be also the case that each file/folder has a VERY SPECIFIC list of users that can read/write the documents. So, you could prevent "top users" from seeing lower files. Additionally, to that list of users, I need a "master" user that "owns" the folder and its documents. It is pretty much like the permissions of the NTFS.

 

I have yet to see if WebDev allows this kind of user access.
I also see a problem with the speed if you have thousands of files and folders. I don't know how much is webdev like ftp, but navigating via FTP through a dense structure of files/folders can be quite slow.

I am considering sending the whole folder tree to the clients. Fortunately, the users are split to groups (a group is like a "root" but I can have multiple of these groups/roots). I only have to send them the folder structure of a single group.

 

I don't expect the users (probably I won't even allow it) to move files around.
The work will be mostly done inside the files (RTF). I am thinking about splitting the document in multiple RTF sections and locking only one section. I could get something like Google Documents where multiple users edit the same doc in parallel.

My idea is to give to each folder/document a unique ID. The client could use these IDs to ask for a specific document (or document section) from the server.

 

 

__

 

Considering how little I know about Indy, I am afraid to start this project 😞 I could easily f*ck up the architecture 🙂
So, please keep sending your considerations.

The application is already implemented, but for single user. I expect to be 20x more complicated to make it multi-user.

 

Edited November 13, 2023 by FreeDelphiPascal

[Anders Melander 1802]

1 hour ago, FreeDelphiPascal said:

Sorry. I took only one snippet, but you can find many

Sigh. Did you actually read the posts you highlighted? None of them are relevant to you.

 

1 hour ago, FreeDelphiPascal said:

I will look anyway if Indy has support for it.

You do not need Indy to support it. You just need to be able to handle HTTP requests on the server and send requests from the client. Your server implementation will be completely custom to your application (like every other WebDAV server (which is also why you won't find a standard WebDAV "component" on any platform)) and since you will not be using a standard WebDAV client application, your client implementation might as well also be completely custom. If there's an Indu client component that can do what you want then sure, use that.

WebDAV is a very simple protocol and would be easy to implement using standard HTTP components.

 

51 minutes ago, FreeDelphiPascal said:

I have yet to see if WebDev allows this kind of user access.

There's nothing special about those requirements. I would think most WebDAV implementations have similar requirements.

User authentication is handled in another layer (pretty standard http layering). The WebDAV layer gets user info from that layer and decides what the user is allowed to access and do.

 

1 hour ago, FreeDelphiPascal said:

I also see a problem with the speed if you have thousands of files and folders. I don't know how much is webdev like ftp, but navigating via FTP through a dense structure of files/folders can be quite slow.

I am considering sending the whole folder tree to the clients.

What would a user do with a list of "thousands of files and folders"?

This sounds a bit like one of my current clients who as a requirement specified that "the grid must be able to display and scroll through a million rows without any lag"... I mean I could limit the grid to 10,000 rows and nobody would ever notice.

 

Regardless, I don't see why sending a list of "thousands of files and folders" would become a performance problem. Regardless of what solution you go for, it's the same information that is going to be sent over the wire.

 

Anyway, it sounds like you are rubber-ducking a bit here and it would probably be smarter to first figure out what it actually is you want to do before asking about how to do it.

[FreeDelphiPascal 19]

> None of them are relevant to you.

 

It is the "pulse". WebDAV seems to fall out of grace? I don't know...
But I think it is smart to take the "pulse" of a technology before you use it. 

 

 

 

> What would a user do with a list of "thousands of files and folders"?

 

Better think about size upfront, than later. I don't want to say to my users, "sorry you cannot create more than 100 files because I haven't thought about that, some years ago".
And I can actually see some users gathering thousands of files per year - could be 10-100 users per group.

To be on the safe side, and looking 10 years AT LEAST into the future, the program should easily handle 50000 items (files/folders), per group, per year. Again, this would be the extreme case... but better be prepared...

 

Anyway, in my single-user version, all the information is in a tree view. But considering how much data is to transfer, I should change to a Windows Explorer view. Thanks for the insight!

 

 

 

>  it sounds like you are rubber-ducking a bit here

 

In a way yes, in a way no:

Yes: I know what I want, I want to see how I do it. But, yeah, I could change my requirements slightly if I find that an existing technology (like WebDAV) could easily offer what I need, EVEN if there might be some minor drawback. I might slightly change my requirements based on the technology used and I might choose the technology based on the requirements. It is a bit circular.

No: As I said, the single user version exists. And my requirements are pretty clear (and described in the original post). Don't you think?

I know, for example, that I need files and folder. How I transmit this info to the user could dramatically change based on the technology I use. Duck or not, I am flexible as a rubber 🙂  That's good, right? 🙂

________

 

Anyway, I haven't dismissed the WebDav yet.
If it does not fully lock the document, then it is still viable to me!

Update: There seems to be a 50MB limit per file in DAV (ha ha that should have been a huuuuge number for 1996). But it seems it can be changed.

 

 

 

Edited November 13, 2023 by FreeDelphiPascal

[Fr0sT.Brutal 900]

WebDAV is just a protocol. You'll have both sides of your channel custom so you can extend it for your needs.

[Kas Ob. 121]

On 11/12/2023 at 9:48 PM, FreeDelphiPascal said:

NTFS-like user permissions

In theory, all the users of a top folder can see down on lower folders. BUT could be also the case that each file/folder has a VERY SPECIFIC list of users that can read/write the documents. So, you could prevent "top users" from seeing lower files. Additionally, to that list of users, I need a "master" user that "owns" the folder and its documents. It is pretty much like the permissions of the NTFS.

Mostly irrelevant to your case, either an approaches, library, protocol, component ... has such functionality out of the box or not, but you are developer and can add such mechanism.

On 11/12/2023 at 9:48 PM, FreeDelphiPascal said:

File-centric

I won't allow users to move files around. The work will be mostly done inside the files (RTF). I am thinking about splitting the document in multiple RTF sections and locking only one section. This way, I could get something like "Google Documents" where multiple users edit the same doc in parallel.

I could give to each folder/document (even document section) a unique ID. The client could use these IDs to ask for a specific document (or document section) from the server.

Like above, wither the ..... (above mentioned) have such mechanism or you have to implement it, it is way easier to choose the one with already have these functionality, but yet implementing these are not rocket science, because even when the library provide authorization and locking you still as developer have to put that part and almost every single one will have an event like to adjust or override.

On 11/12/2023 at 9:48 PM, FreeDelphiPascal said:

WebDev as solution?

I have yet to see if WebDev allows this kind of user access. I also see a problem with the speed if you have thousands of files and folders. I don't know how much is webdev like ftp, but navigating via FTP through a dense structure of files/folders can be quite slow.

In my opinion yes WebDav is the best out of the box, i have my own WebDav, i built it with SecureBlackBox it is using RTC as TCP socket server, working brilliantly, on my Android i use different software (From FDroid and Google Play like https://play.google.com/store/apps/details?id=com.lonelycatgames.Xplore this explorer i am using since Nokia 80 8GB and many other apps that support WebDav) to perform browsing and backup my files etc, because WebDav is a standard.

One more important thing, stay away from FTP and FTPS, they are pain in the back and not faster, just slower.

On other hand SFTP is also viable option for you, SFTP is SSH based, does support locking, but there is a and extension for WbDav https://datatracker.ietf.org/doc/html/rfc3253 this one in particular is adding full file versioning, which is just what you need, i don't have in my expired SSB, also don't know where to find one for you, also my server is checking for User Agent if it is not specific one then it is WebDav and if it is specific one it will be forwarded to RTC Server (socketless) here i added RTC full blown HTTP server with support to Portal for remote controlling the whole PC while i have HTTP server for different.... well these things and modules in RTC are hard to explain but in short you can have many services and functionality without adding a single line of code just add the unit and you will have chat messenger or all RTC demos.

 

Also here is few SFTP libraries for Delphi that can help you start, do you know of this https://www.nsoftware.com/sftpdrive/ and its price, it is simply what doctors recommend too, mount an remote SFTP directory as a drive on Windows !

 

 

Another solution is just convert your single user application to multi user using RTC, which is also brilliant, and will take you few hours or may be days to get know the library, but once you did, you will be able to implement anything with few lines, its remote procedures are most powerful you ever imagine, while threading is talking care of and you don't need to waste a second on it, so i suggest to get know it, also there is RTC Portal which does have remove file sharing, and will really serve you well as how to do things in very fast manner.

 

As Anders explained, and i repeated, many of these fine tuned functionality is up to you and you have to put these, like custom locking or ... all the locks can be timeout, all connection can be limited to speed or time, Kas is taking hell of time writing post then disconnect him for now, let others write to the thread, when he is done, he will post, if the server see a conflict then rename the file/thread in way like being forked then notify the users or someone to solve the conflict, Lars will come and start yelling, deleting ,threatening to ban, oh boy .. Lars will do Lars.

 

Anyway, hope that helps.

[Anders Melander 1802]

On 11/13/2023 at 4:54 PM, FreeDelphiPascal said:

Update: There seems to be a 50MB limit per file in DAV (ha ha that should have been a huuuuge number for 1996). But it seems it can be changed.

Why do you say that?

 

My guess is that you've been googling and once again have found some information that only applies to a specific product's implementation of WebDAV and once again haven't bothered to try to understand the context of the information.

 

Maybe read the WebDAV specification instead.

[FreeDelphiPascal 19]

I had some time to investigate more about this, and it looks like WebSockets provides what I need.