| [Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] |
Good list! See comments below. On Thu, 2003-07-31 at 14:15, Greg Freemyer wrote: > The thread about deadman locks keeps bringing up new race conditions > that ogfs needs to worry about during the recovery process. > > It very much sounds like we need to create a small recovery document > which at the very least tells us what the potential race conditions are. > > Is this already documented somewhere? > > > First Pass at potential problems: > > 1) Single node failure: > > 1.a) Journal Recovery by multiple nodes may occur > ogfs must address > > 1.b) Lock Recovery must occur prior to journal replay > ogfs must address This happens automatically if the ogfs only uses DLM. > > 1.c) Journal Recovery must occur prior to locks held on failed nodes > being granted those same locks > ogfs must address Persistent DLM locks can be used for this. > > 1.d) Failed node holds lock for which no one is waiting. After failure > and lock recovery, but prior to journal recovery, a different node may > request and be granted the lock. > ogfs must address, lock should not be granted until after > journal replay. > > 1.e) Mounting of new nodes should not occur during journal replay > ??? (Mentioned by someone IIRC, I don't understand the problem) I mentioned this. This is how I've seen it done previously. Actually, the mount would abort and allow recovery to complete and then retry the mount. Again, some of this is specific to the FS and the way it uses the DLM. FS recovery might not just be log replay -- it could include log replay, DLM persistent lock recovery, and potentially file record locking recovery. Also mounting could take several steps that required all the mounted nodes to participate -- checking mounting journal is accessible by all nodes is a good one. If a already mounted node died while another node was mounting the mount could not finish, so an abort, retry. > > 1.f) Normal FS activity should be blocked during journal replay > ??? (Mentioned by Jeffrey Orlin, I don't understand the > problem) > This depends on the specific Cluster FS implementation. If an FS had inodes shared a FS block, then blocking activity and flushing and invalidating the block would be required, so that the node doing log replay get valid data and after log replay the other nodes see the new updated data. There might be other things that required activity to be blocked. I'm not sure if this is a problem for ogfs. Of course, if the remaining nodes access a PERSISTENT DLM lock before the replay has finished and cleaned up the persistent locks, they will block. > 2) multiple node failure > > 2.a) Multiple Journal Recoveries should not occur simultaneously. > ??? (Mentioned by Jeffrey Orlin, but I don't understand the > problem) > I mentioned this also. If you use persistent DLM locks, cleaning up the locks would be incorrect if 2 nodes are trying to clean up at the same time. This depends on the how persistent locks are implemented in the DLM and how they are cleaned up. > > > > Once we agree on what the issues are, we should document how ogfs > currently addresses the issue. > > Finally we should decide if deadman locks should be used in the future > to address the above in a more generic way. > > Greg > -- > Greg Freemyer > > Greg ------------------------------------------------------- This SF.Net email sponsored by: Free pre-built ASP.NET sites including Data Reports, E-commerce, Portals, and Forums are available now. Download today and enter to win an XBOX or Visual Studio .NET. http://aspnet.click-url.com/go/psa00100003ave/direct;at.aspnet_072303_01/01 _______________________________________________ Opengfs-devel mailing list Opengfs-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/opengfs-devel