| [Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index] |
fcntl locking seg faults and oopses at program exit, and hangs like
flock if run twice on the same file. The trace back looks similar to one
I got from flock.
fd=open(lfile, O_RDWR|O_CREAT, 0644);
lock.l_type = F_WRLCK;
lock.l_whence = SEEK_SET;
lock.l_start = 0;
lock.l_len = 4;
if(fcntl(fd, F_SETLK, &lock) == -1){
}
>>EIP; f89a7658 <[ogfs]ogfs_plock_act+1a8/570> <=====
>>ecx; c342b268 <_end+30d3518/384b5310>
>>edx; c1000020 <_end+ca82d0/384b5310>
>>esi; e8307980 <_end+27fafc30/384b5310>
>>edi; e2175d40 <_end+21e1dff0/384b5310>
>>ebp; f45b7600 <_end+3425f8b0/384b5310>
>>esp; e8ed1e88 <_end+28b7a138/384b5310>
Trace; c013d78c <__alloc_pages+4c/1a0>
Trace; c0130e03 <do_wp_page+103/300>
Trace; f8985899 <[ogfs]ogfs_lock_file+255/2d4>
Trace; c015ae37 <locks_remove_posix+d7/160>
Trace; c0144647 <filp_close+87/d0>
Trace; c0122fec <put_files_struct+6c/e0>
Trace; c01238fe <do_exit+11e/300>
Trace; c0123b13 <sys_exit+13/20>
Trace; c01093df <system_call+33/38>
Code; f89a7658 <[ogfs]ogfs_plock_act+1a8/570>
00000000 <_EIP>:
Code; f89a7658 <[ogfs]ogfs_plock_act+1a8/570> <=====
0: 8b 43 20 mov 0x20(%ebx),%eax <=====
Code; f89a765b <[ogfs]ogfs_plock_act+1ab/570>
3: 8b 54 24 38 mov 0x38(%esp,1),%edx
Code; f89a765f <[ogfs]ogfs_plock_act+1af/570>
7: 39 42 7c cmp %eax,0x7c(%edx)
Code; f89a7662 <[ogfs]ogfs_plock_act+1b2/570>
a: 89 de mov %ebx,%esi
Code; f89a7664 <[ogfs]ogfs_plock_act+1b4/570>
c: 75 e6 jne fffffff4 <_EIP+0xfffffff4>
Code; f89a7666 <[ogfs]ogfs_plock_act+1b6/570>
e: 0f be 54 24 57 movsbl 0x57(%esp,1),%edx
Code; f89a766b <[ogfs]ogfs_plock_act+1bb/570>
13: 0f 00 00 sldtl (%eax)
flock and fcntl locking both work fine with GFS (ie, not OGFS).
On Thu, 2003-05-08 at 10:31, David B. Zafman wrote:
>
> I'm surprised it would hang / not work. It definitely won't work
> across GFS client nodes, however. The linux kernel does NOT pass
> control to the filesystem for flock (sys_flock() calls
> flock_lock_file()). Only fcntl() locks call the file_operations lock
> op which is where NFS/GFS... can deal with remote / global file locking
> before the base code calls posix_lock_file() locally.
>
>
>
> On Wednesday, May 7, 2003, at 06:50 PM, Franco Broi wrote:
>
> >
> > Are you guys aware that flock doesn't work? It just hangs.
> >
> >
> > On Wed, 2003-05-07 at 13:15, Cahill, Ben M wrote:
> >> FYI,
> >>
> >> An interesting "cheat" for the cidev!
> >>
> >> -- Ben --
> >>
> >> Opinions are mine, not Intel's
> >>
> >>> -----Original Message-----
> >>> From: Franco Broi [mailto:franco@robres.com.au]
> >>> Sent: Tuesday, May 06, 2003 8:17 PM
> >>> To: opengfs-users@lists.sourceforge.net
> >>> Subject: RE: [ogfs-users]Large filesystem problem.
> >>>
> >>>
> >>> Will do, thanks Ben.
> >>>
> >>> Further to Brian's and my comments re the cidev, it seems it is also
> >>> used during stomith operations to get the IP address of other
> >>> machines
> >>> in the cluster. My current solution is to run ogfsconf when
> >>> the machine
> >>> boots to write the information to a local block device, currently
> >>> /dev/ram0. I know this doesn't ensure that all machines in the
> >>> cluster
> >>> are guaranteed to see the same information, but it's simple
> >>> and seems to
> >>> work for now.
> >>>
> >>> BTW, I'm running 2.4.21-rc1.
> >>>
> >>> On Tue, 2003-05-06 at 21:32, Cahill, Ben M wrote:
> >>>> Hi Franco,
> >>>>
> >>>> Thanks for the usage information . . . this brings in some
> >>> interesting
> >>>> problems to think about and solve . . . I'm copying this to
> >>> the ogfs-dev
> >>>> list, and I'd like to invite you to join that list also . . .
> >>>>
> >>>> -- Ben --
> >>>>
> >>>>> -----Original Message-----
> >>>>> From: Franco Broi [mailto:franco@robres.com.au]
> >>>>> Sent: Tuesday, May 06, 2003 1:15 AM
> >>>>> To: opengfs-users@lists.sourceforge.net
> >>>>> Subject: RE: [ogfs-users]Large filesystem problem.
> >>>>>
> >>>>>
> >>>>>
> >>>>> I've been running tests with the no pool version of OGFS and
> >>>>> things look
> >>>>> to be working pretty well. I've currently got a single
> >>> software RAID5
> >>>>> exported over Myrinet using NBD to 2 OGFS clients. I've
> >>> successfully
> >>>>> written a 600GB file to the filesystem - anybody beat that?
> >>>>>
> >>>>> Some general OGFS thoughts:
> >>>>>
> >>>>> Couldn't the cluster information somehow be written to the OGFS
> >>>>> filesystem? I can't partition my block device, so this
> >>> would make life
> >>>>> quite a bit simpler, and as far as I can tell the
> >>> information is only
> >>>>> used at mount time - so I can't see that it really needs
> >>> to be on a
> >>>>> separate device.
> >>>>>
> >>>>> I'm going to have 7 storage servers each running their own memexp
> >>>>> daemons, I'm a bit worried that the failure of a single
> >>> server could
> >>>>> take the whole cluster down. Is anything being done along
> >>> the lines of
> >>>>> taking failed filesystems offline, rather than panicking
> >>> the machines?
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Sun, 2003-05-04 at 12:39, Cahill, Ben M wrote:
> >>>>>> Cheers indeed!
> >>>>>>
> >>>>>> Thanks very much for the debugging input.
> >>>>>>
> >>>>>> -- Ben --
> >>>>>>
> >>>>>> Opinions are mine, not Intel's
> >>>>>>
> >>>>>>> -----Original Message-----
> >>>>>>> From: Franco Broi [mailto:franco@robres.com.au]
> >>>>>>> Sent: Friday, May 02, 2003 4:49 AM
> >>>>>>> To: opengfs-users@lists.sourceforge.net
> >>>>>>> Subject: RE: [ogfs-users]Large filesystem problem.
> >>>>>>>
> >>>>>>>
> >>>>>>> OK 2 clients working, not as fast as I had hoped, but I'm
> >>>>> sure there's
> >>>>>>> some room for improvement.
> >>>>>>>
> >>>>>>> I'll thrash it over the weekend to see how it holds up.
> >>>>>>>
> >>>>>>> Cheers!
> >>>>>>>
> >>>>>>>
> >>>>>>> On Fri, 2003-05-02 at 16:11, Franco Broi wrote:
> >>>>>>>>
> >>>>>>>> Success! I reduced the internal journal size to 128 (looks
> >>>>>>> like this
> >>>>>>>> might be the default), et voila!
> >>>>>>>>
> >>>>>>>> Filesystem 1K-blocks Used Available Use%
> >>>>> Mounted on
> >>>>>>>> /dev/nb0 1194658232 7820316
> >>> 1186837916 1% /gfs
> >>>>>>>>
> >>>>>>>> Storage server is dual Athlon with RAID card and
> >>>>>>> (currently) 7 RAID 5
> >>>>>>>> disks, exported via nbd (/dev/nb0). cidev is /dev/ram0
> >>>>>>> exported via nbd
> >>>>>>>> (/dev/nb1). Storage server is running memexpd.
> >>>>>>>>
> >>>>>>>> Clients are dual Xeons (hope to have 20 running ogfs
> >>>>> eventually).
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> [franco@charlie18]$ ./seq_write -f /gfs/franco/fred.dat
> >>>>> -r 8000 -n
> >>>>>>>> 1000000 -s fsync
> >>>>>>>> File :/gfs/franco/fred.dat
> >>>>>>>
> >>>>>>>> nbrec: 8000
> >>>>>>>> nrecs: 1000000
> >>>>>>>> MB : 7629.39453
> >>>>>>>> Sync : Fsync flush
> >>>>>>>> 7812500. Kbytes written in 211.441162 seconds
> >>> 36.0828247
> >>>>>>>> Mbytes/Second
> >>>>>>>> 4729.44824 IOs/Sec
> >>>>>>>>
> >>>>>>>> That's one client working.... now to try a second.
> >>>>>>>>
> >>>>>>>> On Fri, 2003-05-02 at 15:28, Franco Broi wrote:
> >>>>>>>>> In fs/super.c around line 826, rv is being set to
> >>>>>>> -ENOMEM, if there
> >>>>>>>>> aren't any external journals this then gets returned.
> >>>>>>>>>
> >>>>>>>>> I commented it out and it got a bit futher until the
> >>>>>>> machine crashed at
> >>>>>>>>> line 1044 dio_arch.c, bad metadata at 75, should be 7
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Fri, 2003-05-02 at 14:35, Cahill, Ben M wrote:
> >>>>>>>>>> I'll look into this (unless you find it first!) . . .
> >>>>>>> thanks for the report.
> >>>>>>>>>>
> >>>>>>>>>> -- Ben --
> >>>>>>>>>>
> >>>>>>>>>> Opinions are mine, not Intel's.
> >>>>>>>>>>
> >>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>> From: Franco Broi [mailto:franco@robres.com.au]
> >>>>>>>>>>> Sent: Friday, May 02, 2003 12:04 AM
> >>>>>>>>>>> To: opengfs-users@lists.sourceforge.net
> >>>>>>>>>>> Subject: RE: [ogfs-users]Large filesystem problem.
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> I get this when trying to mount the filesystem:
> >>>>>>>>>>>
> >>>>>>>>>>> [franco@charlie18 mkfs]# mount -t ogfs
> >>> /dev/nb0 /gfs -o
> >>>>>>>>>>> hostdata=140.0.30.18
> >>>>>>>>>>> mount: wrong fs type, bad option.........
> >>>>>>>>>>>
> >>>>>>>>>>> memexp: Cluster Information Device summary
> >>>>>>>>>>> jid: 1, cid: 1, ipaddr: 140.0.30.18, nstomith: 1
> >>>>>>>>>>> jid: 0, cid: 0, ipaddr: 140.0.30.8, nstomith: 1
> >>>>>>>>>>> Node timeout: 30
> >>>>>>>>>>> external journal base = 18446744073709551615
> >>>>>>>>>>> OGFS: can't read journal index (-12)
> >>>>>>>>>>>
> >>>>>>>>>>> journal.cf:
> >>>>>>>>>>> fsdev /dev/nb0
> >>>>>>>>>>>
> >>>>>>>>>>> journals 2
> >>>>>>>>>>>
> >>>>>>>>>>> journal 0 int 256
> >>>>>>>>>>> journal 1 int 256
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> On Fri, 2003-05-02 at 10:53, Franco Broi wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>> My filesystem size was a bit screwy, but I
> >>>>> think I've found
> >>>>>>>>>>> a problem
> >>>>>>>>>>>> with the ioctl to get the device size, if I
> >>>>> make the size
> >>>>>>>>>>> argument an
> >>>>>>>>>>>> unsigned int, it seems to work.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> [franco@charlie18 mkfs]# ./mkfs.ogfs -p memexp -t
> >>>>>>> /dev/nb1 -c
> >>>>>>>>>>>> ~/ogfs/journal.cf
> >>>>>>>>>>>> Device: /dev/nb0
> >>>>>>>>>>>> Blocksize: 4096
> >>>>>>>>>>>> Filesystem Size: 298570800
> >>>>>>>>>>>> Journals: 2
> >>>>>>>>>>>> Resource Groups: 5207
> >>>>>>>>>>>> Locking Protocol: memexp
> >>>>>>>>>>>> Lock Table: /dev/nb1
> >>>>>>>>>>>>
> >>>>>>>>>>>> Syncing...
> >>>>>>>>>>>> All Done
> >>>>>>>>>>>>
> >>>>>>>>>>>> Ok I'll start testing.
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>> On Fri, 2003-05-02 at 10:04, Franco Broi wrote:
> >>>>>>>>>>>>> This looks much the same as for the pool version.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [franco@charlie18 ~/ogfs]# mkfs.ogfs -p memexp -t
> >>>>>>>>>>> /dev/nb1 -c journal.cf
> >>>>>>>>>>>>> -d -F 122356826
> >>>>>>>>>>>>> Command Line Arguments/Defaults:
> >>>>>>>>>>>>> proto = memexp
> >>>>>>>>>>>>> table = /dev/nb1
> >>>>>>>>>>>>> block size (bytes) = 4096
> >>>>>>>>>>>>> seg_size (blocks) = 16
> >>>>>>>>>>>>> config = journal.cf
> >>>>>>>>>>>>> ignore_pool = 0
> >>>>>>>>>>>>> journals = 0
> >>>>>>>>>>>>> jsize (MBytes) = 128
> >>>>>>>>>>>>> debug = 1
> >>>>>>>>>>>>> device =
> >>>>>>>>>>>>> Intern journal size (MB) = 256 MB
> >>>>>>>>>>>>> Intern journal size (segments) = 256 segments
> >>>>>>>>>>>>> Intern journal size (MB) = 256 MB
> >>>>>>>>>>>>> Intern journal size (segments) = 256 segments
> >>>>>>>>>>>>> FS Device = /dev/nb0
> >>>>>>>>>>>>> Raw FS Device size (512B sectors) =
> >>>>> 18446744071804366080
> >>>>>>>>>>>>> Jrnl-Seg-aligned FS Device size (sectors) =
> >>>>>>> 18446744071804366080
> >>>>>>>>>>>>> Unusable sectors = 0
> >>>>>>>>>>>>> journal loop 0, type = 1
> >>>>>>>>>>>>> journal loop 1, type = 1
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Data Subpool 0
> >>>>>>>>>>>>> nrgrp = 40210710954509
> >>>>>>>>>>>>> mkfs.ogfs: unable to allocate memory on line
> >>>>> 99 of file
> >>>>>>>>>>> fs_geometry.c
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> I get the same whether or not I pass the -F
> >>>>> flag so it
> >>>>>>>>>>> looks like it's
> >>>>>>>>>>>>> not getting the correct device size for
> >>> some reason.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Here is a mke2fs to show that the device is
> >>>>> accessible
> >>>>>>>>>>> and working.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> [franco@charlie18 ~/ogfs]# mke2fs /dev/nb0
> >>>>>>>>>>>>> mke2fs 1.27 (8-Mar-2002)
> >>>>>>>>>>>>> Filesystem label=
> >>>>>>>>>>>>> OS type: Linux
> >>>>>>>>>>>>> Block size=4096 (log=2)
> >>>>>>>>>>>>> Fragment size=4096 (log=2)
> >>>>>>>>>>>>> 149372928 inodes, 298722720 blocks
> >>>>>>>>>>>>> 14936136 blocks (5.00%) reserved for the
> >>> super user
> >>>>>>>>>>>>> First data block=0
> >>>>>>>>>>>>> 9117 block groups
> >>>>>>>>>>>>> 32768 blocks per group, 32768 fragments per group
> >>>>>>>>>>>>> 16384 inodes per group
> >>>>>>>>>>>>> Superblock backups stored on blocks:
> >>>>>>>>>>>>> etc...
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> On Fri, 2003-05-02 at 05:54, Cahill, Ben M wrote:
> >>>>>>>>>>>>>> Would you be willing to try the no-pool
> >>> code? (see
> >>>>>>>>>>> other email).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'd be grateful if you could, and
> >>> report results.
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Thanks!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> -- Ben --
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Opinions are mine, not Intel's
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> -----Original Message-----
> >>>>>>>>>>>>>>> From: Franco Broi
> >>> [mailto:franco@robres.com.au]
> >>>>>>>>>>>>>>> Sent: Wednesday, April 30, 2003 11:07 PM
> >>>>>>>>>>>>>>> To: opengfs-users@lists.sourceforge.net
> >>>>>>>>>>>>>>> Subject: Re: [ogfs-users]Large
> >>> filesystem problem.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Sorry didn't see the warning message
> >>> from ptool:
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Warning: Ptool detected 0 blocks for /dev/nb0
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> I guess this is because it doesn't show up in
> >>>>>>>>>>>>>>> /proc/partitions, but it's
> >>>>>>>>>>>>>>> strange how it worked for a smaller
> >>>>> filesystem, maybe
> >>>>>>>>>>> because it just
> >>>>>>>>>>>>>>> used the minimum number of groups on that
> >>>>> occasion?
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> On Thu, 2003-05-01 at 10:00, Franco
> >>> Broi wrote:
> >>>>>>>>>>>>>>>> mkfs_gfs is failing when for my rather
> >>>>>>> large filesystem
> >>>>>>>>>>>>>>> (slightly < 2TB)
> >>>>>>>>>>>>>>>> We have ext2/3 filesystems of this size
> >>>>>>> which work fine.
> >>>>>>>>>>>>>>> I've managed to
> >>>>>>>>>>>>>>>> get smaller gfs filesystems working so
> >>>>>>> maybe there's some
> >>>>>>>>>>>>>>> limitation in
> >>>>>>>>>>>>>>>> gfs which is less than the maximum
> >>>>> allowed by Linux.
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> [franco@charlie18 /]# mkfs_gfs -p memexp -t
> >>>>>>>>>>>>>>> /dev/pool/pool0_cidev -j 2
> >>>>>>>>>>>>>>>> -i /dev/pool/pool0 -d
> >>>>>>>>>>>>>>>> Command Line Arguments:
> >>>>>>>>>>>>>>>> proto = memexp
> >>>>>>>>>>>>>>>> table = /dev/pool/pool0_cidev
> >>>>>>>>>>>>>>>> bsize = 4096
> >>>>>>>>>>>>>>>> seg_size = 16
> >>>>>>>>>>>>>>>> ignore_pool = 1
> >>>>>>>>>>>>>>>> journals = 2
> >>>>>>>>>>>>>>>> jsize = 128
> >>>>>>>>>>>>>>>> debug = 1
> >>>>>>>>>>>>>>>> device = /dev/pool/pool0
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Partition size = 18446744073397553904
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> SubPool Geometry: (in basic blocks)
> >>>>>>>>>>>>>>>> Subpool #0:
> >>>>>>>>>>>>>>>> BStart = 0, BLength =
> >>> 18446744073397029616
> >>>>>>>>>>>>>>>> type = gfs_data
> >>>>>>>>>>>>>>>> Subpool #1:
> >>>>>>>>>>>>>>>> BStart = 18446744073397029616,
> >>>>> BLength = 262144
> >>>>>>>>>>>>>>>> type = gfs_journal
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> SubPool Geometry: (in FS blocks)
> >>>>>>>>>>>>>>>> Subpool #0:
> >>>>>>>>>>>>>>>> BStart = 0, BLength =
> >>> 2305843009174628702
> >>>>>>>>>>>>>>>> type = gfs_data
> >>>>>>>>>>>>>>>> Subpool #1:
> >>>>>>>>>>>>>>>> BStart = 2305843009174628702,
> >>>>> BLength = 32768
> >>>>>>>>>>>>>>>> type = gfs_journal
> >>>>>>>>>>>>>>>> Subpool #2:
> >>>>>>>>>>>>>>>> BStart = 2305843009174661470,
> >>>>> BLength = 32768
> >>>>>>>>>>>>>>>> type = gfs_journal
> >>>>>>>>>>>>>>>> journals = 2
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Data Subpool 0
> >>>>>>>>>>>>>>>> nrgrp = 40210710957983
> >>>>>>>>>>>>>>>> mkfs_gfs: unable to allocate memory on
> >>>>> line 99 of
> >>>>>>>>>>> file fs_geometry.c
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Plus is there any chance of getting
> >>>>>>> passemble to see
> >>>>>>>>>>>>>>> partitions other
> >>>>>>>>>>>>>>>> than those in /proc? ie /dev/nb0 etc..
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> BTW, openGFS using nbd over Myrinet-Sockets
> >>>>>>> goes like a
> >>>>>>>>>>>>>>> train, at least
> >>>>>>>>>>>>>>>> for a single node (can't use HyperSCSI - it
> >>>>>>> doesn't work
> >>>>>>>>>>>>>>> over Myrinet).
> >>>>>>
> >>>>>>
> >>>>>> -------------------------------------------------------
> >>>>>> This sf.net email is sponsored by:ThinkGeek
> >>>>>> Welcome to geek heaven.
> >>>>>> http://thinkgeek.com/sf
> >>>>>> _______________________________________________
> >>>>>> Opengfs-users mailing list
> >>>>>> Opengfs-users@lists.sourceforge.net
> >>>>>> https://lists.sourceforge.net/lists/listinfo/opengfs-users
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> -------------------------------------------------------
> >>>>> This sf.net email is sponsored by:ThinkGeek
> >>>>> Welcome to geek heaven.
> >>>>> http://thinkgeek.com/sf
> >>>>> _______________________________________________
> >>>>> Opengfs-users mailing list
> >>>>> Opengfs-users@lists.sourceforge.net
> >>>>> https://lists.sourceforge.net/lists/listinfo/opengfs-users
> >>>>>
> >>>>
> >>>>
> >>>> -------------------------------------------------------
> >>>> This sf.net email is sponsored by:ThinkGeek
> >>>> Welcome to geek heaven.
> >>>> http://thinkgeek.com/sf
> >>>> _______________________________________________
> >>>> Opengfs-users mailing list
> >>>> Opengfs-users@lists.sourceforge.net
> >>>> https://lists.sourceforge.net/lists/listinfo/opengfs-users
> >>>
> >>>
> >>>
> >>>
> >>> -------------------------------------------------------
> >>> Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
> >>> The only event dedicated to issues related to Linux
> >>> enterprise solutions
> >>> www.enterpriselinuxforum.com
> >>>
> >>> _______________________________________________
> >>> Opengfs-users mailing list
> >>> Opengfs-users@lists.sourceforge.net
> >>> https://lists.sourceforge.net/lists/listinfo/opengfs-users
> >>>
> >>
> >>
> >> -------------------------------------------------------
> >> Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
> >> The only event dedicated to issues related to Linux enterprise
> >> solutions
> >> www.enterpriselinuxforum.com
> >>
> >> _______________________________________________
> >> Opengfs-devel mailing list
> >> Opengfs-devel@lists.sourceforge.net
> >> https://lists.sourceforge.net/lists/listinfo/opengfs-devel
> >>
> >
> >
> >
> >
> > -------------------------------------------------------
> > Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
> > The only event dedicated to issues related to Linux enterprise
> > solutions
> > www.enterpriselinuxforum.com
> >
> > _______________________________________________
> > Opengfs-devel mailing list
> > Opengfs-devel@lists.sourceforge.net
> > https://lists.sourceforge.net/lists/listinfo/opengfs-devel
> >
>
>
>
> -------------------------------------------------------
> Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
> The only event dedicated to issues related to Linux enterprise solutions
> www.enterpriselinuxforum.com
>
> _______________________________________________
> Opengfs-devel mailing list
> Opengfs-devel@lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/opengfs-devel
>
-------------------------------------------------------
Enterprise Linux Forum Conference & Expo, June 4-6, 2003, Santa Clara
The only event dedicated to issues related to Linux enterprise solutions
www.enterpriselinuxforum.com
_______________________________________________
Opengfs-devel mailing list
Opengfs-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/opengfs-devel