Could ceph-deploy handle unknown or custom distribution? (Was: Mourning the demise of mkcephfs)

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



On 13/11/13 16:33, Mark Kirkwood wrote:
I believe he is using a self built (or heavily customized) Linux
installation - so distribution detection is not going to work in this
case. I'm wondering if there could be some sensible fall back for that,
e.g:

- refuse to install or purge
- assume sysv init


It was raining here on Saturday so I thought I'd take a look at if this was even feasible (see attached). These patches are a rough experiment to see what *might* be involved in getting to a point where you could get a system going on one of the distributions (or a custom build) that we currently do not support. So view as a thought-in-progress rather than even wip :-) It could also be quite the wrong approach - that is ok, just let me know what you think. Hopefully even if it is wrong, it may help stimulate discussion (or patches) for how to do it right!

The 1st patch defines an 'unknown' distribution for cases where one of our well known cases is *not* detected. With it applied I can create mons and it will start them directly with ceph-mon.

The 2nd and 3rd patch do similarly for osd, defining a type of startup called 'direct' to mark the osd instance with. I needed to patch ceph-disk as well so it could start the resulting osd directly with ceph-osd, and add it into the crushmap at a (hopefully) sensible place.

(BTW I did want to do this all in ceph-deploy, but right now it does not know how to get an osd's id number from a disk device - mind you this could be useful to add for things like 'list' and 'destroy'... but I digress).

Anyway I have attached a log of me getting a system of 2 Archlinux nodes up. These were KVM guests built identically and ceph (0.72) was compiled from src and installed.

Cheers

Mark



commit 0629a803d7a1ecb651ed34c402c41f72bafaec7d
Author: Mark Kirkwood <mark.kirkwood@xxxxxxxxx>
Date:   Sun Nov 17 00:43:20 2013 +1300

    Initial prototype for handling unknown distribution types.

diff --git a/ceph_deploy/hosts/__init__.py b/ceph_deploy/hosts/__init__.py
index f4ba203..8f6a541 100644
--- a/ceph_deploy/hosts/__init__.py
+++ b/ceph_deploy/hosts/__init__.py
@@ -6,7 +6,7 @@ on the type of distribution/version we are dealing with.
 """
 import logging
 from ceph_deploy import exc, lsb
-from ceph_deploy.hosts import debian, centos, fedora, suse, remotes
+from ceph_deploy.hosts import debian, centos, fedora, suse, remotes, unknown
 from ceph_deploy.connection import get_connection
 
 logger = logging.getLogger()
@@ -62,6 +62,7 @@ def _get_distro(distro, fallback=None):
         'redhat': centos,
         'fedora': fedora,
         'suse': suse,
+        'unknown': unknown,
         }
     try:
         return distributions[distro]
@@ -79,4 +80,6 @@ def _normalized_distro_name(distro):
         return 'scientific'
     elif distro.startswith(('suse', 'opensuse')):
         return 'suse'
+    elif distro.startswith('unknown'):
+        return 'unknown'
     return distro
diff --git a/ceph_deploy/hosts/remotes.py b/ceph_deploy/hosts/remotes.py
index 62766f7..c759982 100644
--- a/ceph_deploy/hosts/remotes.py
+++ b/ceph_deploy/hosts/remotes.py
@@ -18,10 +18,21 @@ def platform_information():
         major_version = release.split('.')[0]
         codename = debian_codenames.get(major_version, '')
 
+    distro = str(distro).rstrip(),
+    distro = str(release).rstrip(),
+    distro = str(codename).rstrip()
+
+    if distro == '':
+        distro = 'unknown'
+    if release == '':
+        release = 'unknown'
+    if codename == '':
+        codename = 'unknown'
+
     return (
-        str(distro).rstrip(),
-        str(release).rstrip(),
-        str(codename).rstrip()
+        str(distro),
+        str(release),
+        str(codename)
     )
 
 
diff --git a/ceph_deploy/hosts/unknown/__init__.py b/ceph_deploy/hosts/unknown/__init__.py
new file mode 100644
index 0000000..d67ad7e
--- /dev/null
+++ b/ceph_deploy/hosts/unknown/__init__.py
@@ -0,0 +1,8 @@
+import mon
+
+# Allow to set some information about this distro
+#
+
+distro = None
+release = None
+codename = None
diff --git a/ceph_deploy/hosts/unknown/mon/__init__.py b/ceph_deploy/hosts/unknown/mon/__init__.py
new file mode 100644
index 0000000..fca0e0d
--- /dev/null
+++ b/ceph_deploy/hosts/unknown/mon/__init__.py
@@ -0,0 +1 @@
+from create import create
diff --git a/ceph_deploy/hosts/unknown/mon/create.py b/ceph_deploy/hosts/unknown/mon/create.py
new file mode 100644
index 0000000..ed2a3ba
--- /dev/null
+++ b/ceph_deploy/hosts/unknown/mon/create.py
@@ -0,0 +1,30 @@
+from ceph_deploy.hosts import common
+from ceph_deploy.lib.remoto import process
+
+
+def create(distro, args, monitor_keyring):
+    hostname = distro.conn.remote_module.shortname()
+    common.mon_create(distro, args, monitor_keyring, hostname)
+
+    # no idea about startup architecture so start directly, run + create keys
+    process.run(
+        distro.conn,
+        [
+            'ceph-mon',
+            '--cluster', args.cluster,
+            '-c', '/etc/ceph/{cluster}.conf'.format(cluster=args.cluster),
+            '-i', hostname,
+        ],
+        timeout=7,
+    )
+
+    process.run(
+        distro.conn,
+        [
+            'ceph-create-keys',
+            '--cluster', args.cluster,
+            '-i', hostname,
+        ],
+        timeout=7,
+    )
+
diff --git a/ceph_deploy/lsb.py b/ceph_deploy/lsb.py
index b7f3e73..2f608de 100644
--- a/ceph_deploy/lsb.py
+++ b/ceph_deploy/lsb.py
@@ -118,4 +118,6 @@ def choose_init(distro, codename):
     """
     if distro == 'Ubuntu':
         return 'upstart'
+    elif  distro == 'unknown':
+        return 'direct'
     return 'sysvinit'
*** ceph-disk.orig	Sun Nov 17 11:34:33 2013
--- ceph-disk	Mon Nov 18 13:21:20 2013
***************
*** 96,101 ****
--- 96,102 ----
      'upstart',
      'sysvinit',
      'systemd',
+     'direct',
      'auto',
      ]
  
***************
*** 1438,1443 ****
--- 1439,1470 ----
                      'osd.{osd_id}'.format(osd_id=osd_id),
                      ],
                  )
+         elif os.path.exists(os.path.join(path, 'direct')):
+             # no idea about which init, start directly and set crush location
+             subprocess.check_call(
+                 args=[
+                     '/usr/bin/ceph-osd',
+                     #'--cluster', args.cluster,
+                     #'-c', '/etc/ceph/{cluster}.conf'.format(cluster=args.cluster),
+                     '-i', '{osd_id}'.format(osd_id=osd_id),
+                     ],
+                 )
+ 
+             # assume weight 1 (wrong but easy to change)!
+             subprocess.check_call(
+                 args=[
+                     '/usr/bin/ceph',
+                     #'--cluster', args.cluster,
+                     #'-c', '/etc/ceph/{cluster}.conf'.format(cluster=args.cluster),
+                     'osd',
+                     'crush',
+                      'add',
+                     'osd.{osd_id}'.format(osd_id=osd_id),
+                     '1',
+                     'root=default',
+                     'host={hostname}'.format(hostname=platform.node()),
+                     ],
+                 )
          else:
              raise Error('{cluster} osd.{osd_id} is not tagged with an init system'.format(
                      cluster=cluster,
***************
*** 1683,1690 ****
--- 1710,1722 ----
                      (distro, release, codename) = platform.dist()
                      if distro == 'Ubuntu':
                          init = 'upstart'
+                     if distro == 'unknown' or distro == '':
+                         LOG.debug('distro is empty, assuming init is direct')
+                         init = 'direct'
                      else:
                          init = 'sysvinit'
+             if init == 'direct':
+                 LOG.debug('init is direct')
  
              LOG.debug('Marking with init system %s', init)
              with file(os.path.join(path, init), 'w'):
After applying patch for unknown distro
=======================================

State:

- monitors created and start ok
- osd created ok
  * not started (startup by ceph-disk)
  * no crush setting (set by init scripts)

Test on Archlinux hosts (zor[2,3]), running ceph-deploy from Ubuntu workstation 
(localhost)


1/ Do src build installing binaries and udev:

(zor2) $ sudo make install
(zor2) $ sudo cp udev/* /lib/udev/rules.d/
(zor2) $ cd /var/lib/ceph;sudo mkdir bootstrap-mds bootstrap-osd mds mon osd tmp
(zor2) $ sudo mkdir /etc/ceph

[same for zor3]


2/ Deploy with (patched) ceph-deploy 

(localhost) $ ceph-deploy new zor2
(localhost) $ ceph-deploy mon create zor2
(localhost) $ ceph-deploy gatherkeys zor2
(localhost) $ ceph-deploy disk zap zor2:/dev/vdb
(localhost) $ ceph-deploy disk zap zor3:/dev/vdb
(localhost) $ ceph-deploy osd prepare zor2:/dev/vdb
(localhost) $ ceph-deploy osd prepare zor3:/dev/vdb
(localhost) $ ceph-deploy osd activate zor2:/dev/vdb1
(localhost) $ ceph-deploy osd activate zor3:/dev/vdb1


3/ Check state

(localhost) $ ceph -c ceph.conf -k ceph.client.admin.keyring -s
    cluster de5d8fac-58b1-411a-8047-dd46d0d91246
     health HEALTH_OK
     monmap e1: 1 mons at {zor2=192.168.122.12:6789/0}, election epoch 2, quorum 0 zor2
     osdmap e36: 2 osds: 2 up, 2 in
      pgmap v55: 192 pgs, 3 pools, 0 bytes data, 0 objects
            70852 kB used, 6052 MB / 6121 MB avail
                 192 active+clean

(localhost) $ ceph -c ceph.conf -k ceph.client.admin.keyring osd tree
# id	weight	type name	up/down	reweight
-1	2	root default
-2	1		host zor3
1	1			osd.1	up	1	
-3	1		host zor2
0	1			osd.0	up	1	



[Index of Archives]     [CEPH Users]     [Ceph Large]     [Information on CEPH]     [Linux BTRFS]     [Linux USB Devel]     [Video for Linux]     [Linux Audio Users]     [Yosemite News]     [Linux Kernel]     [Linux SCSI]
  Powered by Linux