How to configure Storage Box Services with OpenStack
Audio : Listen to This Blog.
OpenStack is a set of software tools for building and managing cloud computing platforms for public and private clouds.
It is a cloud operating system that controls large pools of compute, storage, and networking resources throughout a datacenter, all managed through a dashboard.
Accessing SAN Storage in OpenStack
Cinder is the OpenStack component that provides access to, and manages block storage. The cinder interface specifies a number of discrete functions such as create, delete and attach volume/drive etc.
Cinder provides persistent block storage resources (volumes) to VMs. These volumes can be detached from one instance and re-attached to another.
Cinder supports drivers that allow cinder volumes to be created and presented using storage solutions from vendors. Third-party storage vendors use cinder’s plug-in architecture to do the necessary integration work.
Advantages:
- Driver helps to create LUN and extend the storage capacity
- Driver helps to back-up the data using backup service
- It supports both managed/unmanaged LUN
- It helps to clone and create volumes from a VM image
Minimum system requirement to configure OpenStack Controller
Hardware requirements:
- Dual-core CPU
- 2 GB RAM
- 5 GB disk
Supported OS:
- RHEL
- CentOS
- Fedora
- Ubuntu
- OpenSUSE
- SUSE Linux Enterprise
Network:
- 2 Network Interface card with 100 Mbps/1 Gbps speed
– One network for OpenStack installation
– Another network for storage to make connection with SAN
Note: OpenStack controller means it includes all the services such as nova, cinder, glance and neutron.
SAN Storage:
Need one third party storage-subsystem to configure storage with OpenStack.
Essential steps to configure SAN storage with OpenStack
Step 1
Need to setup the following property values in OpenStack cinder configuration file (/etc/cinder/cinder.conf).
i. Enable storage backend:
enabled_backends = storage name // for example NetApp
ii. Specify volume name: – to identify particular volume in storage
volume_name_template = openstack-%s
iii. Add NFS storage information and backup driver: – to backup data
backup_driver = cinder.backup.drivers.nfs backup_share =nfs storage IP:/nfsshare
iv. Storagebox information
[storagename]//For Ex: We should specify NetApp here volume_driver=storage driver volume_backend_name=storage name san_login=storage username san_password=storage password san_ip=storage ip
v. Enable multipath
use_multipath_for_image_xfer = True
Note: StorageBox information and Enable multipath has to be added at the end of configuration file.
Step 2
Array Vendor’s cinder driver has to be added in below location for creating the volumes in storage
/usr/lib/python2.7/site-packages/cinder/volume/drivers/
Step 3
Restart cinder services:
systemctl restart openstack-cinder-api.service systemctl restart openstack-cinder-backup.service systemctl restart openstack-cinder-scheduler.service systemctl restart openstack-cinder-volume.service
Note: If services are not restarted properties set won’t be effective
Tips and Tricks for trouble shooting
Serial 1: Specify network ID
Symptom:
Error (Conflict): Multiple possible networks found, use a Network ID to be more specific. (HTTP 409) (Request-ID: req-251e6d02-5358-41f7-95a4-b58c52cbc74b). Usually this error occurs only if the given name is ambiguous. It is occurred in OpenStack liberty. Network name is specified while creating the instance. It is failing due to the name given is ambiguous.
Approach to tackle the symptom:
Affected version: Liberty – Instance is created using network name. Usually this error occurs only if the given name is ambiguous.
Fixed version: Mitaka – Instance is created using network id.
Issue got fixed in Mitaka which is next release of openstack liberty.
To solve the issue, we need specify net id while creating instance.
Steps:
- Login to OpenStack controller node using putty
- List all the volume which are created in OpenStack controller node
[root@mitaka-hos1~(keystone_admin)]# cinder list +--------------------------------------+-----------+------------------+------+------+--------------+----------+-------------+-------------+ | ID | Status | Migration Status | Name | Size | Volume Type | Bootable | Multiattach | Attached to | +--------------------------------------+-----------+------------------+------+------+--------------+----------+-------------+-------------+ | eee3f8fc-3306-44c2-84c8-d2ab1ab4c775 | available | success | vol2 | 5 | array | true | False | | +--------------------------------------+-----------+------------------+------+------+--------------+----------+-------------+-------------+ [root@mitaka-hos1 ~(keystone_admin)]#
3. List available networks in OpenStack controller node.
[root@mitaka-hos1 ~(keystone_admin)]# neutron net-list +--------------------------------------+---------+------------------------------------------------------+ | id | name | subnets | +--------------------------------------+---------+------------------------------------------------------+ | 489a3170-0ee3-4ae0-a5ef-8a766c50249f | public | 20ae85c9-a89b-4689-9b76-1c395f842b01 172.24.4.224/28 | | ade84d1d-343c-42ee-a603-df2e84274bd4 | private | ef2e62bc-0b94-44f3-bb2c-82963c2eb705 10.0.0.0/24 | +--------------------------------------+---------+------------------------------------------------------
4. Create instance using net-id and volume id.
nova boot --flavor m1.tiny --boot-volume eee3f8fc-3306-44c2-84c8-d2ab1ab4c775 --availability-zone nova:host1 inst3 --nic net-id=489a3170-0ee3-4ae0-a5ef-8a766c50249f
+--------------------------------------+--------------------------------------------------+
| Property | Value |
+--------------------------------------+--------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | nova |
…..
…..
| OS-EXT-SRV-ATTR:instance_name | instance-00000009 |
| OS-EXT-STS:task_state | scheduling
| created | 2016-08-31T08:39:20Z |
| flavor | m1.tiny (1) |
| hostId | |
| id | 8d202079-a9c2-4175-b5ff-7bc0638e06f4 |
| image | Attempt to boot from volume - no image supplied |
| key_name | - |
| metadata | {} |
| name | inst3 |
| os-extended-volumes:volumes_attached | [{"id": "eee3f8fc-3306-44c2-84c8-d2ab1ab4c775"}] |
| progress | 0 |
| security_groups | default |
| status | BUILD |
| tenant_id | 8c786c64ee8143b4b83bd1109b413ce5 |
| updated | 2016-08-31T08:39:21Z |
| user_id | 7aa859512fa146c4ba355d3499fffa14 |
+--------------------------------------+--------------------------------------------------+
Bug: https://bugs.launchpad.net/python-novaclient/+bug/1569840
Serial 2: Specify multipath in cinder file
Symptom:
2016-07-29 04:53:29.728 2103 ERROR cinder.scheduler.manager [req-9de37842-0da5-4c05-9ce3-38b4b38aa1bf 91327080eb604f0596eec6f3191f8b76 322494d5ae904c9680b318e7231bbeff - - -] Failed to schedule_manage_existing: No valid host was found. Cannot place volume 7c1a314a-b46e-475f-8c03-823ba2ca6179 on host
Approach to tackle the symptom:
To rectify this issue, need to add below line at end of the cinder.conf.
use_multipath_for_image_xfer = True
Serial 3: Specify pool name in cinder file
Symptom:
016-07-20 08:27:44.268 718 ERROR oslo_messaging.rpc.dispatcher response = self._execute_create_vol(volume, pool_name, reserve) 2016-07-20 08:27:44.268 718 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/xxx.py", line 533, in inner_connection_checker 2016-07-20 08:27:44.268 718 ERROR oslo_messaging.rpc.dispatcher return func(self, *args, **kwargs) 2016-07-20 08:27:44.268 718 ERROR oslo_messaging.rpc.dispatcher File "/usr/lib/python2.7/site-packages/cinder/volume/drivers/xxx.py", line 522, in inner_response_checker 2016-07-20 08:27:44.268 718 ERROR oslo_messaging.rpc.dispatcher raise xxxAPIException(msg) 2016-07-20 08:27:44.268 718 ERROR oslo_messaging.rpc.dispatcher xxxAPIException: API _execute_create_vol failed with error string SM-err-pool-not-found 2016-07-20 08:27:44.268 718 ERROR oslo_messaging.rpc.dispatcher
Approach to tackle the symptom:
Need to specify pool name in end of the cinder.config.
ventor_pool_name= pool name
Serial 4: Specify OpenStack controller IP
Symptom:
Unable to connect to OpenStack instance console using VNC using OpenStack Horizon
Error Message: Failed to connect to server (code 1006)
Environment: OpenStack RDO Juno, CentOS7
Approach to tackle the symptom:
You need to update vncserver_proxyclient_address with the OpenStack controller IP address OR novavncproxy_base_url IP address in the nova.conf(/etc/nova/nova.conf)
vncserver_proxyclient_address=openstack controller IP address
then restart you compute service
/etc/init.d/openstack-nova-compute restart
Serial 5: Specify nfs driver in cinder config
Symptom:
[root@hiqa-rhel1 ~(keystone_admin)]# cat /var/log/cinder/backup.log | grep unsupport 2017-02-10 04:21:17.143 22496 DEBUG cinder.service [req-5210ae1c-ae31-41e5-b927-9102c776e941 - - - - -] enable_unsupported_driver : False wait /usr/lib/python2.7/site-packages/cinder/service.py:611 2017-02-10 04:21:17.222 22496 DEBUG oslo_service.service [req-5210ae1c-ae31-41e5-b927-9102c776e941 - - - - -] enable_unsupported_driver = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2622 2017-02-10 04:34:43.918 27333 DEBUG cinder.service [req-95201c9a-8766-4589-b4be-3d076890fc54 - - - - -] enable_unsupported_driver : False wait /usr/lib/python2.7/site-packages/cinder/service.py:611 2017-02-10 04:34:43.977 27333 DEBUG oslo_service.service [req-95201c9a-8766-4589-b4be-3d076890fc54 - - - - -] enable_unsupported_driver = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2622 2017-02-11 22:17:41.654 17423 DEBUG cinder.service [req-9b064618-21cd-4400-a428-70b909c3d141 - - - - -] enable_unsupported_driver : False wait /usr/lib/python2.7/site-packages/cinder/service.py:611 2017-02-11 22:17:41.709 17423 DEBUG oslo_service.service [req-9b064618-21cd-4400-a428-70b909c3d141 - - - - -] enable_unsupported_driver = False log_opt_values /usr/lib/python2.7/site-packages/oslo_config/cfg.py:2622 [root@hiqa-rhel1 ~(keystone_admin)]#
Approach to tackle the symptom:
To rectify this issue, need to specify nfs driver in the cinder.conf file:
backup_driver = cinder.backup.drivers.nfs
Serial 6: Grant permission to backup volume in nfs server
Symptom:
OSError: [Errno 13] Permission denied: '/var/lib/cinder/backup_mount/f
Approach to tackle the symptom:
chown cinder:cinder -R /var/lib/cinder/backup_mount/