Note: this document will be split in several posts in order to fit within characters’ limit. Make sure to browse the second post as well for more information ![:wink:]( ":wink:")
.
Preambule
This is a wikified post documenting how Rockstor implements the rock-on framework, and should thus be considered a live document expected to be updated as necessary. As a user documentation already exists on how to use the framework (see link below), however, this write-up intends to first summarize the overall rock-on framework concept, describe its implementation from a developer’s perspective, and centralize the discussion on its current state and future evolution. By combining these different elements into a single document, this post will thus attempt to provide an integrative view of the current state of the rock-on framework and bring together everybody’s expertise and ideas to foster a continuous, coordinated, and coherent improvement. In this context, recommendations and suggestions are more than welcome!
While many details will be provided, it is important to note that a focus will be placed on simplicity in order to provide a general description of how the different processes work and interconnect. As a result, we will not go into a line-by-line description of all steps, and only the code that directly mediates each step of interest will be shown—the rest will be masked ((...)). For details, however, we will provide the reader with a link to the full code.
For a general description of the rock-on framework, please see the related section in Rockstor’s documentation:
http://rockstor.com/docs/docker-based-rock-ons/overview.html
Implementation overview
In its essence, a rock-on corresponding to an ensemble of settings in a JSON format. Hosted on a public Github repository (rockon-registry), they contain all necessary information regarding the underlying docker containers and their configuration, as well as the rock-on’s metadata. While all parameters are stored in Rockstor’s database following a “key:value” structure, some parameters, such as volumes and environment variables, for instance, will have their value defined by the user during the rock-on install process. Indeed, these settings are surfaced in the webUI during the install process to be customized by the user. Once all these settings are filled by the user, Rockstor updates their values in the database before starting building up the docker commands to run the underlying container(s).
Once the user clicks the “Submit” button at the end of the rock-on install wizard, Rockstor fetches all parameters linked to the given rock-on and builds a corresponding docker run command for each container defined in the rock-on. Once the docker run command is built, Rockstor triggers it as a background ztask surfaced in the webUI as “Installing”. A successful rock-on install thus results in the creation and start of its underlying docker container(s) using the settings defined in the JSON file and by the user during the install wizard.
Finally, toggling ON and OFF a rock-on from the webUI simply translates into starting and stopping the underlying container(s), whereas uninstalling it will remove it/them.
Implementation details
In this section, we will describe in more detail the different aspects of the rock-ons framework and provide heavy references to the corresponding parts of Rockstor’s code. While this is not an exhaustive documentation of the underlying code, this will hopefully help serve as an illustration of Rockstor’s logic and offer starting points and guidance into the specific parts of this framework.
Main items of interest
Database
All rock-ons-related information is stored in the storageadmin database under dedicated but interconnected models (depicted in the picture below).
As you can see above, rock-ons-related models are centered around the DContainer model rather than the RockOn model. This is simply due to the fact that the container is the base unit of the docker environment and thus represents the best entity to link all information. As a rock-on can include multiple containers, however, the RockOn model is the unit used by Rockstor to surface this information to the user to interact with.
Accordingly, the RockOn model mostly includes elements surfaced to the user in the webUI:
link to code
class RockOn(models.Model):
name = models.CharField(max_length=1024)
description = models.CharField(max_length=2048)
version = models.CharField(max_length=2048)
state = models.CharField(max_length=2048)
status = models.CharField(max_length=2048)
link = models.CharField(max_length=1024, null=True)
website = models.CharField(max_length=2048, null=True)
https = models.BooleanField(default=False)
icon = models.URLField(max_length=1024, null=True)
ui = models.BooleanField(default=False)
volume_add_support = models.BooleanField(default=False)
more_info = models.CharField(max_length=4096, null=True)
The DContainer model, on the other hand, is very simple and the only critical information it stores is its name:
link to code
class DContainer(models.Model):
rockon = models.ForeignKey(RockOn)
dimage = models.ForeignKey(DImage)
name = models.CharField(max_length=1024, unique=True)
launch_order = models.IntegerField(default=1)
# if uid is None, container's owner is not set. defaults to root. if it's
# -1, then owner is set to the owner of first volume, if any. if it's an
# integer other than -1, like 0, then owner is set to that uid.
uid = models.IntegerField(null=True)
As noted above, this is due to the fact that the container is a base unit of the docker environment and its model thus mostly represents a hub of all specific information applied onto it. This specific information is thus stored in dedicated models keyed by container and includes:
class DImage(models.Model):
name = models.CharField(max_length=1024)
tag = models.CharField(max_length=1024)
repo = models.CharField(max_length=1024)
-
DPort: list of all ports defined in the JSOn file, and their default host mapping.
link to code
class DPort(models.Model):
description = models.CharField(max_length=1024, null=True)
hostp = models.IntegerField(unique=True)
hostp_default = models.IntegerField(null=True)
containerp = models.IntegerField()
container = models.ForeignKey(DContainer)
protocol = models.CharField(max_length=32, null=True)
uiport = models.BooleanField(default=False)
label = models.CharField(max_length=1024, null=True)
-
DVolume: List of all volumes and their share mapping. Note the
uservol field enabled during post-install customization (see below), as well as its connection with the Share model, populated when the user selects a pre-defined Rockstor share from the drop-down menu at rock-on install.
link to code
class DVolume(models.Model):
container = models.ForeignKey(DContainer)
share = models.ForeignKey(Share, null=True)
dest_dir = models.CharField(max_length=1024)
uservol = models.BooleanField(default=False)
description = models.CharField(max_length=1024, null=True)
min_size = models.IntegerField(null=True)
label = models.CharField(max_length=1024, null=True)
-
DContainerLabel: Used during post-install customization only, this model stores simple key:value mappings for docker container labels.
link to code
class DContainerLabel(models.Model):
container = models.ForeignKey(DContainer)
key = models.CharField(max_length=1024, null=True)
val = models.CharField(max_length=1024, null=True)
-
DContainerEnv: This model stores simple key:value mapping for a container’s environment variable(s).
link to code
class DContainerEnv(models.Model):
container = models.ForeignKey(DContainer)
key = models.CharField(max_length=1024)
val = models.CharField(max_length=1024, null=True)
description = models.CharField(max_length=2048, null=True)
label = models.CharField(max_length=64, null=True)
-
DContainerDevice: This model stores simple key:value mapping for a container’s device binding.
link to code
class DContainerDevice(models.Model):
container = models.ForeignKey(DContainer)
dev = models.CharField(max_length=1024, null=True)
val = models.CharField(max_length=1024, null=True)
description = models.CharField(max_length=2048, null=True)
label = models.CharField(max_length=64, null=True)
-
DContainerArgs: This model stores command argument(s) to be added to the
docker run command, as defined in the JSON file.
link to code
class DContainerArgs(models.Model):
container = models.ForeignKey(DContainer)
name = models.CharField(max_length=1024)
val = models.CharField(max_length=1024, blank=True)
-
DContainerLink: This model store simple source:destination mapping used for docker container links.
link to code
class DContainerLink(models.Model):
source = models.OneToOneField(DContainer)
destination = models.ForeignKey(DContainer,
related_name='destination_container')
name = models.CharField(max_length=64, null=True)
-
ContainerOption: This model stores the values of specific options defined in the rock-on’s JSON file. This information is not surfaced to the user.
link to code
class ContainerOption(models.Model):
container = models.ForeignKey(DContainer)
name = models.CharField(max_length=1024)
val = models.CharField(max_length=1024, blank=True)
As you may have noticed, one last model, DCustomConfig, differs from the others as it exists at the rock-on level rather than at the container level. This model has a very specific use for the installations of two rock-ons (owncloud, and openvpn) for which dedicated scripts exist:
Files
Three main files interact with these different database models:
-
rockon.py: fetches the list of all available rock-ons in the metastores (remote and local) and fills information from JSON definition files into the database.
-
rockon_id.py: handles specific operations on a given rock-on: install, uninstall, update, start, and stop.
-
rockons.js: responsible for all UI operations, such as list listing all available and installed rock-ons, and installation wizard, and post-install customization.
In addition, related helpers are located in rockon_helpers.py.
Rock-ons catalog (list)
To get the list of available rock-ons, Rockstor uses two different sources (termed “metastores”): one remote, and one local. Both are defined in Rockstor’s django settings:
link to code
ROCKONS = {
'remote_metastore': 'http://rockstor.com/rockons',
'remote_root': 'root.json',
'local_metastore': '${buildout:depdir}/rockons-metastore',
}
Remote and local metastore
The remote metastore includes a list of all rock-ons in JSON format (root.json) as well as the JSON definition file for each rock-on in the root.json list. All files can be found in the Github rock-on registry.
Rockstor first fetches information from the remote metastore, loading the JSON definition files for each value in the root.json file. It then proceeds in doing the same for the local metastore.
link to code
This procedure is triggered in rockon.py through the call to _get_available():
def post(self, request, command=None):
(...)
rockons = self._get_available()
(...)
for r in rockons:
try:
self._create_update_meta(r, rockons[r])
(...)
As can be seen above, Rockstor then proceed with updating the database information for each rock-on through _create_update_meta().
Rock-on definition fetching
The entire process is taken care of by _create_update_meta().
link to code
First, a set of default values is used to create a RockOn model instance for the given rock-on. This instance is then completed and updated with the contents of the rock-on JSON definition file. This includes rock-on-related information such as its description, link to project website, version, presence of webUI, more_info section, etc…
Then, the “containers” key in the rock-on JSON definition file is parsed to list the names of all the containers included in the rock-on. Each value (container) is then parsed to extract the container’s settings and update the corresponding models in the database accordingly. While the process is fairly straightforward, there are several important points:
- If no information regarding the docker image’s tag is found in the JSON file, the default is set to
latest.
link to code
defaults = {'tag': c_d.get('tag', 'latest'),
'repo': 'na', }
io, created = DImage.objects.get_or_create(name=c_d['image'],
defaults=defaults)
co.dimage = io
- When filling the default host port for a given host:container port mapping, Rockstor checks whether the host port is already set to be used by other available rock-on (installed or not). If it is indeed the case, it will simply set the default to use the next available port. This explains why the default port presented to the user during a rock-on installation wizard can differ from the one defined in its JSON file. Notably, this check is only performed within ports defined in rock-ons’ definitions and is thus not aware whether a port is actually in use by another service on the local system. This is especially important to keep in mind for users who are running other applications on their machine as conflicts can then arise.
link to code
def_hostp = self._next_available_default_hostp(p_d['host_default']) # noqa E501
Display in UI
Main file of interest: rockons.js
With all rock-on information in the database, Rockstor’s front-end can simply fetch this information and present it to the user. This is done via the Backbone collection RockOnCollection, which in turn refers to the /api/rockons call to get and return the list of all rock-ons from the RockOn model in alphabetical order:
link to code
return RockOn.objects.filter().order_by('name')
This list is then by processed by the renderRockons function to extract the information used in the UI—such as the rock-on’s name, description, “install” button, “link to the webUI”, or status—built through the following handlebars template:
link to code
Note that the status of all rock-ons with pending operations (install, uninstall) is updated automatically every 15 sec:
link to code
this.updateFreq = 15000;
.