TCW22: Building a TEI Release
Martin Holmes
James Cummings
Lou Burnard
Sebastian Rahtz
David Sewell
Kevin Hawkins
Hugh Cayless
Peter Stadler
Elli Mylonas
Elisa Beshero-Bondar
Raffaele Viglianti
Jessica H. Lu
Syd Bauman
Patricia O Connor
TEI Technical Council
2018
This document was originally a section of tcw20.xml, but has now been spun off into its
own document.
Clarified where to check on the Vault for the current releases of Stylesheets and Guidelines needed for building the oXygen plugin.
Revised instructions on how to update the table of releases for the GitHub website.
Added reminder to review release-blocker labelled issues.
Added reminder to mention in the Release Notes any changes affecting ODD customizations.
Updated Docker build info to account for new Roma (was Romabeta) and Roma Antiqua (was Roma). Clarified Stylesheets GitHub release process.
Per Nick Cole, since $PATH is not set up on server, need to specify
current directory in path to some executables.
Adding a step to update the version of p5subset.xml which is in the Stylesheets repo,
prior to doing any of the release steps.
Additional changes based on February 2020 release to make links visible, clarify release steps, confirm primacy of Paderborn Jenkins server.
Update all references, and add links, to Jenkins servers. Revise instructions regarding release version numbers.
Update all references about ssh to the tei server to the new HumaNum server consistently. Correct small grammatical errors.
Add updates and clarifications learned from last release.
Add information about setting JAVA_HOME for updating the Debian packages
Be explicit about contacting Jenkins maintainers ahead of time
Recommendations for issue handling that facilitate preparing the Release Notes.
Beginning edits arising out of the experiences
releasing 3.0.0. More to come.
Fixed error in git command as just reported on
tei-council
Edits for new release process.
Updates for new Jenkins server subdomains.
Update for use of ant build process for oxygen-tei rather
than the old .sh file.
More updates to account for the shift from SF SVN to
GitHub.
Updates to account for the shift from SF SVN to
GitHub.
Updates arising out of release process for 2.8.0.
Added link to http://wiki.tei-c.org/index.php/IRC
Updates to take account of oxygen-tei's move from Google
Code to GitHub, along with a couple of other tweaks.
Updates for changes in procedure ahead of the 2.6.0
release.
Updates following move of some code from SourceForge to
GitHub, and automation of the P5 version number in output.
Updates following Primrose Path release
experience.
Newsfeed blog is no longer in SourceForge.
A few changes based on messages to tei-council on
2012-06-19 and 2012-06-20.
Improvements suggested by various Council members based on
experiences with version 2.1.0.
Explained version numbers and added reminder about
including readme file in announcement.
Document forked from tcw20.xml.
Building a TEI Release
This document aims to provide a set of detailed instructions enabling a "release
technician" (the Council member tasked with implementing a new release of the TEI) to
prepare for and manage the release process. It assumes that the new packages will be taken
from one of the Jenkins servers rather than being built locally by the release technician.
This is easier and more reliable, because we ensure that the Jenkins servers are regularly
updated and are correctly configured to build the TEI products.
Packages on GitHub
The TEI maintains a number of distinct repositories on GitHub under the
TEIC organization. The main repository for developing the P5 Guidelines
and associated schemas is TEIC/TEI, and the TEI Stylesheets, the code for
the Roma web application, and the source code for the Oxygen TEI plugin can be found in
TEIC/Stylesheets,
TEIC/Roma, and
TEIC/oxygen-tei respectively.
The rest of this section describes how to make a new release for the main
P5 package, but similar procedures apply to the others. The instructions
assume you are working on a Linux or MacOSX system with a command line, and know (more or
less) how to do basic command-line operations such as running scripts and logging into a
server with ssh.
What you will need before you start
An account on GitHub, and committer privileges over the TEI repository. If you have
ever pushed a change to the TEI repository, you should have all the
required permissions.
Shell access on the TEI SourceForge project. Contact one of the admins to have this
turned on. Normal committers don't have shell access.
The release manager will need SSH login access to the tei account on the tei-c.org
server. This involves two steps:
Generate an SSH key pair (if you don't have one already). If this is new to you,
look at https://en.wikipedia.org/wiki/Ssh-keygen.
Send the public key to the Council Chair, who will forward it on to the system
administrator.
Make sure you get this set up well in advance of the release day, and make sure
you can ssh tei@cchum-kvm-dockerteic.in2p3.fr successfully. (Note: if you are a member
of the Infrastructure Group, you will already have your own login to the server, but you
should test it.)
Some familiarity with the three TEI Jenkins Continuous Integration Servers.
https://jenkins.tei-c.org
https://jenkins2.tei-c.org
https://jenkins3.tei-c.org
Take a little time to watch them work, and see how a push to the GitHub
repository causes them to start building TEI packages. There are three specific build
jobs associated with P5, and they run in a fixed sequence. Currently, as of February 2020,
the actual build for the release will rely on the first Jenkins server, often referred to as
the "Paderborn Jenkins" (maintained by Peter Stadler in Paderborn, Germany).
Access to the TEI Council Slack
which will be the communication channel during the release process.
Several hours of time. You won't be busy all the time, but the process from
beginning to end takes several hours, especially if something goes a bit wrong and you
have to retrace your steps. It's best to start first thing in the morning, and prepare
to be busy all day.
A copy of the public key that will enable you to sync the release zip with
SourceForge.
log in to the tei server at ssh tei@cchum-kvm-dockerteic.in2p3.fr (this requires that you've completed
the other public key step above). (Note that if you are a member of the Infrastructure Group
with your own login, log in as yourself, but sudo su tei before running any scripts.)
cat ~/.ssh/id_rsa.pub and copy the contents to the clipboard.
paste the result into a text editor and remove any linebreaks added by the
terminal.
copy-paste the result into https://sourceforge.net/auth/shell_services
What this does is to enable you (when logged in as tei to tei-c.org) to connect
to SourceForge (as your SF user) to upload the release files.
Test it by trying to log into SourceForge via ssh from the tei-c.org server:
ssh sfuser,tei@frs.sourceforge.net where "sfuser" is your SourceForge
user name. You should not see a prompt for a password (because of the ssh keys you have
set up). Instead, you should immediately see Welcome! This is a restricted Shell
Account. You can only copy files to/from here. If you see this, then everything is
set up correctly.
Issue Handling Recommendations
Assign tickets to release milestone
Use ticket reference in any commits addressing or fixing the issue
At least for the final commit addressing the issue try to prepare the commit message so it can be used for Release Notes
Label the issue as Release Note (green label) if it’s important to include the note about it in Release Notes
Step-by-step instructions
1-2 weeks before release:
Get p5subset.xml from a fresh build of the P5 dev branch (preferably on Jenkins, at a URL such as
https://jenkins.tei-c.org/job/TEIP5-dev/lastSuccessfulBuild/artifact/P5/release/xml/tei/odd/p5subset.xml) and
update the version of p5subset.xml in the Stylesheets/source directory in the Stylesheets dev branch.
Inform the Jenkins maintainers (who may not be on the Council listserv), and the
TEI sysadmin, of the pending release date, so that they can be available or on-call.
Ask one of the Jenkins maintainers (Peter Stadler, Martin Holmes, and Raffaele Viglianti)
to run a link check on the Guidelines and fix broken links in the dev branch.
Ask for the TEI-C GPG key passphrase. You'll need it for signing the Debian packages.
(The GPG private key itself is hosted on the server at the default location.)
Communicate with the TEI Council chair to make sure that the P5/ReleaseNotes/readme-X.X.X.xml
is compiled Normally, this will be created by the TEI Council chair at the
point when the repository moved from its "alpha" stage to "beta", following these steps:
Confirm the version number for the new release in consultation with Council. TEI
version numbers are based on the Unicode Consortium system (http://www.unicode.org/versions/) but with the first digit for major
changes, the second for schema-changing revisions, and the third for
non-schema-changing revisions. When the first or second digit is incremented, the
following digit or digits is set to zero. During initial development, the version
number is followed by "alpha"; during the pre-release checking stage, it's followed
by "beta"; and when the release takes place, "beta" is removed and the version
number has only digits and periods.
Create the new file by cloning a previous one.
Consult the git log log to check for all the significant changes since the last
release. You can do this by opening a terminal in the root of a working copy of the
TEI repository and running:
git log --after=2015-08-01 where the date is that of the previous
release.
The Release Notes ought to list changes that affect ODD customizations even when they do not affect the resulting schemata, such as (but not limited to) the renaming of a class,
or the moving of an attribute definition from an element to an attribute class.
Add the new file into the repository with git
add.
Review all issues labelled as Release Blocker (red label) At least one week before releasing, we enter a review period, during which the only
changes made to the code to be released should be to fix errors, rather than to add new
features. Release branches for the TEI and Stylesheets repos will be created by the release technician, to which ONLY
release-blocking bug fixes and corrections to typographical errors will be pushed. The
release technician should announce a temporary hold on pushes to the dev branches on the
Council list, then create the branches and push it to GitHub using the following commands, once in the
TEI repo and once in the Stylesheets repo:
git checkout dev (make sure you start from the dev branch) or if you have the dev branch
checked out, git status in order to make sure that you have the current version and no uncommitted changes.
git checkout -b release-X.X.X
git push -u origin release-X.X.X
Immediately after the release branches have been pushed,
the release technician should inform the Council
list and ask the maintainers of the TEI Jenkins
systems to add a build of the release branch so that commits
pushed there are properly tested, and advise them of the
release schedule. Remember that the maintainers (currently
Martin Holmes, Peter Stadler, and Raffaele Viglianti) may not be on the Council
list. Verify with the maintainers that all Jenkins servers are functioning properly.
Pushes to the release branch should be merged back into dev regularly:
git checkout dev
git merge release-X.X.X
On release day:
The instructions below for Git commands are assumed to be run in the release-X.X.X
branches unless otherwise specified. If you did not create the release branch, you can
set up a copy of it using the following commands:
git pull origin (to make sure your copy knows about the release branch)
git checkout --track origin/release-X.X.X
Check the release notes for typos or other glaring errors, one last time.
Edit the P5/VERSION file to the correct number This file
consists only of the bare version number, followed by "alpha" or "beta":
2.8.2beta For the release process, you need to remove the letters from
the end, leaving a pure version number:
2.8.2 This changes the release from beta (or possibly alpha) to the
actual release version number.
Get p5subset.xml from a fresh build of the P5 release branch (preferably on Jenkins, at a URL such as
https://jenkins.tei-c.org/job/TEIP5-release-X.X.X/lastSuccessfulBuild/artifact/P5/release/xml/tei/odd/p5subset.xml) and
update the version of p5subset.xml in the Stylesheets/source directory in the Stylesheets release-X.X.X branch.
Announce a freeze on pushes to the release branch on the
TEI Technical Council mailing list
Merge the release branch into released.
git checkout released
git merge --no-ff -m "YOUR COMMIT MESSAGE" release-X.X.X
Repeat the steps above for the Stylesheets (have a stylesheets expert on hand
when releasing Stylesheets to help debug problems):
Edit the Stylesheets/VERSION number to the correct release number (usually just
remove the 'a').
Run make log to generate the changelog. Then commit the changes.
Merge the release branch into released
git checkout released
git merge --no-ff -m "YOUR COMMIT MESSAGE" release-X.X.X
Push your changes for both P5 and the Stylesheets to the git
repository, git push origin released and watch Jenkins build P5 for you.
This should be the final push for this version, and it will trigger the Jenkins servers
to rebuild the TEI packages. As a reminder, you can find the Jenkins servers here:
https://jenkins.tei-c.org
https://jenkins2.tei-c.org
https://jenkins3.tei-c.org
And now you wait while the Jenkins servers build the packages, keeping in mind that
subsequent steps will require the build packages from the Paderborn Jenkins server
(as of February 2020). This can take a couple of hours, so be patient.
Note: The P5 and Stylesheets builds have reciprocal dependencies,
so the first build of either the Stylesheets or the Guidelines may fail just because
there isn't yet a current build of the other for it to use. This isn't a cause for
panic, but it may mean that (e.g.) the Stylesheets build needs to run twice. In particular,
the Stylesheets build may fail after the TEI release is complete, so it is better to wait
until the TEI release is complete before doing the Stylesheets release. (Council is considering
changing how the dependency is managed).
Ensure all changes have been committed, built, and successfully
passed tests on the continuous integration server When all builds have
completed on all Jenkins servers, click on the job number of the last build for each of the
three TEI jobs to make sure that it was triggered by the commit that you made in the
previous step (you should see your own commit message on the build page). Make sure that
all builds were successful (they should have green balls next to them). In the case of red balls,
indicating a failed build, seek support via the TEI Council Slack.
Log into TEI server and run the tei-install.sh script:
Log into the TEI server via ssh tei@cchum-kvm-dockerteic.in2p3.fr.
Note that if you are a member of the Infrastructure Group
with your own login, log in as yourself, but sudo su tei before running any commands.
Then fetch the current version of the install script by issuing
curl https://raw.githubusercontent.com/TEIC/TEI/dev/P5/Utilities/tei-install.sh -o ~/tei-install.sh.
(If you'll need to tweak that script later during the install process please make
sure to feed the changes back to the original script in the TEI repo.)
Do the following three steps:
Install on tei-c.org: ./tei-install.sh --package=TEIP5 --version=X.X.X
--sfuser=username --install and then go test the version this puts in
the Vault.
If that looks good and everyone agrees then: ./tei-install.sh
--package=TEIP5 --version=X.X.X --sfuser=username --makecurrent and
then test that it appears on website correctly.
If the website looks right then: ./tei-install.sh --package=TEIP5
--version=X.X.X --sfuser=username --upload and then move on to the next
step.
In each of these steps, replace the Xs with your release version. Supply your
SourceForge user name, and type your SourceForge password when prompted. By default, the
script will pull the release package from the first Jenkins server, but you can supply
the URL of the other server if necessary with the --Jenkins parameter, e.g.
--Jenkins=http://jenkins2.tei-c.org/. Make sure the script completes successfully each time
changing the final parameter from --install, to --makecurrent, and then --upload.
Repeat the steps above for the Stylesheets, remembering that the version number is the Stylesheets version, which will be different from
the Guidelines version:
./tei-install.sh --package=Stylesheets
--version=X.X.X --sfuser=username --install
./tei-install.sh --package=Stylesheets
--version=X.X.X --sfuser=username --makecurrent
./tei-install.sh --package=Stylesheets
--version=X.X.X --sfuser=username --upload
Check the TEI website and all downloadable files are displaying the
correct version Everything should now be done, so go to the newly
released version on the TEI site and browse the Guidelines. Check that your
version number is displayed in the footer of the page, and check that at least one
change made since the last release is being reflected online.
Update Roma,
Roma Antiqua, and TEIGarage
Change to the directory ~/repos/infrastructure/humanum, do a git pull, and issue the script
docker-update.sh for every service, i.e.
# ./docker-update.sh roma
# ./docker-update.sh romaantiqua
# ./docker-update.sh oxgarage
# ./docker-update.sh teigarage
NB: Every invocation might take some time but each should succesfully finish with the notification
"Creating $CONTAINER$ ... done"
Make your release the default downloadable version from
Sourceforge Go to the SourceForge site (https://sourceforge.net/projects/tei/files/TEI-P5-all/),
log in, and click the information button on your new release. Make it the default download for all
operating systems. Now make sure that the main Download button links to your package.
Update tags on GitHub Every time a new release is made, a
"tag" is created consisting of a pointer to the state of the P5 tree at release time.
You can do this from the command line on your own computer. Before moving forward, be
sure to do a git pull and update your released branch. Then, still in the released
branch, do:
git tag -a P5_Release_X.X.X -m "Release X.X.X of the TEI Guidelines."
where X.X.X is your new release. Then
git push origin P5_Release_X.X.X
Do the same for the Stylesheets as well, on the released branch:
git tag -a vX.X.X -m "Release X.X.X of the TEI Stylesheets."
where X.X.X is your new release. Then
git push origin vX.X.X
Make a Release on GitHub Go to the TEI Tags page at https://github.com/TEIC/TEI/tags on GitHub. You should
see the tag you just pushed there. Click on it and then click on "Create Release". Add a link to the release notes README, which should be at
https://www.tei-c.org/release/doc/tei-p5-doc/readme-X.X.X.html, into the text box. Add a copy of the zipped
release by downloading it from https://jenkins.tei-c.org/job/TEIP5/lastSuccessfulBuild/artifact/P5/ and then uploading it to the release page.
Do the same for the Stylesheets as well. Go to the Stylesheets Tags page at https://github.com/TEIC/Stylesheets/tags on GitHub and follow the same procedure.
Close/Add GitHub Milestones Go to the Milestones page of both the
Stylesheets and the
Guidelines repo. Add new Milestones by
incrementing the minor version number part and move all open issues from the current Milestone to the new one.
(If Council already decided on the next release date (unlikely) you can add the date to the new Milestone. Otherwise leave empty.)
Finally, close the current Milestone (do not delete it).
Update the Debian Package repository with the new release
The TEI Debian packages are regularily created during each Jenkins build.
For each release you need to update the TEI Debian repository at
http://packages.tei-c.org/deb/
which can be done by simply running ant on the TEI server within the
/data/debian-packages directory.
(This directory is cloned from https://github.com/TEIC/TEI-apt-repo)
Since the repository index needs to be signed, you'll need the passphrase for the GPG key.
Make sure you've received it in advance!
Once the process has finished and the repo is updated, the page at http://packages.tei-c.org/deb/
should reflect the changes immediately. If not, try to restart the NGINX Docker container that serves this directory
with docker restart debian-packages.
Update the table of previous releases of P5 in the GitHub repo for the TEI-C website. The table is found in the src/guidelines/p5 markdown page,
published at https://tei-c.org/guidelines/p5.
Add the new release information and links to the top of the release table.
Update the oXygen-ready distribution of TEI. This involves
building the new package of oxygen-tei, and then updating the distribution file on the
TEI server so that the oXygen software knows about the new release. You may request to
hand this off to one of the maintainers (currently Syd Bauman, James Cummings, or Martin
Holmes) to do this for you if you're not familiar with the project.
Check that you have ant (at least version 1.8) installed on your machine.
Check that the latest versions of the Stylesheets and TEI P5 are
available from the TEI Vault, since the oxygen-tei update/upload script retrieves them from there.
Check for the Stylesheets at https://tei-c.org/Vault/Stylesheets/.
Check for the Guidelines at https://tei-c.org/Vault/P5/.
Check out or update a local copy of the source code from https://github.com/TEIC/oxygen-tei to your local system.
cd into the oxygen-tei folder (it should contain folders called "frameworks" and
"jenkins").
Run the ant build process with the "release" parameter:
ant release This builds the plugin using the latest stable
versions of P5 and the Stylesheets, then offers to upload the result to the TEI's
SourceForge repo to become a release of the TEI-maintained version of the plugin.
This also creates an updated updateSite.oxygen file, by retrieving the latest
updateSite.oxygen file from the tei-c.org site, and asks the user to provide the new
version number before creating a new version of updateSite.oxygen.
Go to the GitHub repo and create a new release, using the Draft new release
button. Copy the previous release info (tag, title and text), and tweak the versions as appropriate. The tag should be vx.x.x, the title will be
Release x.x.x, and the text Release number X of the oXygen TEI plugin, based on TEI P5 x.x.x and Stylesheets x.x.x.
Attach the zip file you just created in the build process, which will be named
something like oxygen-tei-x.x.x-x.x.x.zip, with the numbers from
the current TEI and Stylesheets releases.
Once the tag/release has been published, run ant uploadOxygenUpdateFile
to push the updateSite.oxygen file up to the TEI server.
Inform the TEI Technical Council Chair so they can announce the
release Once you are sure that everything is working correctly, inform the
Council Chair. They will announce the release to the TEI-L mailing list, including the
text of P5/ReleaseNotes/readme-X.X.X.xml in plain text form (which can be generated
using the "readme" profile for teitotxt), and place an announcement on the Text Encoding
Initiative Newsfeed blog in the category of 'News'.
The Chair should contact the Social Media coordinator (currently Anna Sofia Lippolis) to help spread the news.
Lift the freeze on committing changes to the repository
Write to the TEI Council list and let them know that they can once again start
committing changes to the repository.
Increment the build number for the next release cycle
Recalling how release preparation requires confirmation of version number,
use your best judgment to determine the version number for the next release (in consultation with Council,
if possible). TEI version numbers are based on the Unicode Consortium system (http://www.unicode.org/versions/) but with the first
digit for major changes, the second for schema-changing revisions, and the third for
non-schema-changing revisions. When the first or second digit is incremented, the
following digit or digits is set to zero.
After the release process has been completed, the release number for both P5 and the
Stylesheets needs to be updated. On the dev branch, edit the P5/VERSION file and the Stylesheets/VERSION
file to the correct numbers. These files contain nothing except the bare version number.
It should be incremented appropriately, and "a" added to the end of it, so if for example
the release was number 2.8.0, you would change the number in the file to:
2.9.0a signifying that the versions built subsequent to the release
are now in the alpha stage.